Parfor loop and variable length output

Hello everyone,

I would like to speed-up a code using a parfor loop but my function is not straightforwardly suitable for it. In particular, the problem is the for loop contains a function whose output has a variable length and I need to store all the variable outputs in a final vector.

I can predict the maximum length of the final result, but most likely I will only use one tenth of this maximum.

result = zeros(1,10000);
N_result = 0;

for i = 1:N
  result_i = generate_result(i); % This has a variable number of elements
  result(N_result+1:numel(result_i) = result_i;
  N_result = N_result+numel(result_i);
end

I would like to use the parfor instead of a for, but I don't know how merge the variable result_i in a single output result. It would be perfect if I could create a variable length vector for each worker, and then merge those vectors.

How can I do this?

Alessandro.

Answer 1

[Edric Ellis]

You have several options here. Probably the simplest is to return a cell array and then concatenate the contents. It might not be terribly efficient though if N is large compared to the number of outputs you're returning (this is because cell arrays where each cell contains only a small number of elements have memory and performance overhead compared to a numeric array).

Here's an example. (Note the use of feval to work around a PARFOR restriction on using function handles)

% Here's a simple function that generates a random array of a random size:
generate_result = @(x) x * rand(1, randi([0,10]))

parfor idx = 1:7
  output{idx} = feval(generate_result, idx);
end
% Now, concatenate the output elements into a single array
output = [output{:}];

[Alessandro Masullo]

Thank you for the answer. Just out of curiosity, is this the proper way of doing what I want to do? Does the feval affect the speed?

Thank you.

[Edric Ellis]

You don't normally need to use feval - I needed it only because generate_result is a function handle, and there's a limitation with PARFOR which means you must invoke those using feval. In your case, if generate_result is an ordinary function, you shouldn't use feval.