Issue passing reduction variable by reference with parallel for-loop

Hello, I am working on a project that parses ~50GB of text (across several hundred files) and extracts the data into a multidimensional MATLAB array. At the center of my program is a for-loop that iterates over all of the text files in a given directory and extracts their data into a pre-allocated workspace array variable. Since each iteration of this for-loop is independent from the others (each iteration extracts data from a separate textfile), I am using a parallel for-loop to expedite the process. The pre-allocated array into which the information gets extracted is a “reduction variable” by definition since it “accumulates a value that depends on all the iterations together, but is independent of the iteration order” (source: http://www.mathworks.com/help/distcomp/reduction-variables.html ). However, since my pre-allocated array is very large (just barely fits in RAM), I am not passing it by value, but instead passing it by reference (just like a pointer in C). I do this through the use of a custom object that inherits from the handle class.

I have gotten my code to successfully fun while using a normal for-loop, however when I attempt to use a parallel for-loop I run into a cryptic scoping issue. Inside my parallel for-loop, the object which contains the pre-allocated array has correctly extracted values for ONLY THE CURRENT ITTERATION. Outside of the parallel for-loop, the object which contains the pre-allocated array remains in its initial state, IE the code executing in the parallel for-loop has not affected the pre-allocated array.

I have written a test script that showcases the issue (code is attached). When the code is run with a normal for-loop THE CORRECT RESULT EMERGES. However, when the code is run with a parallel for-loop, the described issue comes into play. One can see that the code executing in the parallel for-loop has no effect on the pre-allocated array.

Can you please take a look at the attached code and see if you can help me resolve the issue?

Thanks, Jake

Answer 1

[Edric Ellis]

Unfortunately, handle class objects get copied to and from the workers, as described in the documentation here.

Depending on the nature of your problem, a few potential solutions might be appropriate.

The simplest approach is to use a sliced output variable from parfor. parfor knows not to copy the whole value of a sliced output, and so something like this:

parfor idx = 1:n
  out(idx, :) = readDataFromFile(...);
end

only creates the full-sized variable out on the client. Unusually for MATLAB, it's not actually necessary to pre-allocate sliced outputs from a parfor loop because the parfor machinery already must be able to prove that it can determine the size of outputs.

A completely different approach (that still enables the use of parallel computing) is to use the "big data" facilities such as mapreduce. This is designed for reading large amounts of data from files (even for cases where the data cannot all fit into memory at one time), and perform computations on that data (generally reducing the amount of data down to something that can fit into memory by calculating things like the sum, max, etc.)

[Matt J]

Unusually for MATLAB, it's not actually necessary to pre-allocate sliced outputs from a parfor loop because the parfor machinery already must be able to prove that it can determine the size of outputs.

But if you do pre-allocate "out" then its slices will be cloned and broadcast to the workers?

[Edric Ellis]

If out is purely a sliced output, then in practice, the initial value is not sent to the workers - the client receives the result slices and assigns them into out.

[Matt J]

Good to know!

Answer 2

[Matt J]

Reduction variable operations must be of the form

   X = f(X, expr)

Note that "X" is not an indexed expression here. So, at minimum, you probably need a modification that looks like this,

   databasePointer = setRowToVal(databasePointer, i);

I don't quite understand, however, why you don't use sliced variables instead. Within the code setRowToVal you are writing into separate, slicable portions of matrix.Object.

Finally, you should bear in mind that handle objects get cloned when they are broadcast to parallel labs. In other words, any memory you allocate initially to databasePointer before the parfor will be hardcopied on each of the labs. So, the handle class may not be saving memory as you would hope. Sliced variables would be the better way to mitigate memory consumption, because then each lab only gets a slice of the original array's memory. It's possible, though, that the individual sliced submatrices are still cloned when broadcast to the labs (I'm not sure), meaning that you still effectively have at least 1 deep memory copy of your original array.