Parfeval memory leak? Clear memory on parallel workers?

Hi,

I have some sort of memory leak when using parfeval.

I have a function that starts a number of parfeval-computations and then uses fetchNext in a loop to retrieve all results.

The memory taken up by the Matlab instances in my parallel pool grows with each execution of the function that uses parfeval.

My question: Am I doing something wrong? Do I have to explicitly clear the memory of the workers after using parfeval? How do I do this? I tried "pctRunOnAll clear all" but it did not free up any memory on the workers (just the client instance).

Right now, the only way I can clear this memory is to shut down and re-start my parallel pool, but that's not a feasible solution for my application.

Thanks, Matthias

[Edric Ellis]

How much data are you sending as input arguments to your PARFEVAL calls? Which version of MATLAB are you using?

[Atakan Atay]

Hi Matthias,

Currently I experience the very same problem. How did you manage to solve it? I would really appreciate your help.

Thanks for your time.

[Francis Plamondon]

Hello,

I also recently struggled with parfeval. I would get 'out of memory' errors after several hours of running a huge batch of jobs.

Here is what I found out: each parallel.FevalFuture object holds a Diary property that contains everything that was displayed in the console window for each job. Since I had a function use disp() for monitoring purpose, it was being recorded in the Diary variable, eventually filling all the memory.

After commenting out the disp function, the memory stopped slowly increasing.

Answer 1

[Edric Ellis]

Please could you try applying the workaround from this bug report. (The bug report mentions mapreduce, but it includes a fix that affects only parfeval).

[Seth]

I have a similar problem and tried using this patch and now get errors indicating "no method 'createInstance' with matching signatures found for class 'com.mathworks.toolboox.distcomp.objects.EntityFileProxy'

Do you have any other suggestions? I was running two workers with PARFEVAL and seeing 3gb increase in RAM used each call to them. (The PARFEVAL commands are in a loop)

Thanks.