Parfeval memory leak? Clear memory on parallel workers?
7 views (last 30 days)
Show older comments
Matthias
on 9 Dec 2014
Commented: Francis Plamondon
on 18 Nov 2020
Hi,
I have some sort of memory leak when using parfeval.
I have a function that starts a number of parfeval-computations and then uses fetchNext in a loop to retrieve all results.
The memory taken up by the Matlab instances in my parallel pool grows with each execution of the function that uses parfeval.
My question: Am I doing something wrong? Do I have to explicitly clear the memory of the workers after using parfeval? How do I do this? I tried "pctRunOnAll clear all" but it did not free up any memory on the workers (just the client instance).
Right now, the only way I can clear this memory is to shut down and re-start my parallel pool, but that's not a feasible solution for my application.
Thanks, Matthias
3 Comments
Atakan Atay
on 25 Mar 2020
Edited: Atakan Atay
on 25 Mar 2020
Hi Matthias,
Currently I experience the very same problem. How did you manage to solve it? I would really appreciate your help.
Thanks for your time.
Francis Plamondon
on 18 Nov 2020
Hello,
I also recently struggled with parfeval. I would get 'out of memory' errors after several hours of running a huge batch of jobs.
Here is what I found out: each parallel.FevalFuture object holds a Diary property that contains everything that was displayed in the console window for each job. Since I had a function use disp() for monitoring purpose, it was being recorded in the Diary variable, eventually filling all the memory.
After commenting out the disp function, the memory stopped slowly increasing.
Accepted Answer
Edric Ellis
on 15 Dec 2014
Please could you try applying the workaround from this bug report. (The bug report mentions mapreduce, but it includes a fix that affects only parfeval).
1 Comment
Seth
on 13 Feb 2015
I have a similar problem and tried using this patch and now get errors indicating "no method 'createInstance' with matching signatures found for class 'com.mathworks.toolboox.distcomp.objects.EntityFileProxy'
Do you have any other suggestions? I was running two workers with PARFEVAL and seeing 3gb increase in RAM used each call to them. (The PARFEVAL commands are in a loop)
Thanks.
More Answers (0)
See Also
Categories
Find more on Asynchronous Parallel Programming in Help Center and File Exchange
Products
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!