Memory Parfor Needs more memory than normal
3 views (last 30 days)
Giannis Lantzanakis on 28 May 2020
Hello, i need your help. I dont understand something with parfor loop.
I load an image which is 8GB.
I use parfor loop with 4 threads.
So the total usage of Ram it had to be 8GB + 4thread*8GB = 40GB.
But when the parfor loops starts, the momory increases more than 64GB (this is my total memory)
everything lags, and after some minutes the memory goes back to 40GB and the program runs normally.
I dont understand. Why does it happen? Is there any possible solution? If it needs 40GB why it gets more memory in the beggining?
For bigger data or if i increase the processors i get a blue screen. Pls help!
Edric Ellis on 29 May 2020
When you transfer data to the workers for a parfor loop, there is a transient increase in memory over and above what you'd expect due to the way data is transferred. You can perhaps work around this by loading the image data directly on the workers. One way to do this is by using parallel.pool.Constant. When constructing your Constant, you can pass in a functino handle to load the data, and this gets executed on the worker. Something like this perhaps:
c = parallel.pool.Constant(@() imread('myLargeImage.png'));
parfor idx = 1:N
im = c.Value; % Get the actual image from within the Constant
out(idx) = myImageProcessingFunction(im, idx); % or whatever.