efficient way to work with ~2k variables of size ~400x400x3

Hello,

I am running an optimization algorithm that involves iterating over ~2k variables, each variable representing an image of size ~400x400x3. In my iterations, I need to load these variables, update them, and go back to the next round, something like this

for iter=1:100
  for var_no=1:1225
  % Load var1_i
  load(['var1_', num2str(var_no), '.mat');
  % Load var2_i
  load(['var2_', num2str(var_no), '.mat');
  % Update var1_i, var2_i
  var1_i = update_var1(var1_i);
  var2_i = update_var2(var2_i);
  % Save them again
  save(['var1_', num2str(var_no), '.mat', var1_i, '-v7.3');
  save(['var2_', num2str(var_no), '.mat', var2_i, '-v7.3');
  end
end

As it can be expected, my code is extremely time consuming, and the size and their no. being very large, I cannot keep them in RAM. Can somebody please help me out and suggest me an efficient way to carry this? It has become a huge bottleneck in my run time. Thanks so much!

[AA]

Hello, sorry I missed adding that changing the inner loop to parfor(as all the variables are independent) alleviates the problem to an extent, but still very time consuming due to the load/save operations

Answer 1

[Guillaume]

"I cannot keep them in RAM"

Why not? 400x400x3x2000 is under 8 GB of memory as double. Don't you have that much memory available on a computer capable of running the parallel toolbox? If not, I'd really consider upgrading the memory.

Furthermore, since your processing images it's very possible that your images started as 8-bit integers, in which case keeping them all in memory would only require about 960 MB.

[AA]

Dear Guillaume,

Thanks for your response. Yes, you are right. My algorithm however involves a variable no. of image sizes (say upto ~700x700x3), and I also could have more no. of variables (say upto ~5000). Hence, for the max. case of 700x700x3x5000, the memory consumption is ~39 GB, and since I am running my code on a shared server (being used by others as well), I often run out of space even when the server has 64 GB RAM available.

Thanks

[AA]

Also, I would like to ask, please bear with me for my inexperience, even if I say use the large 4-D matrix given enough RAM, using the variable inside parfor loop seems to be doing worse than individual loading and saving of the variables. What could be the possible reason and its solution?

[Guillaume]

Unfortunately, I don't use the parallel toolbox so can't help you with that last question. However, file IO is orders of magnitude slower than memory access so I would guess that something is wrong with your implementation.

What format did your images start as? If they were 8-bit rgb image (the most common format), then keeping 5000 images of size 700x700(x3 colour channels) as uint8 would only require 8GB of data.

As said, file IO is extremely slow compared to memory access, so getting rid of file IO should greatly speed up your code, and also make it simpler.

[Matt J]

What format did your images start as? If they were 8-bit rgb image (the most common format), then keeping 5000 images of size 700x700(x3 colour channels) as uint8 would only require 8GB of data.

This assumes, though, that the iterative update process can tolerate uint8 precision.

[AA]

Hello Matt and Guillaume,

Thanks for all your help. I am able to store them and run them in parfor all at the same time. I simply stored the entire matrix as single precision variables (since my algorithm is not required to generate very precise results), and I could reduce the memory consumption by half. The performance is in acceptable range now.

Thanks again!

Answer 2

[Matt J]

You could gain some speed-up uing MATFILE to access the data instead of load/save.

[AA]

Dear Matt,

Thanks for your response. I will have a quick check and see if it helps.

Thanks again!

[AA]

Dear Matt,

I tried that, but unfortunately, matfile obj access is not permitted inside parfor loops (my inner loop is parfor, which gives me some performance gain over using for). Do you have another way out?

Thanks!

[Matt J]

I don't see that limitation in the matfile documentation. Where did you read it? The following simple example worked fine for me,

    a = 0;

    save tst1 a
    save tst2 a
    save tst3 a

    m{1}=matfile('tst1','Writable',true);
    m{2}=matfile('tst2','Writable',true);
    m{3}=matfile('tst3','Writable',true);

    parfor i=1:3

        m{i}.a=i;

    end

    load tst1; a
    load tst2; a
    load tst3; a