Out of memory issue
1 view (last 30 days)
I have a sparse matrix which has a size of, (2031616x4096 sparse double).
I want to make 10 subset of this matrix and need to convert this matrix in to "full". So, I have made 10 subset and while processing it was able to covert 5 subset into " full" and for the rest of the five subset it is showing me the message : "out of memory".
One thing I have done that is, when I saved the each subset separately into a external harddisk may be at that time it will be possible but here again the problem appears during the importing the subset matrix. I know that it is not a solution .
I will appreciate if anyone can suggest some helpful solution to tackle this problem.
Steven Lord on 20 Sep 2021
sz = [196608 4096];
numberOfElements = prod(sz);
bytes = 8*numberOfElements;
gb = bytes/1024^3
Each subset is 6 GB in size (assuming real doubles, 12 if complex doubles) when converted into a full array. Depending on how much memory you have available it's quite possible you couldn't fit all ten subsets in memory at once.
Can you avoid the conversion to a full matrix? MATLAB can save sparse matrices to MAT-files and load them from those files perfectly well. Alternately if you need to store them in a format some other program can read perhaps write a coordinate list (which you can generate using the find function) to the disk?
Alternately clear-ing each subset once it has been written to disk may allow MATLAB to reuse that memory for the next subset.