Out of memory issue

1 view (last 30 days)
Amit Chakraborty
Amit Chakraborty on 20 Sep 2021
Commented: Amit Chakraborty on 21 Sep 2021
I have a sparse matrix which has a size of, (2031616x4096 sparse double).
I want to make 10 subset of this matrix and need to convert this matrix in to "full". So, I have made 10 subset and while processing it was able to covert 5 subset into " full" and for the rest of the five subset it is showing me the message : "out of memory".
One thing I have done that is, when I saved the each subset separately into a external harddisk may be at that time it will be possible but here again the problem appears during the importing the subset matrix. I know that it is not a solution .
I will appreciate if anyone can suggest some helpful solution to tackle this problem.
Amit Chakraborty
Amit Chakraborty on 20 Sep 2021
The size of each subset is : 196608x4096 double

Sign in to comment.

Answers (1)

Steven Lord
Steven Lord on 20 Sep 2021
sz = [196608 4096];
numberOfElements = prod(sz);
bytes = 8*numberOfElements;
gb = bytes/1024^3
gb = 6
Each subset is 6 GB in size (assuming real doubles, 12 if complex doubles) when converted into a full array. Depending on how much memory you have available it's quite possible you couldn't fit all ten subsets in memory at once.
Can you avoid the conversion to a full matrix? MATLAB can save sparse matrices to MAT-files and load them from those files perfectly well. Alternately if you need to store them in a format some other program can read perhaps write a coordinate list (which you can generate using the find function) to the disk?
Alternately clear-ing each subset once it has been written to disk may allow MATLAB to reuse that memory for the next subset.
  1 Comment
Amit Chakraborty
Amit Chakraborty on 21 Sep 2021
** All the subsets are with real value no complex value is there.
** I need to convert the sparse matrix into "full" because i need this for further calculation.
** If it is possible for you to show me an example of writing coordinate list with "find" function?
** Lastly, yes I have been clearning the unwanted variable once I worked with it so that MATLAB can reuse the memory.
Thanks to Everyone !

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!