Out of memory issue

I have a sparse matrix which has a size of, (2031616x4096 sparse double).

I want to make 10 subset of this matrix and need to convert this matrix in to "full". So, I have made 10 subset and while processing it was able to covert 5 subset into " full" and for the rest of the five subset it is showing me the message : "out of memory".

One thing I have done that is, when I saved the each subset separately into a external harddisk may be at that time it will be possible but here again the problem appears during the importing the subset matrix. I know that it is not a solution .

I will appreciate if anyone can suggest some helpful solution to tackle this problem.

[Steven Lord]

What sizes are these 10 subsets that you're trying to create?

[Walter Roberson]

It sounds as if you will have multiple subsets in memory ?

[Amit Chakraborty]

The size of each subset is : 196608x4096 double

exactly @Walter Roberson

Answer 1

[Steven Lord]

sz = [196608 4096];
numberOfElements = prod(sz);
bytes = 8*numberOfElements;
gb = bytes/1024^3

gb = 6

Each subset is 6 GB in size (assuming real doubles, 12 if complex doubles) when converted into a full array. Depending on how much memory you have available it's quite possible you couldn't fit all ten subsets in memory at once.

Can you avoid the conversion to a full matrix? MATLAB can save sparse matrices to MAT-files and load them from those files perfectly well. Alternately if you need to store them in a format some other program can read perhaps write a coordinate list (which you can generate using the find function) to the disk?

Alternately clear-ing each subset once it has been written to disk may allow MATLAB to reuse that memory for the next subset.

[Amit Chakraborty]

@Steven Lord:

** All the subsets are with real value no complex value is there.

** I need to convert the sparse matrix into "full" because i need this for further calculation.

** If it is possible for you to show me an example of writing coordinate list with "find" function?

** Lastly, yes I have been clearning the unwanted variable once I worked with it so that MATLAB can reuse the memory.

Thanks to Everyone !