Working with a very large matrix

I would like to make a large matrix of size ~20000000 * 20000 and working with it. But it is not allowed because of memory shortage. as most of my elements at first, are zero, I could make the matrix with sparse function, but then my matrix will be filled with all elements of nonzero. the value of elements at first are 0-9 natural numbers and then real number (with 2 digit precision is fine).

Does anyone have any idea how to deal with such a large matrix? or any alternative way (data structure) to work with?

p.s: somebody suggested "parallel computing server", but I don't have it on my Matlab.

Thank you very much.

Answer 1

[John D'Errico]

You cannot specify that a given number will only have 2 decimal places stored. If floating point, it must be either a single or a double.

Your array has 2e7*2e5=4e12 elements in it. Stored as single precision numbers, that is 4 bytes per element. So a total of roughly 1.6e13 bytes of RAM. Roughly 16 terabytes. Do you have that much memory?

If there are only a TINY percentage of non-zeros, then yes, you CAN use sparse storage, as long as you create the array using the sparse function, without creating a full array first.

If the array will undergo fill-in with further manipulations, then sparse storage is not acceptable.

[sh as]

Yes, and I am looking for a solution. Do you think if I can make the matrix and the process on it with C/C++ code? (I mean using mex file). or it does change anything?

Thanks,

[John D'Errico]

Of course, IF you can create the matrix (in sparse form) and pass it along to use it in C/C++, then you can then do any processing on it that you wish, as long as you do not exceed your memory limits. Since I have no idea what processing you plan on doing, I cannot say that you can or cannot succeed. All of that depends on the degree of fill-in you have due to your processing.