Efficiently handling extremely large matrices (N x N) in MATLAB (e.g., N = 137 million)

I'm facing challenges in handling very large matrices in MATLAB, specifically matrices with dimensions like N x N, where N can reach values on the order of 137 million. Directly creating and manipulating such matrices in MATLAB is impractical due to memory limitations.

I'm seeking guidance on effective strategies for dealing with these large matrices in MATLAB. Here are some specific questions I have:

1. What alternative approaches can be used instead of directly creating and manipulating the entire matrix?
2. Could you provide examples of using sparse matrices, Tall Arrays, or block processing techniques in such scenarios?

facing errors like ("Error using zeros Requested array exceeds the maximum possible variable size.) and sometimes showing required RAM for operation as 1022.1 Gb similar errors .

Answer 1

[Matt J]

Could you provide examples of using sparse matrices

You give us wide latitude on what the example can be be, so here is one:

A=speye(137e6);

whos A

  Name              Size                        Bytes  Class     Attributes

  A         137000000x137000000            3288000008  double    sparse