How to store and deal with very very large matrices?

70 views (last 30 days)
Barbab on 2 Aug 2023
Answered: Animesh on 22 Aug 2023
% input variables
T = 2274;
N = 58;
B = 10000;
H = 250;
K = 3;
% I simulate matrices of similar size to give the idea of the problem
rng('default') % even if replicability is not necessary in this case
% first set (based on dimension)
P0 = lognrnd(0, 1, T, N);
R0 = rand(T, N) - rand(T, N);
% second set (based on dimension)
R = rand(T, N, K) - rand(T, N, K);
V0 = rand(T, N, K);
Z0 = randn(T, N, K);
% third and foruth set (based on dimension)
Z1 = randn(H, B, N, K);
R1 = rand(H, B, N, K) - rand(H, B, N, K);
V1 = rand(H, B, N, K);
P1 = lognrnd(0, 1, H, B, N, K);
Z2 = randn(H, B, N, K);
R2 = rand(H, B, N, K) - rand(H, B, N, K);
V2 = rand(H, B, N, K);
P2 = lognrnd(0, 1, H, B, N, K);
The above code is not executable due to the large size of the matrices involved. While MATLAB can handle the computations, saving and loading these big matrices is extremely slow, and loading the '.mat' files often results in crashes.
I have attempted to use the matfile function to access the variables without loading them, but it doesn't seem to provide a satisfactory solution in terms of speed.
I would greatly appreciate any suggestions or improvements on how to efficiently deal with such big matrices in MATLAB. Is there a more optimized way to handle these large datasets, particularly for saving and loading operations?

Animesh on 22 Aug 2023
Hello Barbab,
I understand that you are encountering difficulty while trying to load and store large-sized matrices.
In this case, I would recommend you to use ‘Tall Arrays’ in MATLAB. Tall Arrays are used to work with out-of-memory data that is backed by a datastore. Datastores enable you to work with large data sets in small blocks that individually fit in memory, instead of loading the entire data set into memory at once.
Please refer the following MathWorks Documentation link on ‘Tall Arrays’ for better understanding on how to use them for large matrices: https://www.mathworks.com/help/matlab/tall-arrays.html
Furthermore, for additional information on ‘Tall Arrays’ in MATLAB, consider going through the reference links below:
1. Documentation for Tall Arrays: https://in.mathworks.com/help/matlab/import_export/tall-arrays.html
2. Using Tall Arrays: https://in.mathworks.com/help/matlab/tall-arrays.html
3. Deferred Evaluation of Tall Arrays: https://in.mathworks.com/help/matlab/import_export/deferred-evaluation-of-tall-arrays.
I hope this helps!

Categories

Find more on Tall Arrays in Help Center and File Exchange

R2023a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!