Error in loading a very large matlab file
73 views (last 30 days)
I need to load a 14.6 GB .mat file. However, the maximum possible array that Matlab can produce is 13.8 GB.
Also, the total memory used by Matlab is 480 MB and the physical memory (RAM) is 8 GB. I cannot resuce the size of my matrix as well.
When I am trying to load this file, after an hour, I get the error: 'Error in load 'filename.mat'.
Titus Edelhofer on 23 Jul 2012
does this .mat file contains several variables or one large matrix? In the latter case, I guess, you will have no chance other then using a bigger machine. In the first case: did you try to load the entire file or only some variables? You might try to load e.g. a single variable A by
aStruct = load('yourFBigFile.mat', 'A');
Jan on 23 Jul 2012
Edited: Jan on 23 Jul 2012
Loading a 14GB file into 8GB RAM will be slow, because the disk cache must be used excessively. Out of memory problems are very likely. Therefore a clean and stable advice must include the installation of more RAM. 16GB would be a start, 32GB would be better. If this seems expensive, calculate the number of hours you will wait for your machine to crash. Double this number and remove this from your live time.
per isakson on 23 Jul 2012
Edited: per isakson on 27 Jul 2012
Version 7.3 MAT-files uses an HDF5 based format. I think one can read them piece-wise with the HDF5 functions of Matlab. I made a tiny experiment with a few [double 1x1]. The functions h5info and h5disp provide a start.
--- Answer to [Answer by Christina on 24 Jul 2012 at 11:41] ---
Now, I know a little more and I'll make a new try to answer your question.
- With R2012a 64bit there are no practical limits regarding the address space.
- R2012a can save to several different mat-file formats, e.g. v7 and v7.3. Only v7.3 is based on HDF5.
- AFAIK: By default R2012a saves to v7, not v7.3. There is "no" upper limit to the size of v7-mat-files. The limits mentioned above applies to data items, e.g. double arrays.
- With matfile it is possible to make partial reads from mat-files v7.3. However, the indexing functionality is limited; "complex" indexing is not supported. Thus, read the indexing part of the documentation on matfile carefully.
- With the HDF5-support it is possible to read any piece of data from mat-files v7.3, e.g. a column of a data array that is the value of a field of a nested structure. However, it might be a bit complicated.
Thus, make sure you use
- mat-file v7.3
- 64bit of R2012a and
- that matfile supports the indexing of the variables you need