Out of memory issue although there is more RAM space than matlab requires
10 views (last 30 days)
Show older comments
I'm trying to apply Singular Value Decomposition (SVD) on a large matrix (80447x80287) which I get from xyz coordinates by applying some grid function. The SVD simply looks like this [U E V]= svd(Z). I've tried couple of tricks:
1) Java Heap Memory increased to the maximum.
2) MATLAB array size limit unchecked.
3) Paging file size in PC virtual memory increased.
None of them seemed to work out the issue. I'm not a Matlab expert but looking at the memory used and the available memory, memory should not be an issue. I also checked the RAM usage which seems to be around 60%. Can anyone think of a way to tackle this? (PC properties: windows 64bit 96 GB RAM)
Name Size Bytes Class Attributes
A 81023x3 1944552 double
Z 80447x80287 51670786312 double
x 81023x1 648184 double
y 81023x1 648184 double
z 81023x1 648184 double
Maximum possible array: 106862 MB (1.121e+11 bytes) *
Memory available for all arrays: 106862 MB (1.121e+11 bytes) *
Memory used by MATLAB: 52273 MB (5.481e+10 bytes)
Physical Memory (RAM): 98226 MB (1.030e+11 bytes)
- Limited by System Memory (physical + swap file) available.
Jan on 21 Aug 2018
Edited: Jan on 22 Aug 2018
Your matrix Z needs a contiguous block of 51 GB RAM. The SVD of a 80447x80287 matrix will create 3 matrices 80447x80447, a 80447x80287 and a 80287x80287. They require about 3 * 51 GB RAM in addition. This is a total of over 200 GB, such that your 96 GB machine will be exhausted, even if you increase the virtual memory.
Increasing the Java Heap memory is counter-productive, because SVD is not calculated in Java, but the memory reserved for Java is not available for the other calculations.
[EDITED] Think of using the "economy size" output of SVD. Either you have to reduce the size of the problem, or install more RAM. Maybe your problem fits into 256 GB RAM, but a secure amount of RAM is the double size of the used arrays.
I agree, and would add that svd also needs some internal memory. I don't remember how much just now, but I'd say at least another 51 GB.
As I remember from your previous post, this large matrix is the result of interpolating scattered data on a grid. One way of course would be to choose a coarser grid size for your matrix, but I assume there is a reason this is not possible?
I think you wanted to compute the SVD as a way to compress your data, so that only the largest singular values and vectors are of interest. In that case, you could consider using the svds function instead, and computing the largest 10, 100, or 500 singular values of this matrix.
Find more on Code Performance in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!Start Hunting!