num2cell out of memory

I have a huge dataset that when loaded in the workbench it turns into a 67x20838735 double. Then problem is that when I try to convert it into cell array I get the following message: Out of memory. Type "help memory" for your options.

inputs = num2cell(inputs)

I trien to find a way to solve this problem with loops, but I don't find any way to succeed.

Answer 1

[Uerm]

Hi,

You are working with quite a large variable. I have tried the following

n = rand(67,20838735);

When running whos it says that the variable n uses a bit more than 11 GB of memory. That is more than what most of the normal computers/laptops have :) I think that your computer simply does not have enough memory (RAM). How much RAM does your computer have?

There are some things that you can try.

1) Before converting to cell arrays, you can try to convert the variable from double to single:

n = single(n);

This will make the variable n use half of the memory it did when it was a double.

2) Normally, MATLAB limits the maximum array size to a percentage of the RAM of the computer. You can get rid of this limit by this path:

Click the Home tab --> Click Preferences --> Choose Workspace in the left menu --> On the bottom of the page there is a box saying "Limit the maximum array size to a percentage of RAM". Make sure that this box is empty --> Click Apply and OK.

I suggest that you start with step 2. If that does not work, supply with step 1.

If none of the above works, you can make MATLAB use virtual memory. You can read more about it here:

https://se.mathworks.com/help/matlab/matlab_prog/resolving-out-of-memory-errors.html

The tradeoff of using virtual memory is the computation speed!

[Jorge Aarón Morán Holguín]

Hello,

My computes has a total 96GB of memory.

>> memory
Maximum possible array:       45018 MB (4.720e+10 bytes) *
Memory available for all arrays:       45018 MB (4.720e+10 bytes) *
Memory used by MATLAB:       93519 MB (9.806e+10 bytes)
Physical Memory (RAM):       96922 MB (1.016e+11 bytes)

*  Limited by System Memory (physical + swap file) available.

I have tried both options combined and the problem remains.

Answer 2

[per isakson]

Why do you want to convert the numerical matrix to a cell array? It requires an enormous amount of memory for overhead.

>> clearvars
>> num = (1:12);
>> cac = num2cell( num );
>> whos num cac
  Name      Size            Bytes  Class     Attributes
  cac       1x12             1440  cell                
  num       1x12               96  double  

I think the resulting cell array will need (1440/96)*11 GB. That's twice as much as your ram.

[Jorge Aarón Morán Holguín]

I am trying to train a neural network using the train() function with quite a big dataset and when loaded it turns into 2 numerical matrixes, X (inputs) and Y (target).

[per isakson]

But that doesn't answer the question why you try to convert a numerical array into a cell array with the statement

inputs = num2cell(inputs)

[Sharif Khalil]

Hi Jorge,

Did you get an answer to your problem?