Storing Large data in matlab files

I have a giant loop and each loop produces giant vectors. Instead of storing this vectors in a giant matrix I am thinking about storing each of these vectors in a file at the end of loop. The pseudo code would look as: for i=1:5000 produce giant vector store the giant vector in a file in columns move on to next loop end

Later I want to be able to read these vectors from the file directly.

[Oleg Komarov]

Can you quantify giant please? M by N sizes for instance.

Answer 1

[Oleg Komarov]

Do you get out of memory problems? Otherways you're gonna slow down the computation with I/O operations.

You can also gain memory by reducing the accuracy (using single arrays) or sparse.

Oleg

Answer 2

[Walter Roberson]

It is tricky to write by columns in a a binary file. Not impossible though, if you know how many columns there will be. Reading back data usually proceeds by row.

You can transpose your data and write what were the columns across by rows.

No matter what you do, if you need to retrieve all of the data at one time, you will need to write in the row and column sizes and allocated enough memory to receive the values at the time you read it.