Storing an unknown number of variables on disk individually

Dear all, I am currently working on a signal generator that generates timeseries for input in simulink.

My proplem is, that I need a "long" signal with an order of 10^12 individual samplepoints. The signal has many components and I will need to store the underlying information needed for generation. This will not fit into my ram for simulation.

My current approach is to generate the signal in smaller chunks, that would fit into the RAM.

So I would generate a part of the signal and store it on disk, generate the next part, store it etc. How could I manage the storing efficiently? Like how do I name the signals/files, where I do not know how many chunks I will need to store?

Sorry if the question may seem trivial for vetarans here, but I have no experience with such resource intensive computations.

Answer 1

[Steven Lord]

See if some of the tools and techniques described in the Large Files and Big Data chapter in the documentation will meet your needs. If you create the files in sequence then perhaps you could create a datastore and use that as the basis for a tall array on which you can perform your calculations. I suspect either a tabularTextDatastore or a spreadsheetDatastore may be appropriate.