how to store large data during the execution of a program without getting out of memory
3 views (last 30 days)
Show older comments
asim nadeem
on 14 Oct 2019
Commented: asim nadeem
on 15 Oct 2019
I am working on partitions. while i run program that generates partition it get out of memory. What i want to do is to store these partitions in a data store and use tall arrays to work on chunks to filter required partitions. partitions are stored in cell arrays. Thanks
4 Comments
Walter Roberson
on 14 Oct 2019
If I were to ask you for partition #107 would you be able to calculate it without calculating all the ones before it in the same call?
Accepted Answer
Walter Roberson
on 15 Oct 2019
You will need to go through calculating the all partitions and writing them to a file, but keeping only a limited working set in memory as you do so. After you have written them all to file, you can create a datastore from the data file and create a tall array from that and start working on it.
There are some methods of building partitions that work recursively, creating partitions of size 1, extending to size 2, then to size 3, and so on, with (for example) all the partitions of size 19 needed to calculate the partitions of size 20. You would want to create as large a partition in memory as your memory could afford before writing that to a file and proceeding through tall array methods to addition additional columns, creating a file as you go, going back and making that file a datastore, and so on. It would probably be a bit of a nuisance, but It should be possible... I think.
Matters would be easier to arrange if you had an algorithm that could translate from an index to a particular partition; perhaps Knuth volume 4A; https://math.stackexchange.com/questions/222780/enumeration-of-set-partitions
More Answers (0)
See Also
Categories
Find more on Large Files and Big Data in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!