Is it possible to save partial arrays without compression? matfile() forces compression during save.

Hi, I would like to incrementally write arrays to a .mat file without compression. I know how to save files without compression using save(...,'-nocompression'). That won't work for me, because I often don't have the RAM to hold the variable in memory before saving. I also know how to save files incrementally with matfile(), but that won't work here because it forces compression. I've tried tricking matfile() into not using compression by initializing the array with an uncompressed variable either with save() or with h5create(), but neither worked. Perhaps I could use h5write(), but it seems so low level--it doesn't support many data types such as logical arrays and char arrays. I could probably write my own class to do this, but it is a bigger job. Is there a better way to write partial arrays to .mat or is there a better file type all together? Thanks for your advice.

You might ask about my application and why I need this feature: I work with large files (50-500 GB) that do not benefit much from compression. With a modern nvme SSD, large portions of uncompressed data load in just a few seconds, but compressed data takes several minutes to load due to singlecore CPU bottlenecks. If multiple cores were used, then compression might be viable.

[Jan]

What about using binary files and memmapfile?

[dpb]

@Jan +1

Answer 1

[Jan]

What about using a binary file together with memmapfile?

[Thomas Richner]

Thank you Jan! Count me as a Jan fan. I didn't know about memmapfile. With a little work that should do nicely.