readtimetable from a java stream instead of file

1 view (last 30 days)
Marc Vaillant
Marc Vaillant on 18 Feb 2020
Commented: Marc Vaillant on 24 Apr 2020
TL;DR
Why the heck doesn't matlab provide overloads for IO functions, e.g. readtimetable, that take a low level stream? In particular, is there a way to have readtimetable read from a lowel level file stream instead of a file?
Details
Timetables and tables are nice datastructures but there is a significant roadblock to their interoperability with other languages. Their binary serialized form in .mat files is proprietary, so no one knows how to read/write them from other languages. You can read/write them as ascii--using say csv as a data exchange format--via readtimetable/writetimetable, but then their size blows up by a factor of 3 or 4. When you have timetables that would be 500 MB as mat files, and all of a sudden they become 2gb, that's a problem. You can compress the csvs, but alas, there's no real way to read compressed files on the fly using standard parsing functions like readtimetable. There's this which consoildates the information out there on trying read zip files on the fly without uncompressing and writing to a file first, which basicaly gets you a java file stream that can read from a zip file directly, but you can't pass it to readtimetable. The only option would be to spin your own csv to timetable parser that operates on a java file stream, but that seems silly given that the functionality is already there in readtimetable. Is there another option?
  2 Comments
Marc Vaillant
Marc Vaillant on 24 Apr 2020
Thanks Peter. I haven't tried using parquet, but that seems like a reasonable compact data exchange format. I will take a look.
Thanks again,
Marc

Sign in to comment.

Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!