Alternative to csv and parquet for arrays

I have pretty massive csv files. It's a pain for both transfer and also in terms of read time.

I have been using parquet but this is only for tables and my functions only work with double arrays. So whenever I load the file, I have to use table2array to create the proper variables. This takes some extra time. Still much better than using csv, but I am wondering if there are any light and efficient alternatives to csv for arrays....

[Jan]

The question is not clear. Why do you use a text file to store large data sets? Text files are useful, if they are read and manipulated by human. The conversion of floating point numbers to strings and back to numbers can cause rounding effects. Therefore using a binary format is recommended and much more efficient.

How do you work with parquet in Matlab?

[Walter Roberson]

How often is each file read compared to being written?

[Pelajar UM]

Thanks guys.

@Jan The data is prepared using another program (written in python). It is a mesh data with various values assinged to each one of the element (all numerical values). The MATLAB code is used to visualize this data. The file does not have to be human readable (parquet is not human readable).

Example:

[filename, folder] = uigetfile ({'*parquet'});
if ~ischar(filename); return; end   %user cancel
filename = fullfile(folder, filename);

input = parquetread(filename);
app.UITable2.Data = table2array(input);

nodes = [app.UITable2.Data(:,1),app.UITable2.Data(:,2),app.UITable2.Data(:,3)];
elements = [app.UITable2.Data(:,4),app.UITable2.Data(:,5),app.UITable2.Data(:,6),app.UITable2.Data(:,7)];
elements=rmmissing(elements);
TR = triangulation(elements,nodes); %generating triangular mesh
[F,P] = freeBoundary(TR); %extracting the surface

@Walter Roberson It's written only once (in Python), and every time you open a new session of MATLAB, you have to load the data. So I would reading time is more than important than writing time.

[Walter Roberson]

I suggest writing it as a binary file, with a header indicating the size. You might want to make it compatible with https://www.mathworks.com/help/matlab/ref/multibandread.html

[Pelajar UM]

@Walter Roberson using fwrite / multibandwrite, it's only slightly lighter:

CSV: 25 mb

fwrite/multibandwrite: 23 mb

Parquet: 12 mb

[Walter Roberson]

That hints to me that your data might perhaps only justify single precision but that you are using double precision.

[Sarah Gilmore]

Hi @Pelajar UM,

Do you mind answering a few questons that may help narrow down the issue?

1. Do you know if it's parquetread or table2array that's taking the most time? You can use the performance profiler to determine which lines are causing the issue.
2. How wide is the table in the Parquet file. Is it just 7 columns?
3. Which version of MATLAB are you running?

Thanks,

Sarah

[Pelajar UM]

Hi Sarah

1. parquetread takes 0.11 s and table2array 0.034 s. No issues per se and relatively speaking, not long at all. But it is just an extra step and I was wondering if it could be avoided.
2. This particular dataset is 23 columns wide with ~114,000 rows.
3. R2021a

[Walter Roberson]

If the original data is 23 columns but you only need 7 of them, then you can improve reading speed by writing in binary one column at a time with a header indicating how many rows are present; then by knowing the size of the header and the number of rows, you can fseek() directly to the beginning of any particular column. Or in your case, since you are using the first 7 columns, just ask to fread [nrow 7] after you have positioned past the header, leaving the other 16 columns unread.

[Walter Roberson]

If those file sizes are a problem, then have the Python write each column into a separate binary file and zip the set of files together. Transfer the zip. Unzip at the destination. open and read only the files corresponding to the columns you want to read.

[Pelajar UM]

@Walter Roberson Thanks. I am actually using all 23 columns. That was just a small part of the code.

[Walter Roberson]

ok then write the file in binary and zip it to reduce file size for transfer.

[Pelajar UM]

Zipping does a wonderful job reducing the size (down to 4 mb), but I don't think MATLAB can unzip the file or otherwise read the zipped file, right?

[Walter Roberson]

unzip()