fprintf for ~million row string matrix

1 view (last 30 days)
Erik Johannes Loo
Erik Johannes Loo on 26 Jul 2022
Edited: Jon on 26 Jul 2022
Hi,
I'm trying to write data to csv files, but using fprintf (or the way I am using it) seems to be extremely slow
The data is formatted in columns containing fields such as: {ID (string), date (string), numeric data, flag (string), ...}.
I get the data as a cell array, with each cell having a column of data.
What I've done is cast everything into a string and take the transpose of the data (it seems MATLAB does things in a column-major order).
Also note I've added a bunch of commas without data inbetween. That's the expected format of the data, so that has to stay.
fid = fopen(fullfile(output, replace(file.name, '.dat', '.csv')), 'w');
fprintf(fid, [repmat('%s,', 1, nCols), repmat(',', 1, 86-nCols), '\n'], T);
fclose(fid);
It looks as if saving a single file takes several minutes (haven't managed to save even one file yet - and I need to do this for 365 files).
Is there a way to speed up this process?
Kind Regards,
Erik

Answers (1)

Jon
Jon on 26 Jul 2022
You could try writecell and see if that is more efficient.
  2 Comments
Jon
Jon on 26 Jul 2022
Edited: Jon on 26 Jul 2022
Or maybe the computer with R2020b was just faster? Also I have found that MATLAB is sometimes very slow writing to network drives. If possible write it to a file on your local hard drive.

Sign in to comment.

Products


Release

R2018b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!