Write and read numeric data line by line
6 views (last 30 days)
Show older comments
Here is a working example of what I have in mind:
N=10000; M=10000;
x=rand(1,M);
dlmwrite('file',x,'delimiter',' ')
for k=1:N-1
x=rand(1,M);
%In the real application x is the result of some calculations
dlmwrite('file',x,'delimiter',' ','-append')
end
toc
tic
for k=1:N
u=dlmread('file',' ',[k-1 0 k-1 M-1]);
%calculations using vector u
end
toc
The writing part is fine (about 5 seconds on my PC), but the reading is
awfully slow - about 43 seconds. My guess is that it would help if reading
is resumed automatically after the last line read. The command "fgetl"
appears to do this, but is designed for character, rather than numeric,
variables.
Any other suggestions?
(The problem entails generating an array
of size 10^5 X 10^5, which is used by another program.)
0 Comments
Accepted Answer
Michael Soskind
on 5 May 2020
Hi Chris,
These are quite a bit faster in my experience, and testing with smaller example arrays, give a much better result.
I compare the two in the following code:
%% Using fprintf and fscanf, which seem order of magnitude faster
N=100; M=100;
x=rand(1,M);
dlmwrite('file',x,'delimiter',' ')
tic
fid = fopen('file.txt', 'a');
for k=1:N-1
x=rand(1,M);
%In the real application x is the result of some calculations
fprintf(fid,repmat('%f ', 1,M), x);
end
fclose(fid);
toc
tic
fid = fopen('file.txt', 'r');
for k=1:N
u=fscanf(fid,repmat('%f ', 1,M),[1 M]);
%calculations using vector u
end
fclose(fid);
toc
%% Using dlmwrite and dlmread
N=100; M=100;
x=rand(1,M);
dlmwrite('file',x,'delimiter',' ')
tic
for k=1:N-1
x=rand(1,M);
%In the real application x is the result of some calculations
dlmwrite('file',x,'delimiter',' ','-append')
end
toc
tic
for k=1:N
u=dlmread('file',' ',[k-1 0 k-1 M-1]);
%calculations using vector u
end
toc
You can test this with any data size you like, I suppose my computer is not nearly as fast as yours at the file processing.
The append feature of fprintf is particularly useful.
Hopefully that helps,
Michael
2 Comments
Michael Soskind
on 5 May 2020
Sorry, a few notes
- I did not remove the dlmwrite at the beginning of the code there.
- Further, the speed up is quite good for a 1000 element case on my computer.
I have ~2.5 seconds for reading and writing with fprintf, and 5.6 and 60 seconds for writing and reading, respectively with dlmread, using a 1000 element array.
More Answers (0)
See Also
Categories
Find more on Low-Level File I/O in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!