help to read and write to .txt faster

15 views (last 30 days)
bwilley
bwilley on 29 Jun 2018
Commented: Guillaume on 29 Jun 2018
I am modifying some text files that contain data from a bunch of sensors. I frequently want to change the data in the text file so I have written some code that I used to change or modify values. I am working with text files that are around 800,000 rows and 10 columns. File size can be up to 80 MB. This code reads a text file, finds an index of where the data starts, then reads each row, finds the values of interest then changes them. I am relatively new to matlab and programming in general. Can anyone suggest some ways to make this code run faster? Currently takes 3-4 minutes to run on my new Mac desktop. Thanks in advance.
if true
%
%read from file
filename='test_20180605_131311.txt';
filename2='test.txt';
fid = fopen(filename,'rt');
tmp_all = textscan(fid,'%s','Delimiter','\n');
fclose(fid);
b=strcmp(tmp_all{1}, '###'); %search each row for '###'
ind=find(b, 1, 'first'); %index for '###'
ind=ind+2; %index to data
%%new load sensor values and column index
lc1='201.8'; lc1_col=5;
lc2='197.0'; lc2_col=6;
lc3='228.0'; lc3_col=7;
lc4='232.5'; lc4_col=9;
%modify lvdt calibration
lvdt1_cal=675.7/651.863; lvdt1_col=4;
lvdt2_cal=505.3/657.883; lvdt2_col=8;
% loop to change data, separates values row by row by ','
% comment unused sections for speed
for i=ind:length(b)
a=strsplit(tmp_all{1}{i},',');
a{lc1_col}=lc1; %change load cell values
a{lc2_col}=lc2; %comment if not used
a{lc3_col}=lc3;
a{lc4_col}=lc4;
%change lvdt cal factors
a{lvdt1_col}=num2str(str2num(a{lvdt1_col})*lvdt1_cal);
a{lvdt2_col}=num2str(str2num(a{lvdt2_col})*lvdt2_cal);
b=strjoin(a,','); %join each element of a row
tmp_all{1}{i}=b;
cyc=cyc+1;
end
%%write new values to new text file named "test.txt"
filename2='test.txt';
fid = fopen(filename2,'w');
x=length(tmp_all{1});
for i=1:x
fprintf(fid,'%s\n',tmp_all{1}{i});
end
fclose(fid);
end
  1 Comment
Guillaume
Guillaume on 29 Jun 2018
Can we have an example text file?
I believe that you can make the whole thing much faster by fileread'ing the file in one go, finding the first '###' with a regexp then doing the replacement on all subsequent lines with a dynamic regexprep. There would be no loop involved.

Sign in to comment.

Answers (1)

Ameer Hamza
Ameer Hamza on 29 Jun 2018
For faster writes, refer to Jan's answer to this question: https://www.mathworks.com/matlabcentral/answers/83870-fastest-way-to-write-data-to-a-text-file-fprintf. The idea being using 'W' instead of 'w' to avoid automatic flushing of data. For reading, there might not be much which you can do. Also if you have to frequently read, modify and save the text file then consider saving the data in a .mat file. It saves data in binary format (mostly less number of bytes for each number). They will be smaller in size, easy to load, modify and save.

Categories

Find more on Data Import and Export in Help Center and File Exchange

Products

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!