how to extract data from the fixed-width-field format using fscanf or textscan
11 views (last 30 days)
Show older comments
Dmitrii Semikin
on 16 Jun 2019
Commented: Dmitrii Semikin
on 18 Jun 2019
I am trying to parase large file, containing among others numerical data fixed in fixed-width-fields format without whitespaces (or any other separators) between the fields. Consider the example:
1223244
55 6 788
1122 3 4
In this text each line contains four numbers each - two digits wide. Expected result after parsing this flie is:
1 22 33 44
55 6 7 88
11 22 3 4
The prerformance is important for me, as the files I read may be hunreds of MBs large. From my experiments I discovered, that fgetl() is drammatically slower, than fscanf (I did not check the performance, but I expect, that textscan should also be significantly faster.
The problem is that I cannot find the way to parse data like this with fscanf or textscan. Could someone tell, if it is possible at all? If not, is there any other way to parse such a text file with good performance?
P.S. format of actual string is, of course, somewhat more complicated. Example string is:
1101332.18685714711829.064533733772535.874264373485 0 0
this string should be parsed into the numbers with the following widths: 8, 16, 16, 16, 8, 8, which in this particular case would result in the following numbers:
110
1332.18685714711
829.064533733772
535.874264373485
0
0
P.S.2: Differnence between the performance of fgetl and fscanf: On my laptop for the file, which contains 35597 lines the following code with fscanf complete in 0.253614 seconds
function [data] = test_fscanf_nodes_only_01()
file_name = 'myfile.txt';
file_id = fopen (file_name, 'rt');
cleanup_obj = onCleanup(@() fclose(file_id));
data = fscanf(file_id, '%8d%16f%16f%16f%8f%8f', [6, Inf]);
end
while the following code with fgetl needs 6.343209 seconds to complete, even though it does much less
function [data] = test_fscanf_nodes_only_02()
file_name = 'myfile.txt';
file_id = fopen (file_name, 'rt');
cleanup_obj = onCleanup(@() fclose(file_id));
lines_count = 0;
while ~feof(file_id)
current_line = fgetl(file_id);
lines_count = lines_count + 1;
end
data = 1;
fprintf('Lines count: %d', lines_count);
end
The main problem with the first snippet is that it returns wrong result (because for fscanf the width of the field counts from the first digit it finds and not from the current position of the file pointer (which means, that the leading whitespaces are not counted as field width).
EDIT: The dimension in fscanf is changed from [5, Inf] to [6, Inf] according to Jan's comment.
8 Comments
Walter Roberson
on 17 Jun 2019
textscan() is not the same as fscanf(): with textscan it is possible to change the Whitespace and Delimiter properties in a way that does not skip leading whitespace. because of the way that the numeric fields parse numbers, a leading whitespace in a numeric field in such an arrangement would be counted as an invalid character, so this turns out to only be useful for fixed-width character fields.
Accepted Answer
Walter Roberson
on 17 Jun 2019
fileread into a character vector. regexprep to insert a space before every field. textscan the character vector.
4 Comments
Walter Roberson
on 17 Jun 2019
newS = regexprep(S, '^(.{8})(.{16})(.{16})(.{16})(.{8})(.{8})', '$1 $2 $3 $4 $5 $6', 'lineanchors');
More Answers (1)
Stephen23
on 17 Jun 2019
A simple solution based on regeprep and sscanf. Before the file-reading loop:
vec = [8,16,16,16,8,8];
rgx = sprintf('(.{%d})',vec);
rpl = sprintf('$%d,',1:numel(vec));
In the file-reading loop:
str = fileread('temp2.txt');
str = regexprep(str,rgx,rpl,'dotexceptnewline');
mat = sscanf(str,'%f,',[numel(vec),Inf]).'
Giving (the test file is attached):
mat =
110.00000 1332.18686 829.06453 535.87426 0.00000 0.00000
220.00000 2332.28686 829.06453 535.87426 0.00000 0.00000
330.00000 3332.38686 829.06453 535.87426 0.00000 0.00000
440.00000 4332.48686 829.06453 535.87426 0.00000 0.00000
550.00000 5332.58686 829.06453 535.87426 0.00000 0.00000
See Also
Categories
Find more on String Parsing in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!