How to read every nth line with textscan

I have a very large data file (.dat), and I cannot read it all with textscan. I get 'Out of memory' error.

It is not necessary for me to import all of the data into MATLAB, I would like to read every nth line to get a sample of the data, and reduce the memory required.

I have found some solutions already posted, but none are working in my case. I think because I am using a '.dat' file, and the data is imported as strings (the reason is the decimal used in the data is a comma, so I must read it as a string, then convert comma to decimal, then change to double).

Here is what I am trying:

MATLAB code
  d = fopen('TempMessung.dat');
  while ~feof(d)
      B = textscan(d, '%s %s %s', 1, 'headerlines', 5);

        for i=1:2
            fgets(d);
        end
    end

So this code should skip 5 headerlines, read 1 line from the file into B, then go into the for loop and skip 2 lines, then return to textscan to read one line into B.

Running this code, I get the results that B is a 1x3 cell. B{1} contains just the last row data point from the 1st column B{2} contains just the last row data point from the 2nd column B{3} contains just the last row data point from the 3rd column

Here is a few sample rows from my data:

 pre-formatted
MTS793|
Datenerfassung            Zeit:  2002,1694  Sec  01.03.2012 10:46:40
Zeit  25 KN-R Zähler  In_12_Temp
Sec  cycles  deg_C
0,40966797  0  22,226631
0,60986328  0  22,125919
0,81005859  0  22,260201
1,0102539  0  22,176275
1,2104492  0  22,226631
1,4106445  0  22,276985
1,6108398  0,5  22,134312
1,8110352  0,5  22,260201
2,0112305  0,5  22,209845
2,2114258  0,5  22,125919
2,4116211  0,5  22,251808
2,6118164  0,5  22,134312
2,8120117  1  22,19306
3,012207  1  22,226631
3,2124023  1  22,083956
3,4125977  1  22,235023
3,612793  1  22,184668
3,8129883  1  22,125919
4,0131836  1  22,243416
4,2133789  1,5  22,117527
4,4135742  1,5  22,19306
4,6137695  1,5  22,226631
4,8139648  1,5  22,092348
5,0141602  1,5  22,235023
5,2143555  1,5  22,184668
5,4145508  2  22,134312
5,6147461  2  22,226631
5,8149414  2  22,109135
6,0151367  2  22,201452
6,215332  2  22,226631
6,4155273  2  22,092348

Answer 1

[Guru]

Honestly what you are trying to do is something that doesn't need textscan to read it in. The benefit that textscan provides is the ability to quickly read in the whole file or chunks of the file at once where the formatting of the data remains the same. Low level file I/O routines would probably work better for your case, which textscan does use within itself - as any other file I/O does ultimately.

Try the following code:

m = 1;
d = fopen('TempMessung.dat');
% Skip 5 headerlines
for n=1:5
  tline = fgetl(d);
end
while ischar(tline)
  tline = fgetl(d);
  A(m).data = tline;
  m = m+1;
  % Skip next two lines
  for n=1:2
    tline = fgetl(d);
  end
end
fclose(d);

After you read it all into A, you can do basic string manipulation

% Preallocate B to a cell array
B = cell(length(A),3);
for n = 1:length(A)
    % Replace the , with .
    B(n,:) = strsplit(strrep(A(n).data,',','.'));
end

This will give you the same results as textscan that you wanted with the change of the comma to decimals. If you want this into a double array:

B = cellfun(@str2double,B);

MATLAB is so simple. HTH

[lukacs kuslits]

I have tried what you suggested and I got the exception: "Conversion to cell from char is not possible." This is also the case if I try textscan.

Answer 2

[Matt J]

You can fix it as follows

 ii=0;

...

 B(ii,1:3) = textscan(d, '%s %s %s', 1, 'headerlines', 5);

 ii=ii+1;

but if it is a large file, it will run very slowly.

[Matt J]

If you know the number of rows in your file, you can make it faster by pre-allocating B and using a for-loop instead of a while loop.

[Kyle]

It seems to work, however you're right, it is very slow.

What is the reason for the slow down?

Maybe it is better to textscan the whole file, but how can I do this without getting the out of memory error??

I can split it up into 3 calls to textscan?

[Matt J]

What is the reason for the slow down?

Assuming you pre-allocated B, the slow down is because each call to textscan has overhead associated with it.

I can split it up into 3 calls to textscan?

If you're saying you can hold 1/3 of the file data in RAM at a time, then that is a valid option.

What I might do is edit your file, doing a search/replace on all the commas and change them to '.' decimal points. Then you could read 1 column at a time using 3 calls to DLMREAD.

[Kyle]

I cannot pre-allocate because what I am doing is going through about 10 folders, each folder has this file, and they are all different sizes. Just one file is too large, causing the out of memory error.

So how I will handle this large file will be the same way I handle the smaller files that I can store in RAM. I am not sure if I can hold 1/3, I was just using this as an example.

I'd like to avoid making changes to the individual data file, because I have over 100 files that I will eventually loop through.

Is there a faster way to split the textscan call into n segments (I think 10 should be sufficient), that will save memory and not be as slow as this method above (I believe this method is calling textscan several thousand times). The data file in question is 336MB.

[Matt J]

I've lost track of what the issue is. You can call textscan chunkwise by doing

 C(ithChunk) = textscan(fid, 'format', N);

    ithChunk=ithChunk+1;

where N is the number of lines per chunk. Textscan will resume from the point you left off with every subsequent call, until you close the file.