Import files with identical names from different directories (no error messages)

I have multiple files with the same names in different subfolders. All files are called runoff.txt. The subfolders are called results1, results2, results3 and so on, so they differ only by a consecutive number.

For the import of txt files with the same names from different subfolders, I have tried an approach to solve this issue like mentioned on:

http://etools.fernuni.ch/matlab/matlab4/de/html/userinteraction_readdir.html

My script looks like:

pathname = 'C:\Users\heute\results\model_standalone\';

Mdir = dir(pathname);
nbentries = size(Mdir, 1);
Mfiles = [];
for entry_i = 1:nbentries
  if Mdir(entry_i).isdir == false

      filename = Mdir(entry_i).name;
      if filename(1) ~= '.'
        [p, n, ext] = fileparts(filename);
        if strcmpi(ext, '.txt')
          Mfiles = strvcat(Mfiles, filename);
        end
      end
    end
  end
  nbfiles = size(Mfiles, 1);

for iFile = 1:length(Mfiles);
    [Date,QSfirn,QSice,QSrock,QSsnow,QSsoil,Qs]=textread(Mfiles(iFile),'%s %f %f %f %f %f %f',-1,'headerlines',1);
end

The program runs without error messages but the txt files are not imported.

For example that the folder model_standalone contains 2 subfolders with the names results1, results2 and within these folders a txt file with the name runoff.txt the following output is returned:

• Mdir: 4x1 struct
• Mfiles: []
• entry_i: 1x1 double
• i: 1X1 double
• nbentries: 1x1 double
• nbfiles: 1x1 double with 0
• pathname: char_

Why aren't the individual txt files recognized?

How can you fix the problem that all txt files from the subfolders are read (perhaps in a different way)?

-----

The same question has already been asked on this page:

http://www.gomatlab.de/einlesen-von-txt-dateien-aus-mehreren-ordner-und-div-perioden-t43788.html

but without any problem-solving answer.

----

For any helpful answer I would be deeply grateful, because this problem prevents all further data evaluation processes. (I'm using Matlab R2012a).

Answer 1

[Stephen23]

Just because something was written by someone at a university does not mean that it is well written. That code is pointlessly complicated, and has several strange "features":

• using dir to get all directory contents, and then later using if-s to pick non-folders and filenames with .txt extension. What a pointlessly complicated piece of code: you should not learn from this. Much simpler: call dir with the correct match string, including wildcard and file extension.
• no preallocation on the output array Mfiles.
• uses obsolete textread, which the documentation clearly recommends against using.

In any case, that code does not actually do what you ask for: your question you state "I have multiple files with the same names in different subfolders. All files are called runoff.txt. The subfolders are called results1, results2, results3 and so on." But that page states (translated) that it gets all files from one folder. So why use that code, which is pointlessly complicated and does not do what you want anyway?

All beginners would be much better off learning MATLAB from the MATLAB documentation, rather than relying on code written by some random academic who still thinks that cell arrays are very daring, or by some teenager who uploads a youtube blog on some neat tricks that they "discovered". The documentation describes the two basic ways of processing a sequence of files or folders:

https://www.mathworks.com/help/matlab/import_export/process-a-sequence-of-files.html

and also on this forum:

https://www.mathworks.com/matlabcentral/answers/57446-faq-how-can-i-process-a-sequence-of-files

In particular this example:

"To read all files that match *.jpg with imread:"

jpegFiles = dir('*.jpg'); 
numfiles = length(jpegFiles);
mydata = cell(1, numfiles);

for k = 1:numfiles 
    mydata{k} = imread(jpegFiles(k).name); 
end

You can easily adapt this to read the same file from multiple directories, perhaps something like this (untested):

F = 'name.txt'; % name of the file
D = '.'; % absolute or relative path of base directory
S = dir(fullfile(D,'results*'));
X = [S.isdir] & ~ismember({S.name},{'.','..'});
N = {S(X).name};
C = cell(size(N));
for k = 1:numel(N)
    T = fullfile(D,N{k},F);
    C{k} = textscan(T,...); % not textread!
end

If you have sequentially numbered files then you might also be interested in processing them in numeric order, in which case you might like to download my FEX submission natsortfiles:

http://www.mathworks.com/matlabcentral/fileexchange/47434-natural-order-filename-sort

and then you can use it on this line:

N = natsortfiles({S(X).name});

[Glazio]

@Stephen Cobeldick: Thanks for the comprehensive help.

Your solution works so far except, that the cell isn't filled.

I have run your code with 2 example files. The result looks like:

C: 1x2 cell

C{1,1}: 1x7 cell

C{1,1}{1,1}...: empty

C{1,2}: 1x7 cell

If everything would run correctly, the cells must have the contents of the txt files.

What could be the problem?

----------------------

I've used the following textscan-code:

T = fullfile(D,N{k},F);
C{k} = textscan(T,'%s %f %f %f %f %f %f','headerLines',1);

The input-files (txt-files) have the following format:

%Datetime r_firn r_ice r_rock r_snow r_Soil runoff

01.10.2013-00:00 -0.0000 -0.0000 -0.0000 -0.0000 0.1500 0.1500

[Stephen23]

"What could be the problem?"

Probably the textscan call is not correct. If you want help with the file importing function then please upload a sample file by clicking the paperclip button. Descriptions are no alternative to a real file.

[Glazio]

@Stephen Cobeldick: There is a sample of 'runoff.txt', with one header line and 7 columns.

[Stephen23]

This code imports that file correctly:

fmt = ['%s',repmat('%f',1,6)];
opt = {'HeaderLines',1,'CollectOutput',true};
fid = fopen('runoff.txt','rt');
Z = textscan(fid,fmt,opt{:});
fclose(fid);
D = Z{1}; % timestamp
M = Z{2}; % data

And checking:

>> size(D)
ans =
       26304           1
>> size(M)
ans =
       26304           6

tested on this file:

[Glazio]

@Stephen Cobeldick: Thanks, it reads one file correctly.

When trying to read two files, the cells are filled, but with the same content.

Z{} looks like the following:

Z{} 1x2 cells

Z{1,1} 1x2 cells: the first cell contains the date/time and the second cell the remaining 6 columns.

The actual problem is that Z{1,2} contains the same values within this 6 columns as the first file.

Where could the problem lie?

I've changed the following code line:

Z = textscan(fid,fmt,opt{:});

to:

Z{k} = textscan(fid,fmt,opt{:});

to write more files into a Cell-array.

[Stephen23]

In my answer I preallocated a cell array C for the data. You should put the data in there, just like I did in my answer. Why do you think I did that? (hint: preallocation makes code more efficient). This works:

C{k} = M

Do not expand Z like that: that would totally defeat the purpose of preallocating the cell array C.

Here is the whole thing anyway:

fmt = ['%s',repmat('%f',1,6)];
opt = {'HeaderLines',1,'CollectOutput',true};
F = 'name.txt'; % name of the file
D = '.'; % absolute or relative path of base directory
S = dir(fullfile(D,'results*'));
X = [S.isdir] & ~ismember({S.name},{'.','..'});
N = {S(X).name};
C = cell(size(N));
for k = 1:numel(N)
    T = fullfile(D,N{k},F);
    fid = fopen(T,'rt');
    Z = textscan(fid,fmt,opt{:});
    fclose(fid);
    C{k} = Z{2}; % data
end

[Glazio]

@Stephen Cobeldick: My whole code looks like:

F = 'runoff.txt'; % name of the file
D = 'C:\Users\heute\results\model_standalone\'; % absolute or relative path of base directory
S = dir(fullfile(D,'results*'));
X = [S.isdir] & ~ismember({S.name},{'.','..'});
N = {S(X).name};
C = cell(size(N));
for k = 1:numel(N)

      fmt = ['%s',repmat('%f',1,6)];
      opt = {'HeaderLines',1,'CollectOutput',true};
      fid = fopen('runoff.txt','rt');
      Z = textscan(fid,fmt,opt{:});
      fclose(fid);

      D = Z{1}; % timestamp
      M = Z{2}; % data
      C{k} = M;
end

The problem remains the same, both cells contain the same values. It seems that the first file is overwritten by the second file. Is there a mistake in this code or have I interpreted something wrong?

[Glazio]

@Stephen Cobeldick: Thanks, now it seems to work!

[Stephen23]

"Is there a mistake in this code or have I interpreted something wrong?"

Your code always reads the same file because you do not provide the path when opening the file. My code (where D is the base directory, N{k} is one of the subdirectories, and F is the filename) was:

T = fullfile(D,N{k},F);
fid = fopen(T,'rt');

Whereas your code only looks in the current path:

   fid = fopen('runoff.txt','rt');

Basically you did not use the part of my code which actually answers your question of how to read a file (with the same name) from multiple directories. So by removing that part of my code that actually can open files in multiple directories, your code does not open files in multiple directories.

If that is your code then I have no idea how you managed to read two files with this (as you claimed earlier), as no operating system I know of allows two files with the same name in one folder.

I would recommend that you use F rather than typing the name again.

[Amanda]

I've been working on a very similar problem and using this code am trying to import multiple files with the same name from different directories. After 2 days of trying different code I need help.

It's almost working for me, however the data C{k} = M returns NaN into a cell array of the appropriate size. C{k} = D returns a cell array but the data is entered into a single column separated by commas. How do I get the data into the array so that the numbers are entered into their own columns (should be 6) instead of separated by commas in a single column? Essentially, get the numbers to replace the NaN? I hope that made sense.

[Stephen23]

@Amanda: did you make the required changes to textscan so that it can read your specific file format? (e.g. format string, delimiter, etc.)

Or if textscan is not suitable, did you select and use an appropriate file importing function? (e.g. readmatrix, readtable, etc.)

https://www.mathworks.com/help/matlab/data-import-and-export.html

[Amanda]

Hi @Stephen Cobeldick, I thought I did but maybe not! When I try adding delimited, it also imports differently than I expect. Here is my code and an example file:

file_name = 'trialsummary.txt'; % name of the file
directory = '/Users/amanda/Dropbox/FMoving/Mouse_2140/100-0/'; 
% absolute or relative path of base directory
S = dir(fullfile(directory,'**/*')); %go into session subfolder (i.e. 1-61), then pull .txt file
X = [S.isdir] & ~ismember({S.name},{'.','..'});
N = ({S(X).name}); %session numbers
C = cell(size(N));
fmt = ['%s',repmat('%f',[1,6])];
opt = {'CollectOutput',1};
for k = 1:numel(N)
    T = fullfile(directory,N{k},file_name); 
    %build file from directory folder, then subfolder, then file name
    fid = fopen(T,'rt');
    session_info = textscan(fid,fmt,opt{:});
    fclose(fid);
    C{k} = session_info{1}; % data
end

[Stephen23]

Here are four options for reading that very neatly formatted text file.

1. readtable:

M = readtable('trialsummary.txt')

M = 26×6 table
    Var1    Var2    Var3    Var4     Var5      Var6 
    ____    ____    ____    ____    ______    ______

      1      0       1       0           4    12.802
      2      0       1       0      48.149    60.907
      3      0       2       0      98.858    219.61
      4      0       2       1      233.56    244.31
      5      0       2       0      319.29     331.7
      6      0       1       1      361.88    403.75
      7      3       1       0      440.05     455.2
      8      0       1       0      472.82    476.15
      9      0       1       0      601.41    605.51
     10      0       1       0      650.75    654.12
     11      0       2       0      671.12    690.84
     12      0       2       1      827.02    833.78
     13      0       2       1       917.7    938.23
     14      0       1       0      964.38    968.53
     15      0       1       0      978.17    981.26
     16      3       1       0      995.14      1002

2. readmatrix:

T = readmatrix('trialsummary.txt')

T = 26×6

1.0000 0 1.0000 0 4.0000 12.8025 2.0000 0 1.0000 0 48.1486 60.9070 3.0000 0 2.0000 0 98.8581 219.6091 4.0000 0 2.0000 1.0000 233.5568 244.3127 5.0000 0 2.0000 0 319.2868 331.6952 6.0000 0 1.0000 1.0000 361.8825 403.7528 7.0000 3.0000 1.0000 0 440.0502 455.2032 8.0000 0 1.0000 0 472.8177 476.1478 9.0000 0 1.0000 0 601.4062 605.5133 10.0000 0 1.0000 0 650.7510 654.1190

3. dlmread:

M = dlmread('trialsummary.txt')

M = 26×6

1.0000 0 1.0000 0 4.0000 12.8025 2.0000 0 1.0000 0 48.1486 60.9070 3.0000 0 2.0000 0 98.8581 219.6091 4.0000 0 2.0000 1.0000 233.5568 244.3127 5.0000 0 2.0000 0 319.2868 331.6952 6.0000 0 1.0000 1.0000 361.8825 403.7528 7.0000 3.0000 1.0000 0 440.0502 455.2032 8.0000 0 1.0000 0 472.8177 476.1478 9.0000 0 1.0000 0 601.4062 605.5133 10.0000 0 1.0000 0 650.7510 654.1190

4. textscan:

opt = {'CollectOutput',1, 'Delimiter',','}; % !!! specify the delimiter !!!
fmt = repmat('%f',1,6); % !!! specify the correct number and type of columns !!!
fid = fopen('trialsummary.txt','rt');
tmp = textscan(fid,fmt,opt{:});
fclose(fid);
M = tmp{1}

M = 26×6

1.0000 0 1.0000 0 4.0000 12.8025 2.0000 0 1.0000 0 48.1486 60.9070 3.0000 0 2.0000 0 98.8581 219.6091 4.0000 0 2.0000 1.0000 233.5568 244.3127 5.0000 0 2.0000 0 319.2868 331.6952 6.0000 0 1.0000 1.0000 361.8825 403.7528 7.0000 3.0000 1.0000 0 440.0502 455.2032 8.0000 0 1.0000 0 472.8177 476.1478 9.0000 0 1.0000 0 601.4062 605.5133 10.0000 0 1.0000 0 650.7510 654.1190

Note that you do not need the intermediate variable session_info, you can allocate directly to C like this:

C(k) = textscan(..)
%^ ^ parentheses, not curly braces.

https://www.mathworks.com/help/matlab/matlab_prog/access-data-in-a-cell-array.html