MATLAB Answers

How to call data from files named with dynamic index

3 views (last 30 days)
I know that a similar question has been turned down many times as we don't EVER want to use dynamic vaiable names in programming. The fact is that I'm trying to make my life easier when plotting measurement results. The files are named after the sample as in sample001.csv or SampleName_<Date>.txt (and I cannot change the export or naming format). All I want to do is:
% for each of these text files, plot a set column as x and another as y
figure
hold on
% k is a high number and the number of files to import for plotting
for n = 1:k
plot(sample<n>(:,14),sample<n>(:,17))
end
I can think of two ways to do this:
  1. Break the Rules of Matlab and somehow implement the dynamic variable name anyway (since the files per default imprt with the file name as variable name). I've tried some things with num2str or putting a char or string variable into plot but couldn't get it to work
  2. Get the files to import to a multidimensional (in my case 3D) array or table from which I can call the data in a conventional method
I presume the second will be the experts' choice, therefore, how can I
  • select a great number of files for importing (best from a Windows Explorer "open a file" dialog)
  • forget the header in the files (it would be nice to keep the column names but it makes accessing the rest of the data so much more complicated so I'd rather scrap the header and import the actual data as numeric matrix)
  • make sure the decimal separator is correct
  • import them in a way that I can access the data of "sample345.txt" as array(:,:,345) or something like that
  • get the whole thing to be robust so that it works if the data has different (arbitrary) height (and possibly width) in each file, contains strings as well as numbers or files are missing
Any help much appreciated.
______
PS: I've come up with something similar already but the problem is that it only works for numeric data with no header etc. and overwrites the variable every time. Sorry for the syntax, I'm not a programmer
for k=(1:3:72)
textFileName = ['sample_' num2str(k) '.txt'];
textData = readmatrix(textFileName);
sz = size(textData);
x = sz(1);
dehnung = textData(:,4);
spannung = textData(:,5);
displacement = textData(:,10);
length = textData(1,17);
z=displacement(1);
for i = 1:x
displacement(i) = displacement(i) - z;
dehnung(i) = displacement(i)/(length(1)*1000)*100;
end
plot(dehnung,spannung)
hold on
end

  1 Comment

Peter Perkins
Peter Perkins on 19 Nov 2020
What about (3) read each file into a field of a scalar struct named for the file?

Sign in to comment.

Accepted Answer

Stephen Cobeldick
Stephen Cobeldick on 19 Nov 2020
Edited: Stephen Cobeldick on 19 Nov 2020
"select a great number of files for importing (best from a Windows Explorer "open a file" dialog)"
If you really have a "great number" of files your user is going to dislike this way of selecting them. Most likely simply selecting the folder and using dir with a filename filter would be simpler and less fiddly to use. But each to their own...
[C,P] = uigetfile(..,'MultiSelect','on');
N = numel(C);
D = cell(1,N);
for k = 1:N
F = fullfile(P,C{k});
M = .. whatever file importing function you want to use
.. whatever processing you want to do
D{k} = M;
end
You can access the filenames and filedata using basic indexing, e.g. for the third file:
C{3} % filename
D{3} % imported data
A simple alternative is to import the data into the same non-scalar structure as dir returns, e.g.:
P = 'absolute/relative path to the folder where the files are saved';
S = dir(fullfile(P,'sample_*.txt'));
for k = 1:Nnumel(S)
F = fullfile(P,S(k).name);
M = .. whatever file importing function you want to use
.. whatever processing you want to do
S(k).data = M;
end

  2 Comments

H Mueller
H Mueller on 19 Nov 2020
Great, the wavy parentheses did the trick. I find it very confusing when to use which kind. Thank you!
Here is the solution I derived from your remarks. It outputs an array that contains the data and a string array with the corresponding sample names. Calculations, plotting etc. can be done at the same time.
% Select files to import
[files,path] = uigetfile('D:\filepath\*.*','MultiSelect', 'on');
% Separate file name from extension (optional, could do with "files")
i = 1;
for fn = files
[pathx,namex,extx] = fileparts(append(path,"\",fn));
sample(i) = cellstr(namex);
i = i+1;
end
% Import data to struct
for n = 1:length(files)
data{n} = readtable([path,files{n}]);
end
% Do things with the data (in my case plot)
for k = 1:length(files)
figure
hold on
plot(data{k}{:,19},data{k}{:,28:end})
end
clearvars -except sample data
Stephen Cobeldick
Stephen Cobeldick on 20 Nov 2020
Do NOT use the variable path as this will shadow the important inbuilt path function.
You don't need to add the filepath just to remove it again with fileparts.
I recommend looping over indices rather than looping over data values, it is usually simpler.
I strongly recommend that you preallocate the cell arrays before the loops.
I strongly recommend that you use fullfile (the correct tool) instead of concatenating strings.
[C,P] = uigetfile('D:\filepath\*.*','MultiSelect', 'on');
N = numel(C);
D = cell(N,1); % preallocate
[~,S] = fileparts(C); % sample
% Import data to cell array:
for k = 1:N
F = fullfile(P,C{k});
D{k} = readtable(F);
end

Sign in to comment.

More Answers (1)

Steven Lord
Steven Lord on 17 Nov 2020
Edited: Steven Lord on 17 Nov 2020
%{
% for each of these text files, plot a set column as x and another as y
figure
hold on
% k is a high number and the number of files to import for plotting
for n = 1:k
plot(sample<n>(:,14),sample<n>(:,17))
end
%}
Where are you importing the files? Before you create the figure (where the first comment is located) or inside the for loop (omitted to make the example simpler?)
Do you need to store all the data in memory at once?
If you don't need all the data in memory at once, you don't need to store the data in different named variables. In this example I'm reusing the variable y in the loop.
% Sample data
x = 0:360;
% Set up figure
figure
axis([0 360 -1 1]);
hold on
% Y will simulate the data loaded from your files
for k = 1:5
y = sind(k*x);
h = plot(x, y);
end
shg
If you do need all the data at once, and each data set should be associated with its filename, consider using a struct or something similar.
x = 0:360;
axis([0 360 -1 1]);
hold on
% Create struct to hold data
allData = struct('multiplier', {}, 'yData', {}, 'handle', {});
for k = 1:5
y = sind(k*x);
h = plot(x, y);
allData(k).multiplier = k;
allData(k).yData = y;
allData(k).handle = h;
end
disp(allData(4))
multiplier: 4 yData: [1×361 double] handle: [1×1 Line]

  5 Comments

Show 2 older comments
Peter Perkins
Peter Perkins on 19 Nov 2020
Try a using scalar struct:
filenames = ["sample001" "sample002"];
for fn = filenames
% readtable(fn+".csv",...)
t = array2table(rand(5,2));
s.(fn) = t;
end
for i = filenames
% do something to each table
size(s.(fn))
end
You might ask, "how is this different than putting each table in the workspace?", and the answer is that you never need to eval anything. s.(fn) gets you each table. You can even iterate over them using structfun.
Steven Lord
Steven Lord on 19 Nov 2020
You may need to use matlab.lang.makeValidName or perform some other processing to create the struct field name if the filenames are not valid MATLAB identifiers.
name = ["sample 001"; "grades (MA101 Fall 2020)"];
matlab.lang.makeValidName(name)
ans = 2×1 string array
"sample001" "grades_MA101Fall2020_"
H Mueller
H Mueller on 19 Nov 2020
I don't understand how to feed the data into the struct. Somehow the fields need to be named beforehand and I can't name them after every entry of a string array? Here is what I have so far:
% Select files to import
[files,path] = uigetfile('D:\Path\*.*','MultiSelect', 'on');
% Separate file name from extension (is there a simpler way?)
i = 1;
for fn = files
[pathx,namex,extx] = fileparts(append(path,"\",fn));
file(i) = cellstr(namex);
i = i+1;
end
% Import Data to struct
AllData = struct
for n = (1:length(file))
import = readtable([path,files{n}]);
SampleName = file{n};
ColumnHeaders = import.Properties.VariableNames;
RowNames = (import{:,1});
Data = table2array(import(:,2:end));
AllData.(SampleName) = struct(ColumnHeaders{:}; RowNames{:}, Data(:)); % please fix this line
end
(Why is it an such an absolute mess if I put import into the struct? That would be way easier, but it puts every property associated with the table into a separate field.)
[EDIT]: Stephen proposed a simple way to do precisely that. Problem solved. Thanks for your answers!

Sign in to comment.

Products


Release

R2020a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!