How to accumulate values ​​from time 1 to the last time

3 views (last 30 days)
I have more than 10000 files and I want to accumulate the values of given variable between all these times and then plot it on a single image. How can I do it? The files are 1000x1000 matrices.
  2 Comments
the cyclist
the cyclist on 23 Sep 2021
Let's just suppose, to keep things easy for understanding, that you only have two files, and each one has a 3x3 matrix. Suppose those two matrices are
M1 = magic(3)
M1 = 3×3
8 1 6 3 5 7 4 9 2
M2 = magic(3) + 17
M2 = 3×3
25 18 23 20 22 24 21 26 19
What exactly should the output be?
Also, what is the file type? Excel, CSV, MATLAB mat file?
pink flower
pink flower on 26 Sep 2021
This must be my output (M3):
M1 = magic(3);
M2 = magic(3)+17;
M3 = M1 + M2;

Sign in to comment.

Answers (2)

Sulaymon Eshkabilov
Sulaymon Eshkabilov on 23 Sep 2021
Here is one of the possible routines to read all 1e4 files and collect all data into a signle variable and the rest is relatively simple:
clearvars
FILE = fullfile('C:\Users\...', '*.txt'); % Directory where the files are residing. Your data file extension, e.g.: .txt, .xlsx, .csv, etc
LIST = dir(FILE);
N = length(LIST); % N = 1e4 as you stated that 1e4 files
D = zeros(1e3, 1e3, N);
for ii = 1 : N
FFName = fullfile(LIST(ii).folder, LIST(ii).name);
DATA = readmatrix(FFName);
D(:, :, ii) = DATA; % NOW D contains all data from 10000 files
end
... % Select and process the part of D that is necessary
  2 Comments
pink flower
pink flower on 26 Sep 2021
I tried these commands, but I was unsuccessful. This was the mistake:
Error using zeros
Requested 1000x1000x48282 (359.7GB) array exceeds maximum
array size preference. Creation of arrays greater than
this limit may take a long time and cause MATLAB to
become unresponsive. See array size limit or preference
panel for more information.
Error in test (line 10)
D = zeros (1e3, 1e3, N);
Sulaymon Eshkabilov
Sulaymon Eshkabilov on 26 Sep 2021
These are not coding problems but rather memory problems. That means either you should process your data chunk by chunk or increase the memory in your matlab settings.

Sign in to comment.


Stephen23
Stephen23 on 26 Sep 2021
P = 'absolute or relative path to where the files are saved';
S = dir(fullfile(P, '*.txt')); % Select the file extension to suit your data files
M = 0;
for k = 1:numel(S)
F = fullfile(P, S(k).name);
M = M + readmatrix(F);
end
  6 Comments
pink flower
pink flower on 26 Sep 2021
For example, in row 400 and column 5, I have:
M1 = NaN;
M2 = NaN;
M3 = 0.9
And the result it returns to me is NaN. Do you understand now? If not, I can explain it better.
Stephen23
Stephen23 on 26 Sep 2021
Edited: Stephen23 on 26 Sep 2021
To ignore NaN when summing you can use SUM with the appropriate flag, e.g.:
M = zeros(1000,1000);
for k = 1:numel(S)
F = fullfile(P, S(k).name);
N = whateverCodeYouUseToImportYourFileData;
M = sum(cat(3,M,N),3,'omitnan');
end

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!