import over 1000 files

i want to change many files including files in subfolders.

but if file outnumber 1001, my code stops.

i'd like to import over 1000 files to "files" struct.

[Rik]

What code are you using to select the files?

[Jeong Dae Kim]

@Rik

i edited my Question. take a look

Answer 1

[Paul Kaufmann]

Don't know what the problem is with exceeding N=1000. Works for me:

path = "data/"
files = [ dir(fullfile(path,'**','*.txt')); dir(fullfile(path,'**','*.txt')) ]; % do it twice, to exceed 1000
N = numel(files)

Gives this result for my datafolder:

ans = 1890

As I assume you want to rename files or something, you can now just iteratively work on them.

However the following will get you into trouble:

>> numel(files.name)
Warning: Number of elements exceeds maximum flint 2^53-1.
The result may be inaccurate. 

ans =

   Inf

Answer 2

[Chunru]

You may hit the limit os the number of files that can be opened simutaneously. While it possible to increase the maximum number of opened files in OS, you may want to try open file one by one and then close the file once it is not used.

Answer 3

[Image Analyst]

Try code in the FAQ: https://matlab.fandom.com/wiki/FAQ#How_can_I_process_a_sequence_of_files.3F

It will work for more than 1000 files, if you have more than that.

% Specify the folder where the files live.
myFolder = 'G:\';
% Check to make sure that folder actually exists.  Warn user if it doesn't.
if ~isfolder(myFolder)
	errorMessage = sprintf('Error: The following folder does not exist:\n%s\nPlease specify a new folder.', myFolder);
	uiwait(warndlg(errorMessage));
	myFolder = uigetdir(); % Ask for a new one.
	if myFolder == 0
		% User clicked Cancel
		return;
	end
end
% Get a list of all files in the folder with the desired file name pattern.
filePattern = fullfile(myFolder, '*.png'); % Change to whatever pattern you need.
theFiles = dir(filePattern);
numFiles = length(theFiles)
Result = zeros(numFiles, 1);
for k = 1 : numFiles
	baseFileName = theFiles(k).name;
	fullFileName = fullfile(theFiles(k).folder, baseFileName);
	fprintf('Now reading %s, file #%d of %d.\n', fullFileName, k, numFiles);
	% Now do whatever you want with this file name,
	% such as reading it in as an image array with imread()
	imageArray = imread(fullFileName);
	imshow(imageArray);  % Display image.
	drawnow; % Force display to update immediately.
	Result(k) = mean(imageArray(:)); % Whatever you want.
end

[Jeong Dae Kim]

any other solution with my code?

[Image Analyst]

@Jeong Dae Kim What was wrong with my code? Just put your 3 lines of code at the top of mine (replacing my dir()) and change variable names appropriately and you're good to go.

[Jeong Dae Kim]

@Image Analyst

it fails

[Rik]

No, your code fails. The reason is that you're telling matlab you want a 1x2x3x4x5x...x1000 array, instead of a 1x1000 vector.

[Stephen23]

@Jeong Dae Kim: you request a huge array, much larger than any normal computer could hold in memory:

Result = zeros(1:numFiles);

This requests an array of size 1x2x3x4x5...xnumFiles. For 1000 files this array will contain exactly 402387260077093773543702433923003985719374864210714632543799910429938512398629020592044208486969404800479988610197196058631666872994808558901323829669944590997424504087073759918823627727188732519779505950995276120874975462497043601418278094646496291056393887437886487337119181045825783647849977012476632889835955735432513185323958463075557409114262417474349347553428646576611667797396668820291207379143853719588249808126867838374559731746136085379534524221586593201928090878297308431392844403281231558611036976801357304216168747609675871348312025478589320767169132448426236131412508780208000261683151027341827977704784635868170164365024153691398281264810213092761244896359928705114964975419909342221566832572080821333186116811553615836546984046708975602900950537616475847728421889679646244945160765353408198901385442487984959953319101723355556602139450399736280750137837615307127761926849034352625200015888535147331611702103968175921510907788019393178114194545257223865541461062892187960223838971476088506276862967146674697562911234082439208160153780889893964518263243671616762179168909779911903754031274622289988005195444414282012187361745992642956581746628302955570299024324153181617210465832036786906117260158783520751516284225540265170483304226143974286933061690897968482590125458327168226458066526769958652682272807075781391858178889652208164348344825993266043367660176999612831860788386150279465955131156552036093988180612138558600301435694527224206344631797460594682573103790084024432438465657245014402821885252470935190620929023136493273497565513958720559654228749774011413346962715422845862377387538230483865688976461927383814900140767310446640259899490222221765904339901886018566526485061799702356193897017860040811889729918311021171229845901641921068884387121855646124960798722908519296819372388642614839657382291123125024186649353143970137428531926649875337218940694281434118520158014123344828015051399694290153483077644569099073152433278288269864602789864321139083506217095002597389863554277196742822248757586765752344220207573630569498825087968928162753848863396909959826280956121450994871701244516461260379029309120889086942028510640182154399457156805941872748998094254742173582401063677404595741785160829230135358081840096996372524230560855903700624271243416909004153690105933983835777939410970027753472000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 elements (approx. 1e2567). Multiply that by 8 to get the required number of bytes.

Does your computer have that much memory? (hint: the observable universe has around 3.28e80 particles)

[Jeong Dae Kim]

@Rik, @Stephen Cobeldick

oops i mistaked. but same result

[Image Analyst]

@Jeong Dae Kim, Why did you declare Results twice? And why is the final one a cell array? Do you need a cell array?

Please attach your entire .m file so we can fix it.

[Jeong Dae Kim]

@Image Analyst

to preallocate, but not working.

that's all of my code. i need to change that filenames.

At first, i store over 1000 filename.

[Rik]

Well, then you have only 1000 that match your pattern. I don't believe there are releases without the recursive ** for dir that have live scripts.

How do you want to store your results? In a vector? An array? And what data type should it be?

Also, next time you're posting code, please post it as code (not as an image). You could even attach your mlx file.

[Jeong Dae Kim]

@Rik

the file name contains korean, chinese, japanese and Special characters like (, _ .

can convert to another data type?

[Rik]

Why would you? The char data type supports full Unicode, so even CKJ characters are fully supported. As far as I tested the string data type also support full Unicode (although the readlines function does seem to have some issues with emoji (or more correctly: characters outside the BMP)).

Why do you want to convert them? To what?

And why don't you answer the questions you're getting?

[Image Analyst]

I believe this should work to find all .mp4 and .mpg files in the folder plus subfolders. Make sure to assign the desired starting folder, myFolder. If it doesn't work, attach your m-file.

% Specify the folder where the files live.
myFolder = pwd; %'G:\';
% Check to make sure that folder actually exists.  Warn user if it doesn't.
if ~isfolder(myFolder)
	errorMessage = sprintf('Error: The following folder does not exist:\n%s\nPlease specify a new folder.', myFolder);
	uiwait(warndlg(errorMessage));
	myFolder = uigetdir(); % Ask for a new one.
	if myFolder == 0
		% User clicked Cancel
		return;
	end
end
% Get a list of all files in the folder with the desired file name pattern.
filePattern = fullfile(myFolder, '**/*.mp4') % Change to whatever pattern you need.
pngFiles = dir(filePattern);
filePattern = fullfile(myFolder, '**/*.mpg') % Change to whatever pattern you need.
bmpFiles = dir(filePattern);
theFiles = [pngFiles; bmpFiles];
numFiles = length(theFiles)
Result = zeros(numFiles, 1);
for k = 1 : numFiles
	baseFileName = theFiles(k).name;
	fullFileName = fullfile(theFiles(k).folder, baseFileName);
	fprintf('Now reading %s, file #%d of %d.\n', fullFileName, k, numFiles);
	% Now do whatever you want with this file name,
	% such as reading it in as an image array with imread()
	Result(k) = whatever; % Whatever you want.
end

[Jeong Dae Kim]

not working. that is all of my code

[Rik]

Please show all your code, exactly as you try to use it.

Answer 4

[michael adams]

I can confirm this is an issue and it is not consistent. If I query many directories (all with more than 1000 files), sometimes 'dir' returns the correct structure of all files, but sometimes it just stops at 1000.

I do not run into this issue on local drives. I see this issue on network drives only.

[Rik]

How did you confirm this is an issue with Matlab and not with the network architecture and/or the OSes involved?