Can't open anymore files with fopen above some threshold

2 views (last 30 days)
Hi guys!
I am trying to open all files contained in a folder, with something like
folder = 'some/folder/path'
files = dir(folder);
for i = 1:length(files)
filenames{i} = files(i).name;
output(i).name = filenames{i};
output(i).file = sprintf('%s/%s',folder,filenames{i});
output(i).fid = fopen(output(i).file);
if (output(i).fid == -1)
output(i).status = 'not read';
else
output(i).status = 'read';
end
end
The folder contains few thousands of very light text files.
The problem is that, after having read some number of this files (in my case always 1020) the files in the list are not read anymore. So I get something like
Now, the problem is not the limit of files which can't be opened by MATLAB, because for my system it is apparently 10240.
If, in this situation, I try to open again manuallt that qc.dat, for example, I get -1 and 'Operation not permitted'. BUT if I try to read again manually that q_plates.dat, i.e. the last file which was read, it is read again with new fid 1021.
And, it seems also not a problem related with the files. If i close everything and only after that I try to manually read that 'qc.dat' and so on, I succeed.
What's the problem? The only thing I can think is a limit of the number of files opened in the same moment which sit in the same folder... is it the case?
I use MATLAB R2021b on a MacBook 10.15.7.
  2 Comments
Rik
Rik on 29 Apr 2022
I don't have a solution for your problem, so let me try to figure out the deeper motive here: why exactly do you want to open that many files at once? I try to keep my file interactions as short-term as possible, so I hardly ever have more than 1 active FID.
Steven Lord
Steven Lord on 29 Apr 2022
Having two files open simultaneously (one for reading, one for writing) is a common pattern. But I agree with @Rik; why do you need thousands of files open at once?
Open the file, do whatever processing you need to do with it, then close it. Repeat until you get to the end of the list of files. [Depending on the details of what processing you're performing, you may even be able to parallelize it using a parfor with this approach if you have Parallel Computing Toolbox installed.]

Sign in to comment.

Answers (2)

Walter Roberson
Walter Roberson on 29 Apr 2022
There is a per-process limit on the number of open files,
!ulimit -n
65535
The limit for the operating system running MATLAB Answers is 65535, but the limit on my Mac is 10240 by default.
If you wanted to change the limit for your system, see the "Sierra and newer systems" description at https://wilsonmar.github.io/maximum-limits/
  1 Comment
Antonello Zito
Antonello Zito on 29 Apr 2022
Yes, I know that. Also for my case the limit is 10240. Indeed I have no problems in open more files.
My problem is that I cannot open more than N files which sit in the same folder. After I run that code, I am not able to open qc.dat and so on as long as the previous 1020 in that folder are open... but I am able to open other files in other folders which will have resulting fids 1021 and so on.

Sign in to comment.


Image Analyst
Image Analyst on 29 Apr 2022
Why do you need to have them ALL open simultaneously? What I'd do is open one, get it's contents and do something with it, then close it and repeat for all the other files. So you'd have only one oepn at a time.

Categories

Find more on Startup and Shutdown in Help Center and File Exchange

Tags

Products


Release

R2021b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!