- Are you timing the entire parfor block? e.g. tic, parfor ... end, toc
- What is the spread of timings? Is it usually 14 seconds, and occasionally up to 110 seconds?
- Machine information (operating system), and MATLAB Release.
Parfor Execution time variation
42 views (last 30 days)
Show older comments
Anshika Goel
on 6 Jun 2024
Commented: Anshika Goel
on 21 Aug 2024 at 11:46
Hi,
I am using parfor for reading 600 .raw files.
c=zeros(1536,1536,600,'uint16');
parpool('threads',4);
parfor i=1:600
fileName=[folder,'/',fileList(i).name];
a=fopen(fileName,'r');
Z=fread(a,[1536 1536],'uint16');
fclose(a);
c(:,:,i)=Z;
end
However, I am observing significant variability in the execution time, which ranges from 14 seconds to 110 seconds across different runs.
Why is this discrepancy occurring? Is there a way to achieve more consistent execution times?
4 Comments
Christopher Mirfin
on 12 Jun 2024
Do you observe the same variability when running with a standard for-loop, or a process-based pool parpool("Processes",4) ?
Also, are you reading from your local hard drive or a network location?
Accepted Answer
Swastik
on 21 Aug 2024 at 11:03
I also have an Intel Xeon processor (4 cores) with 16GB RAM. I executed your code after generating 600 files as follows:
matrix = uint16(ones(1536));
folder = 'nums';
for i=1:600
fid = fopen([folder '/' num2str(i)], 'w');
mat = matrix .* i;
fwrite(fid, mat, 'uint16');
fclose(fid);
end
In my tests, the variability in execution time was not as significant as you mentioned; it ranged from 189 seconds to 200 seconds. the main bottleneck is likely due to file I/O operations.
To optimize execution time, consider performing file reads asynchronously. I developed the following code using “parfeval” to read 600 files asynchronously:
c = zeros(1536, 1536, 600, 'uint16');
folder = "nums";
pool = gcp('nocreate');
if isempty(pool)
pool = parpool('threads');
end
futures = parallel.FevalFuture.empty(600, 0);
for i = 1:600
fileName = fullfile(folder, num2str(i));
futures(i) = parfeval(@readFile, 1, fileName);
end
for i = 1:600
c(:, :, i) = fetchOutputs(futures(i));
end
function Z = readFile(fileName)
a = fopen(fileName, 'r');
Z = fread(a, [1536 1536], 'uint16');
fclose(a);
end
In my tests, this approach reduced the execution time to between 14-16 seconds.
You can learn more about “parfeval” from here:
I hope this helps.
More Answers (0)
See Also
Categories
Find more on Parallel Computing Fundamentals in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!