can I accelerate parfor loop running external code?

Here is my question:

if I run following code, the toc time is 0.296749.

tic
lambda=1;
for k=1:lambda
    filename_nam=sprintf('mf2005 test_%03d.nam',k);
    [~,~]=system(filename_nam);
    .
    .
    .
end
toc

Otherwise, if I run following code, the toc time is 12.920165. This result is reasonable because 0.296749 * 43 is approximately 12.9.

tic
lambda=43;
for k=1:lambda
    filename_nam=sprintf('mf2005 test_%03d.nam',k);
    [~,~]=system(filename_nam);
    .
    .
    .
end
toc

Lastly, if I run following code, the toc time is 3.902111. This is obviously improvement by using parfor but I wonder if there is further way of improving the efficiency. In my parallel pool, I have 36 worker (36 core CPU) and the required memory for the exe code is very small compared to my 64Gb memory. The operating system is Windows 10. What is the limiting condition? code? a specific hardware? or any other?

tic
lambda=43;
parfor k=1:lambda
    filename_nam=sprintf('mf2005 test_%03d.nam',k);
    [~,~]=system(filename_nam);
    .
    .
    .
end
toc

[Mohammad Sami]

See the answer from Walter. you can just start all asynchronously and check back their states at the end

https://www.mathworks.com/matlabcentral/answers/497886-finding-pid-of-external-process-called-from-matlab#answer_407569

[Eungyu Park]

The part inside the parfor loop consists of an execution subpart and a subpart that reads the execution result. It looks like the code using .NET framework is trying to read the results before execution ends. This causes an error. If I just run the code without the reading subpart, the computation rate seems to be very fast. However, the problem is that the toc time is much earlier than termination of all executions (The toc time is already displayed in the command window even though it is still being calculated). Also, the shell window popup consumes computer resources and slows down computation. Are there any solutions?

[Walter Roberson]

You can test if a process started with .NET is still going, or you can wait for it (with an optional timeout)

https://support.microsoft.com/en-ca/help/305369/how-to-wait-for-a-shelled-application-to-finish-by-using-visual-c

https://stackoverflow.com/questions/3147911/wait-until-a-process-ends

Anyhow, typically the answer to the question of the limiting condition is either communications overhead or use of a limited resource (for example disk I/O bandwidth.)

Answer 1

[Eungyu Park]

Thank you Mohammad and Walter for your help! It seems that 'system' and 'System.Diagnostics.Process()' take about same time for my parfor loop execution of external code.

[Walter Roberson]

Sometimes when multiple processes use a shared resource but do not each need 100% of the resource, then it can be beneficial to start a limited number of the processes and wait for one to finish before starting the next, so that there might be a small number of processes accessing the resource but not too many. This can reduce contention for the resource. This can be especially important for disc access.