Parfor with options = statset('UseParallel',true);
4 views (last 30 days)
Show older comments
Christopher McCausland
on 22 Nov 2023
Commented: Edric Ellis
on 28 Nov 2023
Hi
I have access to a large cluster many CPU's. I am hoping to prefrom the same operations on several hundred .edf input files with a parfor loop. So far so good. Part of this processing would benifit from an additional pararellel loop for all-vs-one testing.
I am therefore wondering if it is possiable to use both of these options at once. For example if I have a parpool(6) and a classifier with five classes. Therefore 6*5 = 30 cpu cores, which I have access too. I know that nested parfor loops are not generally supported, however I am wondering if there is anyway to 'trick' matlab into preforming as such?
Alternatively I could just use parfor(30) for example, however I am intrested to see what the preformacne gains/losses are of implementing this.
Kind regards,
Christopher
0 Comments
Accepted Answer
Edric Ellis
on 23 Nov 2023
If you are able to "flatten" the parallelism so that it is all exposed to a single parfor loop with enough iterations to keep all your workers busy, that will always be the most efficient. But that's not always possible.
The restriction on nested parfor loops is that you cannot nest a parfor loop directly inside another parfor loop. However, you can "hide" the inner parfor loop inside a function. Today, achieving nested parallelism is a bit tricky. One way to achieve that is to have each worker create its own parpool. Depending on the workload, you might be able to set your cluster to launch one multi-threaded worker per host on the cluster, and then use parpool("Threads") on the cluster workers to launch an inner thread-based pool.
2 Comments
Edric Ellis
on 28 Nov 2023
The thing you're trying to avoid is having idle workers at the back end of your computation because they've run out of things to do. When you try to use nested parallelism, this might increase the amount of time wasted in that way. The happiest scenario is when you've got plenty of iterations of a top-level parallel construct so that they can be load-balanced by the parfor internals to smooth out any differences in execution times between the iterations etc.
More Answers (1)
Walter Roberson
on 22 Nov 2023
I know that nested parfor loops are not generally supported, however I am wondering if there is anyway to 'trick' matlab into preforming as such?
You would need to start separate MATLAB sessions.
You should perhaps also consider queuing up at lot of parfeval() sessions.
Hmmmm -- I wonder if parfeval() can call parfeval() to queue more work? I don't see why not... though you do risk deadlock if all of the sessions are locked up waiting for other sessions that cannot execute because there are no available cores...
2 Comments
Edric Ellis
on 23 Nov 2023
A worker cannot call parfeval to schedule more work on the pool that it is part of (one reason is to avoid the risk of deadlock). You can use afterEach to have the client schedule more work.
See Also
Categories
Find more on Parallel Computing Fundamentals in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!