run background function on subset of allocated parpool nodes on cluster?
2 views (last 30 days)
Show older comments
Science Machine on 23 Jul 2022
Commented: Edric Ellis on 25 Jul 2022
Is it possible to start a parpool, then run a background function on a subset of the parpool's allocated nodes (on a cluster)?
Suppose I allocate (total Procs) = (48 procs) x ( 6 nodes) using
Then can I then invoke some command such as
Result1 = parfeval(backgroundPool_1,@rand,1,myNum(1)); % suppose bgPool_1 is a subset of parpool somehow
Result2 = parfeval(backgroundPool_2,@rand,1,myNum(2)); % suppose bgPool_1 is a subset of parpool somehow
where backgroundPool_1 would be like 48*6/2, half of the number of allocated nodes, and same for bgP_2.
- I would not want to allocate two different parpools, because I am running interactively on a cluster and want all nodes to be pre-allocated on the slurm queue
Edric Ellis on 25 Jul 2022
This isn't possible today. Can you explain more about what you want to do on the pool subset if you could create one? What do you want to run there, and how would it use multiple workers of the subset? One thing I can imagine is doing something like this:
p = parpool(..);
[subPool1, subPool2] = somehowDividePool(p);
fut = parfeval(subPool1, @myFunctionThatCallsParfor, numOuts, in1, in2);
Here, myFunctionThatCallsParfor has a parfor loop inside, and it gets to use the workers from subPool1 to run that.
Is that the sort of thing you're interested in? If not, it would be useful if you could explain what you would like to be able to do.
Find more on Parallel Computing Fundamentals in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!Start Hunting!