optimization in cluster

Hi, I am trying to run a large piece of optimization in a university cluster. The files are in LSF, and I was trying to invoke multiple cores to speed up the optimization. The problem is that somehow, matlab judege I run the code locally and only allows me to use 8 maximum workers(I did it quite in standard manner, i.e. matlabpool open 20 followed with multistart code). I am wondering how may I use more than 8 workers in cluster.

The second part of the question is about multistart optimization. The objective function is 3hrs+, So I am trying to parallelize it as much as possible. If I invoke 100 workers but only assign 30 start points, do the remaining 70 workers help with parallel calculation in each strating point? Is there a way to utilize all idle workers?

Thanks, Danyang

Answer 1

[Walter Roberson]

In order to run with a distributed cluster, you need to specify the scheduler in your matlabpool open call, as the default scheduler is the local scheduler.

My understanding (possibly wrong) is that the additional workers will not help in the computation -- though that might perhaps depend on which kind of multiprocessing you are doing (e.g., parfor vs spmd). It is legal, however, for a worker to use matlabpool to request additional workers and then to use parallel code within the worker. However, to do this, you should only open the original pool with the number of primary workers that you need, as the secondary workers must be allocated from the unallocated workers and will never allocate from the workers that were requested before but are idle.

MATLAB will automatically run some operations in parallel even without any parallel code, such as some BLAS library calls. I do not know how that interacts with "workers" or the matlabpool.

[Danyang]

Thanks for you answer! So for the second part, I am not sure what you mean exactly. The function I want to optimize is y=f(x), which is regular function(no parfor. etc). However, since it takes 3hrs to get y from x, I want the optimization to run parallel. I use MultiStart in the optimization toolbox, which enable parallel feature. Nevertheless, for each single start point, it will apply one of the solvers(e.g. fmincon), which runs f(x) hundreds of time. So I am wondering if I can parallelize that part as well. Otherwise it is still unbearably long to run the whole optimization.

[Walter Roberson]

It appears that fmincon can use parallel computing, but you will need a different pool for each solver invocation.

http://www.mathworks.com/help/toolbox/optim/ug/briutty.html

Answer 2

[Jason Ross]

To answer the first part of the question, you can have MATLAB and Parallel Computing Toolbox integrate with LSF a few ways. There is a direct integration available through the Parallel menu, and there are example scripts for using the generic scheduler integration. Which is appropriate for your use depends on your university's policy and cluster configuration. Once you have a working configuration, though, you can use it to send computations to the LSF cluster.

As for the number of workers actually used, that's another policy / setup question. If your university lets you start 100 workers, then maybe it will help. But if you only start 20, then the tasks will be scheduled to deal with that.

You can also change the default to use the LSF cluster by default if you wish, but that might not be appropriate for your daily use.

Also, in 2011b, the number of allowed local workers has been increased from 8 to 12 -- although that makes the assumption that your machine can work with 12 workers and that this would speed up your computation.

[Danyang]

Thanks! Well I guess I didn't fully understand the parallel computing after all. I may hsve to spend sometime working on how to change default number of workers first