Matlab python engine won't run in parallel after another instance of the engine has quit

I am using the python ro run some matlab scripts in parallel using python's multiprocessing library, and each of the parallel threads uses it's own isntance of the matlab engine. Everything works fine unless my python script includes an instance of the matlab engine outside of the parallel loop.

This issue is much easier to understand with an example python script, below. I used both python and matlab comment symbols so that the comment lines would still be valid comments in python, and at the same time would be easier to read as comments in the matlab markup.

import matlab.engine
from multiprocess import Pool

# %example function to do something in matlab
def do_some_matlab(num):
	with matlab.engine.start_matlab() as eng:
		result = eng.ones(num)
	return result

# %run two instances of this function in parallel
n_proc = 2
with Pool(n_proc) as pool:
	results = pool.map(do_some_matlab, [3,3])

# %so far, everything works, we can see that
# %it returned two 3x3 matrices of ones
print(results)

# %now we use an instance of the matlab engine
# %to do something in the main level of the script
with matlab.engine.start_matlab() as eng:
	print(eng.ones(3))

# %we can still run a single instance of the function
print(do_some_matlab(3))

# %but now we can't run the function in parallel
# %anymore, the matlab engine starts up but just
# %freezes without doing anything
with Pool(n_proc) as pool:
	results = pool.map(do_some_matlab, [3,3])
print(results)

[Tyler Ward]

I just ran into this problem too.

I found this online but I have recieved mixed feedback from others on whether or not the latest engine can utilize multiprocessing: https://www.mathworks.com/help/matlab/matlab_external/limitations-to-python-support.html

Answer 1

[ZHIYAO LUO]

Hi,

I notice that you're trying to re-open MATLAB session for many times.

Remeber that every time you do a 'with matlab.engine.start_matlab()' you actually start and end a matlab session, which is super time consuming. Try to use a list to store all matlab engine before you run multiprocessing.

Hope it helps.

[信博 赵]

Hi,

I've met the same problem and I found that I couldn't pass the matlab engine as a params to multiprocessing.

Could you please give an example on how to use a list to store muliple engines and pass them to multiprocessing

Thanks a lot!

[Julio Rodillo Climent]

Hello! I have encountered the same problem. Any solution for this? Pass the engine as an argument to the map or starmap functions?