Increase mdce heap space?

Hello, I´ve been experiencing problems with mdce. I have a CentOS machine with 4 processors (4x20 cores) and 320GB of RAM, currently 83 workers running. mdce-service.log keeps saying (several times):

 Group-10:out:PeerRmiDispatcher: Error in dispatch while sending return message: 
 java.lang.OutOfMemoryError: Java heap space

So I think I should increase the java heap space for mdce, but searching for -Xmx I found several parts on the MDCE where this value is mostly hardcoded. What is the right way of doing it?

BTW only 100GB/320GB of RAM is being used starting all of the workers.

Answer 1

[Kyle Allison]

The right way to increase the Java heap size for MDCE is to adjust the JOB_MANAGER_MAXIMUM_MEMORY setting in the mdce_def file, which is located at

• matlabroot/toolbox/distcomp/bin/mdce_def.sh (on Linux or Macintosh operating systems)
• matlabroot\toolbox\distcomp\bin\mdce_def.bat (on Windows operating systems)

By default, it is set to 1024MB:

# JOB_MANAGER_MAXIMUM_MEMORY: The maximum heap size of the job manager java
# process.
JOB_MANAGER_MAXIMUM_MEMORY="1024m"

[Juan Mucarquer]

After increasing the memory to 4096m and restart mdce, still got the same error several times:

 Group-10:out:PeerRmiDispatcher: Error in dispatch while sending return message: 
 java.lang.OutOfMemoryError: Java heap space

should I keep increasing? is normal for mdce to consume too much memory for the administration of each worker or this is because I so many workers (~100 workers)? I assume that are the mdce/mjs processes that are out of memory, not the workers itself.

Answer 2

[Kyle Allison]

In most cases, 1024MB is sufficient for the job manager heap memory and 4096MB is plenty (even for a job manager with 100 workers or more). I would recommend not increasing the heap memory above 4096MB in an attempt to fix the issue and instead contact support with more details about how the cluster is being used to cause the OutOfMemoryError.