Titan V using huge amount of memory

Recently I added a second GPU (titan V) to my workstation. When I try to perform my simulations which use a parpool with each worker acces to the GPU's in the system I noticed that the Titan V uses a lot more memory than my other GPU (Tesla K40c) which results in out of memory and invalid object handles errors. The Titan V uses for the initialization of matlab on the GPU around 514MB whilst the tesla only takes 126 MB on the workers.

+-----------------------------------------------------------------------------+

| NVIDIA-SMI 390.65 Driver Version: 390.65 |

|-------------------------------+----------------------+----------------------+

| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |

| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |

===============================+======================+======================

| 0 TITAN V TCC | 00000000:04:00.0 Off | N/A |

| 33% 48C P8 30W / 250W | 709MiB / 12186MiB | 0% Default |

+-------------------------------+----------------------+----------------------+

| 1 Quadro K6000 TCC | 00000000:A1:00.0 Off | Off |

| 32% 57C P8 25W / 225W | 459MiB / 12214MiB | 0% Default | +-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+

| Processes: GPU Memory GPU PID Type Process name Usage | =============================================================================

| 0 14600 C ...iles\MATLAB\R2018a\bin\win64\MATLAB.exe 514MiB |

| 1 4436 C ...iles\MATLAB\R2018a\bin\win64\MATLAB.exe 126MiB |

| 1 24908 C ...iles\MATLAB\R2018a\bin\win64\MATLAB.exe 126MiB |

+-----------------------------------------------------------------------------+

Answer 1

[Joss Knight]

To avoid this cost, do not use the same GPU on multiple workers. Although the GPU is a data parallel system, it is not task parallel - in other words, anything you run on the GPU from different workers will run in serial, gaining you no benefit of overlap. In addition you are experiencing the additional problem that every new CUDA context being opened on the same GPU is causing this extra memory overhead.

To solve this issue, open your parpool with gpuDeviceCount workers, and no more.

Answer 2

[Joss Knight]

The CUDA runtime in the latest drivers appears to be allocating a huge amount of main memory when it is initialized on a Volta card. We are following this up with NVIDIA.

[arvid Martens]

Is there any new on the driver issue? It is really hampering my workflow.

Another strange thing is that when I load an instance of matlab using only the tesla K40c it occupies a significant amount (300-400MB) of memory on the titan V even if it is not being used.

[Joss Knight]

NVIDIA have indicated to us that this memory allocation size is expected, holding kernels and page tables. But we are still chasing them further.

The allocation on the Titan V occurs when the CUDA runtime is initialized, which happens when the device is selected. To avoid this, select your second device as the first GPU command you execute, i.e. gpuDevice(2).