GPU computations slowing down

3 views (last 30 days)
Peter
Peter on 5 Jan 2017
Commented: Joss Knight on 8 Jan 2017
Within a loop, my code carries out some iterative solving procedure on the GPU. With every loop iteration, the time taken by the solving procedure increases. There doesn't seem to be a memory leak: gpuDevice consistently reports approximately the following:
TotalMemory: 1.2885e+10
AvailableMemory: 5.6773e+09
However, when I pause the loop for a while by means of a debug stop, after relaunching again, computations are up to speed again, after which they start slowing down again...
I'm running matlab2016b on Windows 10, using a Nvidia Titan X (Pascal) GPU. Any suggestions for this problem?
  3 Comments
Peter
Peter on 6 Jan 2017
thanks Jan, I already started to be suspicious about temperature and monitored it. After launching the matlab script, temperature starts rising, GPU performance starts being capped by Temperature. (See performance monitor screen shot below) After reaching 95(!) degrees, the crash occurred again. So we do have strong indications for a cooling issue... What does this mean?
  • a) I got myself a wrong system set-up, need better cooling ?
  • b) this GPU card doesn't suit my needs, perhaps go for one of the Keplers (more expensive for similar memory)?

Sign in to comment.

Accepted Answer

Peter
Peter on 6 Jan 2017
Monitoring the GPU performance revealed that most probably the temperature is causing the issue: Slowing down of performance goes with rising of temperature and performance is capped by temperature.
Crash of the GPU occurred when GPU reached 95 degrees...
  1 Comment
Joss Knight
Joss Knight on 8 Jan 2017
Most people who report issues like this are simply seeing an issue of asynchronous execution. They see fast times until the kernel execution queue is full, and then they start seeing the actual complete computation times.
To avoid this try using wait(gpuDevice) to add synchronisation points to your code.

Sign in to comment.

More Answers (0)

Categories

Find more on Loops and Conditional Statements in Help Center and File Exchange

Products

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!