Summing array elements seems to be slow on GPU

I am testing the times of execution for the following function on CPU and GPU

function funTestGPU(P,U,K,UN)

    for k = 1:P
        H = exp(1i*K);
        HU = U.*H;
        UN(k,:) = sum(HU,[1,3]);
    end
end

where , are complex arrays of size and Kis a complex array of size . So in each iteration I perform element-wise exp(), element-wise multiplication of two arrays and summing elements of 3D array along two dimensions.

I test the execution time on CPU and on GPU with the help of the following script

P = 200;

URe = 1/(sqrt(2))*rand(P);
UIm = 1/(sqrt(2))*rand(P);

KRe = 1/(sqrt(2))*rand(P,P,P);
KIm = 1/(sqrt(2))*rand(P,P,P);

% CPU
U = complex(URe, UIm);
K = complex(KRe, KIm);
UN = complex(zeros(P), zeros(P));

fcpu = @() funTestGPU(P,U,K,UN);
tcpu = timeit(fcpu);
disp(['CPU time: ',num2str(tcpu)])

% GPU
U = gpuArray(complex(URe, UIm));
K = gpuArray(complex(KRe, KIm));
UN = gpuArray(complex(zeros(P), zeros(P)));

fgpu = @() funTestGPU(P,U,K,UN);
tgpu = gputimeit(fgpu);
disp(['GPU time: ',num2str(tgpu)])

and I obtain the results

CPU time: 9.0315
GPU time: 3.3894

My concern is that if I remove the last operation from the funTestGPU (summing array elements) I obtain the results

CPU time: 8.0185
GPU time: 0.0045631

So it looks like the summation is the most time-consuming operation on GPU. Is that an expected result?

I wrote the analogical codes in cuPy and in Pytorch and there the summation does not seem to be the most time consuming operation.

I use Matlab 2019b. My graphics card is NVIDIA GeForce GTX 1050 Ti (768 CUDA cores), my processor is AMD Ryzen 7 3700X (8 physical cores).

[Matt J]

So it looks like the summation is the most time-consuming operation on GPU. Is that an expected result?

That's what I would expect. It's the only operation in the chain that is not element-wise.

[Damian Suski]

@Matt J Thank you for your comment. Before I run tests, I imagined that the exponential will be the most time consuming operation, but it turns out that element-wise operations are not the bottleneck of calculations. I just wanted to make sure that I do not miss something obvious.

Answer 1

[Joss Knight]

These are my results that I got on my (somewhat old) GeForce GTX 1080 Ti:

CPU time: 16.1288
GPU time: 0.96266

If I change the datatype to single I get:

CPU time: 14.9785
GPU time: 0.35102

That's maybe 2x faster?

So on the one hand your GPU is pretty slow and your CPU is pretty fast, and on the other maybe you could try using single precision instead, if you don't mind the loss of accuracy.

[Damian Suski]

Well, I would also say that my CPu is quite fast and GPu is rather weak (only 800 CUDA cores, 4GB RAM). Several years ago I have bought the cheapest graphics card, without parallel computations in mind.

The results for your card (over 3.5k CUDA cores, 11GB RAM) are pretty impressive, I have tried GeForce RTX 3060 (over 3.5k CUDA cores, 12GB RAM) on another computer and it gave 1,5s for double precision. For the analogical code in pytorch, I have tried Tesla T4 card (freely available on Google Colab), which gave also 1,5s. So the proper choice of the GPU card makes the difference.

I will definitely try single precision, but at the moment it is hard for me to say whether the precision loss will be acceptable for my purpose.

Answer 2

[Joss Knight]

Why are you recomputing H and HU inside the loop? They do not change. If you remove the sum, because the results are never used from the first (P-1) iterations, only the last computation of those values will actually take place.

[Matt J]

Very strange. I wonder if it is wise to have this "optimization". Essentially, it causes the user's instructions to be disobeyed.

[Joss Knight]

Most people using the GPU want every optimization they can get. For instance, strictly speaking a user who has written

C = A.'*B;

Has requested that A be transposed, but in fact this never happens.

There is no way the user can see the underlying behaviour. All the instructions are recorded and if the user attempts to access the results of any operation it will be computed.

[Damian Suski]

@Joss Knight Sorry for not explaining that it in the original message.

That calculations are performed only to test the GPU performance. In the real scenario the matrix K will be different in each iteration (calculated on the basis of some external data) and I will use from the current iteration as U in the next iteration. There will also appear another opeartions inside the loop.

Before I run tests, I imagined that the exponential will be the most time consuming operation, but it turns out that element-wise operations are not the bottleneck of calculations. I just wanted to make sure that I do not miss something obvious.

[Joss Knight]

Yes. You will get better performance from computing the result in a single sum, but you will probably run out of memory so would have to do it in batches:

function funTestGPU(P,U,K,UN)
    HUall = zeros(P,P,P,P,'like',U);
    for k = 1:P
        H = exp(1i*K);
        HUall(:,:,:,k) = U.*H;
    end
    UN = sum(HUall,[1,3]);
    UN = permute(HUall,[4 2]);
end

[Damian Suski]

I have tried batching approach on my GPU, but have not noticed any speed-up. I will try it on a better GPU and decribe the deatiled results.

[Damian Suski]

I made the experiments and I haven't noticed the speedup in the case of batching. Time of computations increases proportionally to the batch size.

I have implemented the proper procedure and I was able to reproduce the discussed speedup results for the dummy example. The computations time was reduced from 186s on CPU to 42s on GPU. On a better graphics card the computations time is even shorter - 21s. Summing up, I'm satisfied with the results.

What still concerns me is that in Matlab the element-wise exp() is much faster than summing elements along two dimensions. For the analogical calculations in cuPy or pytorch, the situation seems to be the opposite. Can I place here the detailed results of my findings or should I start a new topic?