FFT slowdown even after workspace reset

I'm experiencing behavior with the fft() function that is causing me to have to restart Matlab between executions of a long script that is both processing and memory intensive and requires, among other things, millions of fft's on the CPU and GPU. If I run bench() prior to running the script, my computer (i9-13950HX w/64GB of ram, running Windows 11, Matlab R2024a) clocks in very fast. After I run my script, all performance metrics are basically identical except for fft() which clocks >10x slower than before.

No matter what I do to the workspace (clear all, clear classes, clear functions, close all hidden force, clc, reset(gpuDevice), etc.), or the fft planner I cannot bring the performance of fft() back to what it was before execution of the script.

Am I overlooking anything that could reset the performance of the fft short of restarting Matlab itself? I would like to let the computer loop over a bunch of datasets but right now the slowdown in the fft is making this very inefficient. I am currently considering calling the Matlab engine from Python so that I can restart it between script calls to prevent this. I am running Matlab on 2024a and may be able to update to 2024b but cannot upgrade past 2024b.

[dpb]

I believe probably only Mathworks can address this. Contact official support request at <Product Support Page>

[Paul]

@Timothy

If the culprit is identified, would you mind posting back here as others may be interested in the findings.

[Walter Roberson]

I note that the i9-13950HX has 8 performance cores and 16 efficiency cores. I wonder if later in the run, most of the computation is being shunted to the efficiency cores?

[Timothy]

@dpb, thanks I might give that a shot.

@Paul, I found the culprit function, which calls another function which splits a bunch of large complex double precision N x M arrays into three dimensional 32 x M x (N/32) arrays, computes the FFT in the column dimension, multiplies with the conjugate of another 32 x M x (N/32) array, inverse Fourier transforms & normalizes to create a bunch of cross-correlations. However, it seems that if I isolate this sub function and run it a bunch of times by itself it doesn't affect the performance of the fft() function. Still, if I delete the sub function from the larger function I also have no reduction in performance, so I know that it is tied to it somehow. If I can get more specific or create a simple toy function that creates the fft performance loss I'm seeing I will post it here.

@Walter Roberson: Maybe this is happening, I don't know how to monitor what cores are used. Matlab is the main process on my computer however, and this slowdown can be created in under 10 minutes of operations by iterating the function described above in a loop. If I call the bench in each iteration of the loop and store the FFT time I can watch it slow down each iteration starting with the 4th (I get about 25 iterations in 10 minutes, by which time the FFT score has gone from ~0.15 seconds to ~1 second, and keeps slowing down the more times I call the function).

[Paul]

@Timothy

fft used to have a memory leak, but that was fixed. Maybe a similar, yet different, issue has reared up since then. Matlab 2020a/b fft function memory leak - MATLAB Answers - MATLAB Central

Also, this thread How can I solve memory leak in fft? - MATLAB Answers - MATLAB Central, which isn't really about a memory leak, discusses memory management with fft and seems like it might be on point based on the problem description. Maybe the

fftw(wisdom,[])

command is worth a try. Though it sounds like all of the FFTs are 32-point, so maybe this isn't the issue.

[Walter Roberson]

@Paul I think you mean

fftw('wisdom',[])

[dpb]

The doc for <fftw> is may be a little confusing for that case -- it does show the form as @Paul used, but uses wisdom as a place holder for either 'swisdom' or 'dwisdom'. 'wisdom' alone isn't documented but doesn't error on local system...but it does need to be a character string (or a variable that would contain the string).

[Paul]

Yes, as dpb suggested I was using wisdom as a variable that would have the value ‘dwisdom’ or ‘swisdom’ as appropriate.

[dpb]

Was going to comment that was good spelunking @Paul to find the thread and refer to fftw; something like that was what I had in mind that would get reset on restart but wasn't affected by normal memory clearing, etc., ...back in days of yore before MATLAB and had to use the libraries directly in FORTRAN (before Fortran days, too) I knew about fftw but it had completely slipped my mind in the ensing 40 years. Of course, that also predated having multi-cores, GPUs, parallel computing TBs so one had far more direct knowledge of what was going on inside.

[Timothy]

@Paul, @dpb, @Walter Roberson I'm sorry I wasn't specific enough in my original post, I have definitely tried reseting the wisdom in fftw for both single and double precision and it did not affect anything.

[Steven Lord]

You indicate you found the "culprit function". After running that, is it FFT calls in isolation that slow down, or is it subsequent runs of the culprit function as a whole?

You also stated "large complex double precision N x M arrays into three dimensional 32 x M x (N/32) arrays" -- how large is "large" in this context? What are typical values for N and M for the data on which you're operating?

I think without seeing that culprit function it's likely going to be difficult to determine what's going on. Please send it to Technical Support so they can work with the developers to understand the problem and try to determine the root cause of the slowdown.

[dpb]

I figured from the git-go this would take the developers being able to poke at the innards.

Besides the isolation of the given function, that it is something else being done to the state of the GPU on a restart before recovers performance is curious...

First of all will whether it is reproducible on a Mathworks machine or is something unique to @Timothy's particular system. Not too likely, probably, but ya' never know.

[Timothy]

@Steven Lord, the arrays are ~6000 x 16000 complex double precision matrices. Happy to contact tech support but I posted here to check if I'm missing something obvious which is frequently the case. I'm trying to drill down a bit further to see if I can reproduce the problem with a simpler script before I contact tech support.

[dpb]

It might be interesting/useful to see if the symptom were to go away for some smaller size?

I'd suggest if were able to create such a sample case to go ahead and post it here -- those who do have the TB and could run it (I don't) could also see if it is reproducible on other systems.

[Steven Lord]

After running your culprit function is it FFTs on the CPU that are slow, FFTs on the GPU, or both?