How can I spped up my shuffle test for a large array?

I'm working on shuffling a large array(Jy) which is about 400 (numtrial)x 5(numtaper) x 80(numf) x 96(npair). I need to shuffle the first dimension for each column. My way is: 1.generate the index array by randperm, e.g. index is a 400 x 500(nPerm) matrix; 2.repmat the index array up to 400 x 5 x 80 x 96 x 500; 3.repmat the original array up to 400 x 5 x 80 x 96 and index the original large array by the index array to fully shuffle it. But it costs a large amount of time at the third step which is to index a huge array. Do you have any idea to improve this and save time?

%first step
I = zeros(numtrial,1,1,1,nPerm);
for iP = 1:nPerm
    I(:,1,1,1,iP) = randperm(numtrial);
end
%second step
i = numtrial*((1:1:(numtaper*numf*npair*nPerm))-1);
I = repmat(I,1,numtaper,numf,npair,1)+reshape(i,1,numtaper,numf,npair,nPerm);
%third step
Jyrand = repmat(Jy,1,1,1,1,nPerm);
Jyrand = Jyrand(I);%this line costs too much time

Answer 1

[Arjun]

Hi @Meng Wang,

I understand that you have a large multidimensional array, and you want to shuffle it along the first dimension.

To optimize this process, we can avoid the need to create a large, replicated version of the original array and instead focus on directly shuffling the desired dimension.

You can optimize the process by creating shuffled indices along the first dimension and then apply these indices directly to shuffle the array without replicating it. This method will also avoid creating large intermediate arrays, thereby saving memory and reducing computational overhead.

Kindly refer to the following sample code to understand the process in detail:

% Parameters
numtrial = 400;
numtaper = 5;
numf = 80;
npair = 96;
nPerm = 5;

% Original array Jy
Jy = rand(numtrial, numtaper, numf, npair); % Example data

% Initialize the array for shuffled data
Jyrand = zeros(numtrial, numtaper, numf, npair, nPerm);

% Generate shuffle indices for each permutation
for iP = 1:nPerm
    % Generate random permutation indices for the first dimension
    shuffleIdx = randperm(numtrial);
    
    % Directly apply the permutation to shuffle the first dimension
    Jyrand(:,:,:,:,iP) = Jy(shuffleIdx, :, :, :);
end

% Jyrand now contains the shuffled versions of Jy across the first dimension

I hope this will help!