How to replace values in a very large array

Hi

I have an array idx_Laser which is very large (~1 billion values). Whenever a certain condition is fulfilled for another array tdcRelTime I want to replace the values in my idx_Laser array.

The trivial solution would probably be:

for xx = 1:length(tdcRelTime)
    if tdcRelTime(xx) < 1.5e10
        idx_Laser(idx_Laser==xx) = 0;
    end
end

However, as both arrays are quite big, this takes extremely long.

Is there a solution for that problem that is computionally more efficient?

I thought about iterating only through a part of the idx_Laser array, as this is sorted by the size of the values, but I couldnt come up with a good solution.

[dpb]

Can you hold the arrays in memory?

If that is the case, it's already vectorized for you --

ix=(tdcRelTime < 1.5e10);
idx_Laser(ix) = 0;

If the arrays are being held in virtual memory and you're paging to address them in the above code, that will be slow.

In that case, investigate tall arrays...

[Jonathan Weber]

Hey :)

Thanks for the comment.

Yes I can hold the arrays in memory. But

idx_Laser(ix) = 0;

is not really what I was searching for.

The Problem is in the line

idx_Laser(idx_Laser==xx) = 0;

Because sweeping through that array is what takes so long.

I will take a look at these tall arrays though.

[Jonathan Weber]

What I came up with is a bit confused, but the performance is not so bad.

As I mentioned the array idx_Laser is sorted and the longest bin has "noElements" values.

Maybe someone has a better idea:

for zz = 1:length(tdcRelTime)
    if tdcRelTime(zz) < 1.5e10
        idx = find(idx_Laser == zz, 1);
        for xx = idx:idx + noElements
            if idx_Laser(xx) == zz
                idx_Laser(xx) = NaN;
            end
        end
    end
end

Answer 1

[Mohammad Sami]

You can modify what @dpb suggested as follows.

xx=find(tdcRelTime < 1.5e10);
idx_Laser(ismember(idx_Laser,xx)) = 0;

[Jonathan Weber]

yes, thats much faster than my approach...

Thanks to both of you!