Quickly finding maximum value of a large (~8,000,000) vector
2 views (last 30 days)
Show older comments
I was trying to see if I could speed up one of my functions so I ran it through profiler. The script loads big files and writes out smaller portions of the data for me to use later. Before writing however, I check to see if the data has any abnormally large values and if it does, I skip it. After looking at the profiler output, I saw that checking to see if there are any abnormally large values is what takes the most time by far (for 70 files, about 450 seconds out of 550 seconds). I never would have thought would have been the slowest part of the code so I'm wondering if I'm missing something that I could easily do to speed it up.
So here's the abridged version of the code:
for i = 1:nFiles % loop over files
Data = load_data(file(i)); % data is a structure with metadata and the actual
% data is in Data.trace which is 8640001x10x3
for s = 1:10 % loop over stations
badStation = 0;
for c = 1:3 % loop over components
% if one component is bad, skip the station
if max(abs(Data.trace(:,s,c))) > badThreshold
badStation = 1;
break;
end
end
if ~badStation
for c = 1:3 % write loop
write_data(Data.trace(:,s,c);
end
end
end
end
I've checked that memory isn't an issue as there's still 9 GB free during run time. Anything simple I can do to improve this?
Thanks in advance.
0 Comments
Accepted Answer
More Answers (2)
Walter Roberson
on 22 Jan 2013
You can vectorize a bit:
for a = 1 : 10
if max( reshape( abs(Data.trace(:,s,1:3)), [], 1) <= badThreshold
for c = 1 : 3
write_data(data.trace(:,s,c));
end
end
end
Jan
on 24 Jan 2013
Edited: Jan
on 24 Jan 2013
See FEX: anyExceed : This can test the values of the array on the fly, such that abs(data) is not created as a temporary variable. The testing stops early, when any element outside the wanted range is found. Therefore a positive test can be found without searching the complete array.
See Also
Categories
Find more on Data Type Identification in Help Center and File Exchange
Products
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!