Remove outliers until there are none left
2 views (last 30 days)
Show older comments
Dear community,
I apologize that I can't offer a better first try. I have a double array. I want to write a Loop for removing outliers from every column. The idea is: The code test for outliers, remove them, do it again, as long as there are outliers. If no outliers are found anymore, it should stop and give me back an double array without these outliers.
I tried it:
directory_name=uigetdir('','Ordner mit Messungen auswählen');
[nur_file_name,pfad]=uigetfile({'*.csv','csv-files (*.csv)';'*.*','all Files'},...
'Die csv-Files der Proben oeffnen (probe_001.csv=',[directory_name '/'], 'Multiselect', 'on');
nur_file_name=cellstr(nur_file_name);
nur_file_name=sort(nur_file_name);
filename=strcat(pfad,nur_file_name);
anzahl_files=size(filename,2);
for xy=1:anzahl_files
fid_in=fopen(char(filename(xy)),'r');
filename_s = matlab.lang.makeValidName(nur_file_name);
filename_s=string(filename_s);
filename_s = erase(filename_s,"_csv");
filename_s = erase(filename_s,"LiqQuant_");
filename_c=cellstr(filename_s);
for c=1:anzahl_files
filename_f{c}=extractBefore(filename_c{c},11);
end
filename_s=string(filename_f);
%----------------Import elements and intensity--------------------
clear element_RL
clear intens_RL
tmpImport = importdata(filename{xy},',');
element_RL = tmpImport.colheaders;
element_RL(:,[1 6 8 10 12 14 16 17 19 21 23 26 27 29 30 32 33 36 38 43 45 48 57 59 61 64 67 69 94 97 99 102 106 223 298 303 304 305])=[];
element_RL=string(element_RL);
[anzahl_zeile,anzahl_elemente]=size(element_RL);
intens_RL=tmpImport.data;
intens_RL(:,[1 6 8 10 12 14 16 17 19 21 23 26 27 29 30 32 33 36 38 43 45 48 57 59 61 64 67 69 94 97 99 102 106 223 298 303 304 305])=[];
[anzahl_runs,anzahl_elemente]=size(intens_RL);
%---------------remove outliers----------------
while intens_RL=ismember(NaN) %Wrong, because will run forever
threshold = mean(intens_RL)+3*std(intens_RL);
intens_RL(bsxfun(@(x, y) x > y, intens_RL, threshold)) = NaN; %outliers removing, set to NaN
end
I am sorry that my loop is so horrible, but I never wrote a while-loop before.
I am grateful for every small help
THANK YOU
0 Comments
Accepted Answer
Mathieu NOE
on 3 May 2023
hello
I updated the end of your code
the plot is for myself to see the difffences before / after thresholding (if hot spots are indeed removed)
%---------------remove outliers----------------
figure(1)
clim = [-5 7];
subplot(211),imagesc(log10(abs(intens_RL)),clim);colormap('jet');colorbar("vert")
title('before thresholding');
threshold = mean(intens_RL,1,'omitnan')+3*std(intens_RL,1,'omitnan');
ind = intens_RL>(ones(anzahl_runs,1)*threshold);
% ind = intens_RL>threshold; % works too
intens_RL(ind) = NaN;
subplot(212),imagesc(log10(abs(intens_RL)),clim);colormap('jet');colorbar("vert")
title('after thresholding');
4 Comments
More Answers (0)
See Also
Categories
Find more on NaNs in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!