Attempting to detrend and remove noise from surface roughness data

I am currently trying to remove the waviness from my data and also remove the noise from it to get the surface roughness profile out of the data set that I need. I attached what the graph looks like after attempting to use the detrend function on the data, but it still has significant waviness to the data set. Any help would be appreciated. Below is the code that I am currently working with. I attmepted to use a fft to remove the noise, but the data points are not equally spaced.

clc 
clear all 
close all 
%Clear all variables and open figures and clear the command window 

dt = 0.001 
t = 0:dt:1; 

[file,path] = uigetfile(); %Opens the file browser to select the desired file

imported_data = readtable(strcat(path,file)); %Reads .DAT file into a table 

x_height = imported_data{:,2}; %Saves the first column of the data to the variable 
z_distance = imported_data{:,1}; %Saves the second column of the data to the variable 

x_height_corrected = detrend(x_height); %Removes the negative slope of the data

% n = length(z_distance); %finds length of data set
% fhat = fft(x_height_corrected, n); %Computes the fast foureir transform for data set
% PSD = fhat.*conj(fhat)/n; %Power spectrum (power per freq) 
% freq = 1/(-1.96848491018642e-05*n)*(0:n); 
% L = 1:floor(n/2); 
% f = (0:n-1)/n; 

% Plots the data that is imported and detrended 
plot(z_distance, x_height_corrected)
ylabel('Height (mm)')
xlabel('Distance (mm)')

[Image Analyst]

Please attach screenshots in PNG form using the frame or paperclip icon. If they are a .fig file, clicking on it launches a whole, new, separate second instance of MATLAB which takes time. If you put in a picture here directly with the frame icon we can avoid all that trouble.

Answer 1

[Mathieu NOE]

hello

my prefered function for smoothing data is smoothdata , but there are other options

example below :

clc
close all

Fs = 1000;
samples = 1000;
dt = 1/Fs;
t = (0:samples-1)*dt;
y = square(2*pi*3*t) + 0.1*randn(size(t));


% %%%%%%%%%%%%%%%%
figure(1)
N = 25;
ys = smoothdata(y, 'gaussian' , N);
plot(t,y,t,ys);legend('Raw','Smoothed');
title(['Data samples at Fs = ' num2str(round(Fs)) ' Hz / Smoothed with smoothdata' ]);

% %%%%%%%%%%%%%%%%
figure(2)
N = 25;
ys = medfilt1(y, N,'truncate');
plot(t,y,t,ys);legend('Raw','Smoothed');
title(['Data samples at Fs = ' num2str(round(Fs)) ' Hz / Smoothed with medfilt1' ]);
grid on

%%%%%%%%%%%%%%%%
figure(3)
ys = sgolayfilt(y,1,21);
plot(t,y,t,ys);legend('Raw','Smoothed');
title(['Data samples at Fs = ' num2str(round(Fs)) ' Hz / Smoothed with sgolayfilt' ]);
grid on

%%%%%%%%%%%%%%%%
NN = 2;
Wn = 0.25;
[B,A] = butter(NN,Wn);

figure(4)
ys = filtfilt(B,A,y);
plot(t,y,t,ys);legend('Raw','Smoothed');
title(['Data samples at Fs = ' num2str(round(Fs)) ' Hz / Smoothed with butterworth LP' ]);
grid on