X-ray diffraction (XRD) Base line removal

Hello all,

I am a material engineering student and we have to analyse x-ray diffraction data. What happens with this type of analysis is that the data have a baseline, that must be removed for correct analysis. The image below demonstrates such line (all above zero and a slight upwards shift on left side...though sometimes can be a lot worse!).

What we do is, in Microsoft Excel, just select various points throughout the data (NOT ON ANY OF THE PEAKS, just the "flat" parts) and plot a graph with those selected points, determine it's function by trendline (whatever fits best...polynomial, power function...), create a column with f(x) values and subtract from our original y values.

Once this function is determined, we plot the "new" y values

As you can see, no more baseline (flat parts are at approximately zero, and no upward tendency towards the left).

I want to be able to plot using MATLAB, and have no difficulties doing such. However, when it comes to removing the baseline, I still have to use MS Excel to find a suitable fuction in MATLAB...is there anyway I could do that directly in MATLAB? Or is it easier to carry on doing in Excel, and just copying the function? Any help is greatly appreciated and I can clarify any confusion, thanks.

Answer 1

[Star Strider]

See if the detrend function will do what you want. (This appears to be a relatively straightforward application of it.)

More sophisticated approaches are available if necessary, specifically using a highpass (or bandpass) filter to eliminate the baseline variation, baselline drift (and high-frequency noise), if necessary.

[Herbert Middleton]

Detrend worked brilliantly!! Thanks ever so much :D

Yeah, for these types of graphs, I don't think I'll need to do any filter design.

[Star Strider]

As always, my pleasure!

They’re straightforward to design with Signal Processing Toolbox functions if you need them.

.

[Herbert Middleton]

Have been experimenting. Linear detrending isn't quite good enough, and any polinomial function above it ends up distorting quite a lot.

I ended up dividing the detrends through different portions of the spectrum, but I basically wanted to have a bit of "code" that could be universal (as I am going to be processing a lot of these graphs for my thesis), and am trying to avoid designing a filter, as I'm very much a "noob".

I don't suppose you know if there is a detrend for powerfunctions? (y=a*m^(b*x) )

[Star Strider]

The filter functions are quite easy to use. The signal (the spectrum here) needs to have constant x-axis differences between the various samples for filters to work optimally, however that’s easily done through interpolation (ideally the resample function) if necessary. The filters themselves are straightforward, since there are functions that design them automatically (such as bandpass), so it’s not absolutely necessary to know a lot about filters or signal processing to get a good result. I can certainly help with that if necessary.

Calculating the fft of the signal first will make the filter design easier, since the signal frequencies and low-frequency baseline variation band limits are readily apparent.

If necessary use resample first, before doing any other signal processing.

‘I don't suppose you know if there is a detrend for powerfunctions? (y=a*m^(b*x) )’

I’m not aware of any. It would likely be necessary to approximately identify the baseline first (possibly using the islocalmin function) and then estimate the parameters of the nonlinear regression function with lsqcurvefit, nlinfit, or similar functions, then do the subtraction. Another option would be to simply do a polynomial fit to it with polyfit and polyval, then subtract.

Highpass or bandpass filtering is likely easier.

.

[Herbert Middleton]

I did try to do by myself, but I'm having some difficulty getting the results as good as I can via Excel. I have included both orginial "raw" data and data that I "processed" using the way I said before, by subtracting the function "ytl=6427.8*(x).^(-0.859)". I have also included images of the respective graphs.

Thank-you, any help is greatly appreciated, but I also understand if I'm asking way too much.

[Star Strider]

Unfortunately, the filter approach failed because there are simply too many low-frequency components in the signal itself, and it was too difficult to distinguish them from the baseline variation.

So, I went with my second option, that being to identify the baselline using islocalmin, fitting a polynomial to it, and then subtracting that polynomial from the entire signal. I used a degree polynomial here, however anything greater than 1 and less than 7 could likely work. It would be necessary to experiment with different spectra to determine that.

The transfer function calculated from the Fourier transform of the detrended signal divided element-wise by the Fourier transform of tthe original signal could help identify the correct filter design. I did not go that far here, however I may experiment with it.

T1 = readtable('https://www.mathworks.com/matlabcentral/answers/uploaded_files/634170/rawdata.txt','VariableNamingRule','preserve');
x = T1.Var1;
y = T1.Var2;

figure
plot(x, y)
grid

TF = islocalmin(y, 'MinProminence',15, 'MinSeparation',60);

hold on
plot(x(TF), y(TF), '.r')
hold off
ylim([0 8000])
legend('Signal','Points Used To Identify Baseline Trend', 'Location','best')
title('XRD Spectrum With Identified Baseline')

p = polyfit(x(TF), y(TF),4)

p = 1×5

1.0e+03 * 0.0000 -0.0000 0.0032 -0.1004 1.5436

BL = polyval(p, x);

figure
plot(x, y-BL)
grid
title('Baseline-Detrended Signal')

.

[Herbert Middleton]

Wow, I can't thank you enough. I've just tested it out and have managed to do 15 XRD plots without any baseline really quickly !! Amazing!

[Star Strider]

As always, my pleasure!

Thank you!

(A Vote would be appreciated!)

.