Removing spikey data from a sensor readout.

I have a loop which reads out a sensor, once in a while the data is completely off, and gives a spike. I would like to remove those spikes.

The data given moves from -180 to 180 degrees, so when my measurement moves from -180 to 180 this should not be filtered (as this is normal).

However, if it moves from -180 to 90 then this counts as a spike.

It's not possible to do data processing after getting every readout, ideally there should be a filter of some sort in the loop itself.

Any ideas?

Answer 1

[Image Analyst]

Do you have the Signal Processing Toolbox? If so, you could use the median filter. If you want to replace spikes with the median value around them, then compute the median and subtract it from the original and take the absolute value. Then replace those elements with a high value with the median.

medianSignal = medfilt1(signal, 7);
diffSignal = abs(signal - medianSignal);
spikes = diffSignal > 100; % Whatever...
% Replace
fixedSignal = signal; % Initialize
fixedSignal(spikes) = medianSignal(spikes);

Or you can use the Savitzky-Golay filter, sgolayfilt(). A demo is attached.

If this is not right for you, then post your data - both a picture and the actual data in a txt or .mat file.

[luc]

I left my reply below.

[Image Analyst]

Does it work? You posted it as an "Answer" instead of as a comment here, which means it's supposed to solve your original question.

[luc]

Sorry , no it does not, I think I must have written in the wrong box.

[Image Analyst]

luc's "Answer" moved here.

Hmm, I don't think that would work. I want the spikes to either just be nan's and ill get a new value, or I want them averaged with data that came before the spike.

here's the data and a pic of the data.

This is after 1000 measurements, however, the data keeps on coming in. This means I can't use a method that averages based on the entire matrix as this would slow down the loop when we get to higher numbers.

So I would probably need to grab the last few values and compare those with the rest... The spikes you see around 120-140 are random. around 80 you can see a sudden shift in values and at 880 or so you can see a switch from +180 to -180.

clc;    % Clear the command window.
close all;  % Close all figures (except those of imtool.)
clear;  % Erase all existing variables. Or clearvars if you want.
workspace;  % Make sure the workspace panel is showing.
format long g;
format compact;
fontSize = 25;

% Make a noisy sine wave signal
for x=1:100;

period = 50
y = sin(2*pi*x/period);
noiseAmplitude = 0.8;
y = y + noiseAmplitude * rand;
yplot(x)=y;
xplot(x)=x;
subplot(2,1,1);
plot(xplot, yplot, 'b-', 'LineWidth', 2);
grid on;
title('Noisy Signal', 'FontSize', fontSize);
% Enlarge figure to full screen.
set(gcf, 'Units', 'Normalized', 'OuterPosition', [0 0 1 1]);
% Give a name to the title bar.
set(gcf, 'Name', 'Demo by ImageAnalyst', 'NumberTitle', 'Off') 

% Now smooth with a Savitzky-Golay sliding polynomial filter
windowWidth = 27
polynomialOrder = 3
smoothY = sgolayfilt(y, polynomialOrder, windowWidth);
subplot(2,1,2);
plot(x, smoothY, 'b-', 'LineWidth', 2);
grid on;
title('Smoothed Signal', 'FontSize', fontSize);
end

This is the filter you attached, only edited.

It does not work but if you could get this setup to work I would know how to implement it.

[Image Analyst]

That's because you messed it up in several ways. You have y inside the loop and are trying to run sgolayfilt() on just a single value of y instead of the whole array. Don't use a loop. And you misused xPlot and yPlot by assigning an index, and you didn't add random numbers to each element. You added the same random number to all elements.

I fixed your code and include it here:

clc;    % Clear the command window.
close all;  % Close all figures (except those of imtool.)
clear;  % Erase all existing variables. Or clearvars if you want.
workspace;  % Make sure the workspace panel is showing.
format long g;
format compact;
fontSize = 25;

% Make a noisy sine wave signal
x=1:100;
period = 50
y = sin(2*pi*x/period);
noiseAmplitude = 0.8;
y = y + noiseAmplitude * rand(size(y));
yplot=y;
xplot=x;
subplot(2,1,1);
plot(xplot, yplot, 'b-', 'LineWidth', 2);
grid on;
title('Noisy Signal', 'FontSize', fontSize);
% Enlarge figure to full screen.
set(gcf, 'Units', 'Normalized', 'OuterPosition', [0 0 1 1]);
% Give a name to the title bar.
set(gcf, 'Name', 'Demo by ImageAnalyst', 'NumberTitle', 'Off') 

% Now smooth with a Savitzky-Golay sliding polynomial filter
windowWidth = 27
polynomialOrder = 3
smoothY = sgolayfilt(y, polynomialOrder, windowWidth);
subplot(2,1,2);
plot(x, smoothY, 'b-', 'LineWidth', 2);
grid on;
title('Smoothed Signal', 'FontSize', fontSize);

[luc]

Thanks!