Main Content

Filter outliers using Hampel identifier

The `dsp.HampelFilter`

System object™ detects and removes the outliers of the input signal by using the Hampel
identifier. The Hampel identifier is a variation of the three-sigma rule of statistics that is
robust against outliers. For each sample of the input signal, the object computes the median
of a window composed of the current sample and $$\frac{Len-1}{2}$$ adjacent samples on each side of current sample. *Len* is
the window length you specify through the `WindowLength`

property. The
object also estimates the standard deviation of each sample about its window median by using
the median absolute deviation. If a sample differs from the median by more than the threshold
multiplied by the standard deviation, the filter replaces the sample with the median. For more
information, see Algorithms.

To filter the input signal using a Hampel identifier:

Create the

`dsp.HampelFilter`

object and set its properties.Call the object with arguments, as if it were a function.

To learn more about how System objects work, see What Are System Objects?

returns a Hampel
filter object, `hampFilt`

= dsp.HampelFilter`hampFilt`

, using the default properties.

sets the `hampFilt`

= dsp.HampelFilter(`Len`

)`WindowLength`

property to `Len`

.

sets the `hampFilt`

= dsp.HampelFilter(`Len`

, `Lim`

)`WindowLength`

property to `Len`

and the
`Threshold`

property to `Lim`

.

`hampFilt = dsp.HampelFilter(11,2);`

specifies properties using `hampFilt`

= dsp.HampelFilter(`Name,Value`

)`Name,Value`

pairs. Unspecified properties
have default values.

To use an object function, specify the
System object as the first input argument. For
example, to release system resources of a System object named `obj`

, use
this syntax:

release(obj)

For a given sample of data, *x _{s}*,
the algorithm:

Centers the window of odd length at the current sample.

Computes the local median,

*m*, and standard deviation,_{i}*σ*, over the current window of data._{i}Compares the current sample with

*n*×_{σ}*σ*, where_{i}*n*is the threshold value. If $$\left|{x}_{s}-{m}_{i}\right|>{n}_{\sigma}\times {\sigma}_{i}$$, the filter identifies the current sample,_{σ}*x*, as an outlier and replaces it with the median value,_{s}*m*._{i}

Consider a frame of data that is passed into the Hampel filter.

In this example, the Hampel filter slides a window of length
5 (*Len*) over the data. The filter has a threshold
value of 2 (*n _{σ}*). To
have a complete window at the beginning of the frame, the filter algorithm
prepends the frame with

Current sample:

*x*= 0._{s}Window of data:

*win*= [0 0 0 0 1].Local median:

*m*= median([0 0 0 0 1]) = 0._{i}Median absolute deviation: $$ma{d}_{i}=\mathrm{median}\left(\left|{x}_{i-k}-{m}_{i}\right|,\dots ,\left|{x}_{i+k}-{m}_{i}\right|\right)$$. For this window of data, $$mad=\mathrm{median}\left(\left|0-0\right|,\dots ,\left|1-0\right|\right)=0$$.

Standard deviation:

*σ*=_{i}*κ*×*mad*= 0, where $$\kappa =\frac{1}{\sqrt{2}{\mathrm{erfc}}^{-1}\left(1/2\right)}\approx 1.4826$$._{i}The current sample,

*x*= 0, does not obey the relation for outlier detection._{s}$$\left[\left|{x}_{s}-{m}_{i}\right|=0\right]>\left[\left({n}_{\sigma}\times {\sigma}_{i}\right)=0\right]$$

Therefore, the Hampel filter outputs the current input sample,

*x*= 0._{s}

Repeat this procedure for every succeeding sample until the
algorithm centers the window on the $${\left[End-\frac{Len-1}{2}\right]}^{\text{th}}$$ sample, marked
as `End`

. Because the window centered on the last $$\frac{Len-1}{2}$$ samples cannot
be full, these samples are processed with the next frame of input
data.

Here is the first output frame the Hampel filter generates:

The seventh sample of the appended input frame, 23, is an outlier. The Hampel filter replaces this sample with the median over the local window [4 9 23 8 12].

[1] Bodenham, Dean. “Adaptive Filtering and Change Detection for Streaming Data.” PH.D. Thesis. Imperial College, London, 2012.

[2] Liu, Hancong, Sirish Shah, and Wei Jiang. “On-line outlier detection and
data cleaning.” *Computers and Chemical Engineering*. Vol. 28,
March 2004, pp. 1635–1647.

[3] Suomela, Jukka. Median Filtering Is Equivalent to Sorting, 2014.