Compute Quantization Error

This example shows how to compute and compare the statistics of the signal quantization error when using various rounding methods. Quantization occurs when a data type cannot represent a value exactly. In these cases, the value must be rounded to the nearest value that can be represented by the data type.

First, a random signal is created that spans the range of the quantizer object. Next, the signal is quantized, respectively, with rounding methods 'fix', 'floor', 'ceil', 'nearest', and 'convergent', and the statistics of the signal are estimated.

The theoretical probability density function of the quantization error is computed with the errpdf function, the theoretical mean of the quantization error is computed with the errmean function, and the theoretical variance of the quantization error is computed with the errvar function.

Create Uniformly Distributed Random Signal

Create a uniformly distributed random signal that spans the domain -1 to 1 of the fixed-point quantizer object q.

q = quantizer([8 7]);
r = realmax(q);
u = r*(2*rand(50000,1) - 1);
xi = linspace(-2*eps(q),2*eps(q),256);

Fix: Round Towards Zero

With 'fix' rounding, the probability density function is twice as wide as the others. For this reason, the variance is four times that of the others.

q = quantizer('fix',[8 7]);
err = quantize(q,u) - u;    
f_t = errpdf(q,xi);
mu_t = errmean(q);
v_t  = errvar(q);

qerrordemoplot(q,f_t,xi,mu_t,v_t,err)

Estimated   error variance (dB) = -46.8586
Theoretical error variance (dB) = -46.9154
Estimated   mean = 7.788e-06
Theoretical mean = 0

The theoretical variance is eps(q)^2/3 and the theoretical mean is 0.

Floor: Round Towards Negative Infinity

'floor' rounding is often called truncation when used with integers and fixed-point numbers that are represented using two's complement notation. It is the most common rounding mode of DSP processors because it requires no hardware to implement. 'floor' does not produce quantized values that are as close to the true values as 'round' will, but it has the same variance. Using 'floor', small signals that vary in sign will be detected, whereas in 'round' they will be lost.

q = quantizer('floor',[8 7]);
err = quantize(q,u) - u;    
f_t = errpdf(q,xi);
mu_t = errmean(q);
v_t = errvar(q);

qerrordemoplot(q,f_t,xi,mu_t,v_t,err)

Estimated   error variance (dB) = -52.9148
Theoretical error variance (dB) = -52.936
Estimated   mean = -0.0038956
Theoretical mean = -0.0039062

The theoretical variance is eps(q)^2/12 and the theoretical mean is -eps(q)/2.

Ceil: Round Towards Positive Infinity

q = quantizer('ceil',[8 7]);
err = quantize(q,u) - u;    
f_t = errpdf(q,xi);
mu_t = errmean(q);
v_t  = errvar(q);

qerrordemoplot(q,f_t,xi,mu_t,v_t,err)

Estimated   error variance (dB) = -52.9148
Theoretical error variance (dB) = -52.936
Estimated   mean = 0.0039169
Theoretical mean = 0.0039062

The theoretical variance is eps(q)^2/12 and the theoretical mean is eps(q)/2.

Round: Round to Nearest; In a Tie Round to Largest Magnitude

'round' is more accurate than 'floor', but all values smaller than eps(q) get rounded to zero and are lost.

q = quantizer('nearest',[8 7]);
err = quantize(q,u) - u;    
f_t = errpdf(q,xi);
mu_t = errmean(q);
v_t  = errvar(q);

qerrordemoplot(q,f_t,xi,mu_t,v_t,err)

Estimated   error variance (dB) = -52.9579
Theoretical error variance (dB) = -52.936
Estimated   mean = -2.212e-06
Theoretical mean = 0

The theoretical variance is eps(q)^2/12 and the theoretical mean is 0.

Convergent: Round to Nearest; In a Tie Round to Even

'convergent' rounding eliminates the bias introduced by ordinary 'round' caused by always rounding the tie in the same direction.

q = quantizer('convergent',[8 7]);
err = quantize(q,u) - u;    
f_t = errpdf(q,xi);
mu_t = errmean(q);
v_t  = errvar(q);

qerrordemoplot(q,f_t,xi,mu_t,v_t,err)

Estimated   error variance (dB) = -52.9579
Theoretical error variance (dB) = -52.936
Estimated   mean = -2.212e-06
Theoretical mean = 0

The theoretical variance is eps(q)^2/12 and the theoretical mean is 0.

Compare Nearest and Convergent Rounding

The error probability density function for convergent rounding is difficult to distinguish from that of round-to-nearest by looking at the plot.

The error probability density function of convergent is

f(err) = 1/eps(q),  for -eps(q)/2 <= err <= eps(q)/2, and 0 otherwise

while the error probability density function of round is

f(err) = 1/eps(q),  for -eps(q)/2 <  err <= eps(q)/2, and 0 otherwise

The error probability density function of convergent is symmetric, while round is slightly biased towards the positive.

The only difference is the direction of rounding in a tie.

x = (-3.5:3.5)';
[x convergent(x) nearest(x)]

ans = 8×3

   -3.5000   -4.0000   -3.0000
   -2.5000   -2.0000   -2.0000
   -1.5000   -2.0000   -1.0000
   -0.5000         0         0
    0.5000         0    1.0000
    1.5000    2.0000    2.0000
    2.5000    2.0000    3.0000
    3.5000    4.0000    4.0000