Good value for root mean square error?

Hello,

I was working on the calculation of some RMSE values. I have some force trials in the time domain (normal force, and tangential force). I wanted to see if the reproduction of these forces was accurate when a robotic platform reproduces the same force we apply. I calculated the RMSE for all the trials I have (450).

The mean RMSE for the normal force was 0.11 (SD=0.08) and for the tangential force was 0.15 (SD=0.1). Do you think that these are decent values to conclude that the reproduction of the force was quite accurate?

Thank you very much in advance for your help!

ps: I also computed the root mean square range error that was found to be even lower. Does it also make sense to compute the range error?

[dpb]

All depends. If mean(appliedForce)=1000, would seem error would be pretty small. If, otoh, mean(appliedForce)=1, "not so much". We've no way to judge.

[imo88]

Dear dpb, thank you very much for your reply! It helped a lot! I was thinking about how to calculate the range for the RMSE but I'm not sure if my function is correct.

let's say we have one trial in active touch with 2 rows (1st for normal force, and 2nd for tangential force) = F1. the same is for passive touch = F2.

first, I run the difference between the signals: Fdif=F2-F1;

then the RMSE: RMSE=sqrt(mean(Fdif.^2,2));

I'm not sure if these functions are correct for the range:

RMSE_min_range=RMSE./abs(min(F1,[],2))

RMSE_max_range=RMSE./abs(max(F1,[],2))

I would really appreciate it to read your opinion on this..

Thank you very much in advance!

[dpb]

Matlab has builtin function rms, might as well use it... :)

That will give you a fractional deviation of the rms value as compared to the min/max of the observations, yes. That's certainly one measure (or two, actually :) ); how useful that measure is would likely depend upon just how much difference there is between the min/max values as well as the absolute values thereof. IF the min value were very near zero, say, the first of those two values might numerically be very large since 1/x-->Inf as x-->0 and you're aware of the shape of that curve.

Hence, often folks use the mean of the observations but again if that is a zero-mean process the same effect may occur.

Again without the actual values and some more background, it's not possible to say much except in generalities like the above.

The answer is there is is no one unequivocal answer to the question posed, "it depends"...

[Jonathan]

rms function was about 15 times slower than a simple manual computation! My simple manual computation:

rmse = sqrt( (length(Y)^-1).*sum((Y_hat - Y).^2) )

where Y is your target data, and Y_hat is your estimate.

Answer 1

[John D'Errico]

As dpb said, it is impossible to know if some arbitrary value for RMSE is good or bad. Only you know if it is good, because only you can know how much noise you would expect in the data.

The point is, when you use a model on some data that generates an RMSE, there are TWO components to the error, noise and lack of fit.

Any model will not be able to predict random noise in the data, so the predictive capability of the model can be no better than that noise. If it is better, then you are overfitting the noise. This is a bad thing to do.

At the same time, a model is just a model. It is an approximation to the real thing, with some aspects of reality that are left out. In some cases, a model is just a metaphor for the real system under study, so your model arises from some completely different system, but one that you hope behaves similarly to the real world case under study. In either case, a model will typically have some degree of lack of fit, since it is never a perfect model of the process. There is always something you have not included.

So we think of the residuals for any model to a physical system as something composed of both noise and lack of fit. These two components mix together. If the model is a good one, and the lack of fit is very small, then all we will see after the model is removed is noise. So our hope is the RMSE will be roughly the same as the underlying noise standard deviation, something that you SHOULD have some feel for if you are modeling any process.

So if the RMSE is significantly very small, more so than we would expect, then you are overfitting the problem. That is bad, since your model is now compensating for (chasing after) random noise.

If the RMSE is significantly larger than you would have expected, then this is again a problem, since it indicates there is a potential problem in the model. The model is inadequate to represent the system under study, or you did a poor job of estimating anything that must be estimated in the model.

But the point is, RMSE means NOTHING unless you have some a-priori knowledge about the process. You must have some expectation as to what is acceptable for the problem you are studying. IF you have no clue, it is impossible for someone else to know, since did not develop the model, we did not generate the data under study.