Fit a custom function to data with errors in all dimensions (i.e. nonlinear total least square, deming, odrpack?)

Hi,

I have been looking for a Matlab function that can do a nonlinear total least square fit, basically fit a custom function to data which has errors in all dimensions. The easiest case being x and y data-points with different given standard deviations in x and y, for every single point. This is a very common scenario in all natural sciences and just because most people only know how to do a least square fit with errors in y does not mean it wouldn't be extremely useful. I know the problem is far more complicated than a simple y-error, this is probably why most (not even physicists like myself) learned how to properly do this with multidimensional errors.

I would expect that a software like matlab could do it but unless I'm bad at reading the otherwise mostly useful help pages I think even a 'full' Matlab license doesn't provide such fitting functionality. Other tools like Origin, Igor, Scipy use the freely available fortran package "ODRPACK95", for instance. There are few contributions about total least square or deming fits on the file exchange, but they're for linear fits only, which is of little use to me.

I'd be happyfor any hint that can help me out, but I kinda lost the mathematical background to just go and implement this fitting/optimization functionality myself without at least a good paper telling me how to.

kind regards

[John D'Errico]

Nonlinear total least squares can be nasty as hell to do. Don't try to write it yourself.

[Andreas Brand]

Hi John,

that was my thought exactly after looking at some papers too. It's too complicated/error prone to just try and implement something myself. Most software just uses ODRPACK95 but Matlab has no implementation of it or any proprietary alternative (as far as I can tell). I find that a bit disappointing, I have for quite some time now.

That doesn't change the fact that it would make a lot of sense to take the x-errors of the measured data into account when fitting models, on the contrary, I consider it crucial to take them into account as that can drastically improve the fit - obviously it is more "correct" to consider those errors than just to ignore them.

At the moment I'll have to interrupt my entire workflow and projects and export data to then fit in Scipy for instance. I tried running scipy.odr from Matlab but was unsuccessful. I need something automatic for this!

Do you have any other suggestion?

kind regards Andreas

[Andreas Brand]

does nobody have an implementation of this for matlab? I'd also take a Matlab wrapper for scipi.odr of course, would be just as helpful.

Maybe I open a new question regarding such a wrapper, because I can't get it to run myself, but Matlab's ability to call python functions seems more limited than I thought (never use it though).

[Andreas Brand]

new matlab release is out and.... nothing on total least squares, not to mention weighted total least squares. I guess I'll have to switch to python with this too. Less and less reasons remain for me to use matlab :-/

[Matthew Cooper]

I've submitted a feature request for this previously. You should too (and anyone reading this). Note that the bionformatics toolbox has something called a 'combined' error model, which is similar but not exactly the same as your 'total least squares' error model. It's better. Total least squares does not account for correlated errors in the x-y or higher dimensions. Total least squares will give a biased fit if the errors are correlated. If you don't think the errors are correlated, why not use principal components? However, if you think they're correlated, then to get an unbiased fit, you need to minimize a different distance (not orthogonal), but the distance depends on the error structure, and requires iteration. Look up York (2004) for a simple implementation in two dimensions. There are file exchange submissions for this. For higher dimensions, you can find methods in various disciplines ranging from econometrics to system identification. I'd bet the econometrics toolbox has something that works. Hopefully the Mathworks engineers will extend the bionformatics error model options to other toolboxes, like statistics and machine learning, where this functionality is really needed!

Answer 1

[Bjorn Gustavsson]

Well if you want some tools for matlab they aren't too difficult to find: What a quick search at the file exchange offers up

HTH

[Andreas Brand]

Hi Bjorn,

of course I searched the file exchange before posting here. The reason is, that all contributions I could find there only work for fitting a line - which is not useful for me. I need to be able to do a fit using user definable functions/models.

All other major scientific tools that are being used for fitting support this out of the box. For Matlab I can't even find a third party implementation. Kind of frustrating but as John stated above, this is very error prone so implementing it yourself if you have no considerable time for the task might proof too complex.

If you can point me to a file that can do nonlinear total least squares, for a user definable model, I'd really appreciate it.

kind regards Andreas

[Bjorn Gustavsson]

Well, not to get chippy or anything but one of the demo-scripts in the first listed contribution compares a ordinary LSQ-fitting of a 2nd degree polynomial and a TLSQ-fitting of a 2nd degree polynomial, to me it appears possible to modify that to a more general parameterized model-function - I might be wrong, but...

HTH

[Walter Roberson]

? you can define fittype() using a string expression or using an anonymous function.

Even cftool allows you to input a custom model.

[Andreas Brand]

@ Bjorn, no worries, I'm not offended, I see your point but I looked through these contribution and the documentation seems very poor (at least to my eye). Various variables unexplained, who am I to say if it'd be mathematically correct to change any of this. Some correlation coefficient decision determines the output... this could be anything for an arbitrary, nonlinear model. I'm talking about this file now, or did you mean another one? total-least-squares (fit_2D)

If John (D'Errico) thinks it's no good advice to try and implement it yourself, I'm tempted to believe him. Having (passively) known him/his work for years I consider it good advice to believe his statements. I'm not so sure it makes sense to fiddle with some code one doesn't understand (or that is poorly documented).

@ Walter, sure you can define custom models for all kinds of fitting & optimization functions in matlab, but I haven't found a single Matlab function (built in or 3rd party) allowing custom models for fitting and then using total least squares, instead of ordinary least squares (only considering the y-distance). If you can point one out that would be greatly appreciated.

so bottom line. I still consider this to be a valid request. I see no obvious solution here yet the necessity to consider x and y errors when fitting models seems obvious to me. I've contacted Mathworks and they told me the same thing as some of you: "There is something on the fileExchange." I find this nonchalant answer by MathWorks a bit frustrating as it tells me they didn't even understand my request, because I was never talking about simple line fits. Every other major tool can to this (especially the expensive ones like Matlab). As a scientist who likes to use Matlab as a set of very accessible high-level tools this left me a bit speechless. I got no response regarding my question if this could be added in a future version.

[Bjorn Gustavsson]

I agree with your general opinions, however, I'm a bit more optimistic about this (easy to be since I wont do any of this work for money or research so failures wont bite me). The file I looked at was demoNRM.m, and that seemed possible to modify to more complicated model-functions. I have to admit that I've not looked under the hood of this tool so I have no idea whether it is general enough, but in the case I would start working on TLS I'd start looking at this (and then perhaps other) file exchange contribution(s).

HTH