polynomial doesnt fit with original data

Hi, I give Matlab 2009a vectors X and Y. I plot(X,Y) and I want to fit a polynomial on it. on the plot, I can see a good fit with 6th order polynomial. Then I save the coefficients of polynomial (say vector A) and again plot(polyval(A,X). But they dont match with the original curve. They are not even close!

these are X and Y:

         1000     0.003719
         1100     0.004066
         1200     0.004507
         1300     0.005044
         1400     0.005683
         1500     0.006432
         1600       0.0073
         1700       0.0083
         1800     0.009449
         1900      0.01077
         2000      0.01227
         2100        0.014
         2200      0.01597
         2300      0.01824
         2400      0.02087
         2500      0.02391
         2600      0.02749
         2700      0.03175
         2800      0.03693
         2900      0.04343
         3000      0.05208
         3100      0.06516
       3129.9      0.07103

and I got A from basic fitting tools as below:

Coefficients:
  p1 = 0.00097657
  p2 = 0.001743
  p3 = -0.00084778
  p4 = -0.00046857
  p5 = 0.0064581
  p6 = 0.01318
  p7 = 0.013886

Norm of residuals = 
     0.0011309

I must mention that when I was fitting I got this warning: "polynomial is badly conditioned" but the fit looks good.

what am I doing wrong. I could see this problem in Excel too.

• I need to have a function (Y=f(X)) for my data. If I shouldn't use polynomial fitting, do you have other suggestions?

[Image Analyst]

You probably didn't use S and mu. See my code below.

Answer 1

[Image Analyst]

I don't know what you did, but when I did it, it looks great:

xy = [1000     0.003719
  1100     0.004066
  1200     0.004507
  1300     0.005044
  1400     0.005683
  1500     0.006432
  1600       0.0073
  1700       0.0083
  1800     0.009449
  1900      0.01077
  2000      0.01227
  2100        0.014
  2200      0.01597
  2300      0.01824
  2400      0.02087
  2500      0.02391
  2600      0.02749
  2700      0.03175
  2800      0.03693
  2900      0.04343
  3000      0.05208
  3100      0.06516
  3129.9      0.07103]
% Extract x and y.
x = xy(:, 1);
y = xy(:, 2);
% Get the fit coefficients.
[coeffs, S, mu] = polyfit(x, y, 6);
% Get the fitted values.
fittedY = polyval(coeffs, x, S, mu);
% Plot everything.
plot(x, y, 'r*', 'LineWidth', 2);
hold on;
plot(x, fittedY, 'bd-', 'LineWidth', 2, 'MarkerSize', 15);
grid on;
% Enlarge figure to full screen.
set(gcf, 'units','normalized','outerposition',[0 0 1 1]);

[Habib]

Thanks. my purpose is to have a function describing y vs x, (y=f(x) , so that I can take derivative from that function. If I use S and mu as you said, I can fit the generated fittedY to the original plot. but when I take derivative of the produced polynomial, S and mu cant be taken into account!

[coeffs, S, mu] = polyfit(x, y, 6);
der=polyder(coeffs) % this gives wrong answers becoz S and mu are ignored.

and the polynomial derivative is wrong again. right?

[Image Analyst]

I haven't used polyder. You can do it analytically - you know the coefficients and S and mu, so just do it yourself using the equations.

[Habib]

For polynomials analytically d/dx= n*x^(n-1). but when some correcting factors (mu and S) are attached to the polynomial how can we take them into account for taking derivative? Is there a tutorial for that? I dont know. If there is a method please let me know.

[Habib]

I just taught of another solution for taking derivative. I can take numerical derivative. suppose that i want to take the deviative at x0:

[coeffs, S, mu] = polyfit(x, y, 6);
dx=1 % a small value compared to x values
A= polyval(coeffs, x0+dx, S, mu);
B= polyval(coeffs, x0-dx, S, mu);
der=(A-B)/(2*dx) % derivative with enough precision at x0

This way I can take the derivative of my data set at any x0, say 2231.5. do you agree?. If I had used the original data set, the derivative of all points between 2200 and 2300 would be the same and there would be a sudden break at provided x points in the original data set.

I will examine it tomorrow.

Answer 2

[John D'Errico]

Sigh. This happens over and over again. Fit a linear model, then a quadratic. It does better. So a cubic model must be better yet. Why not fit a 100th order model? It should do really well.

The problem is with high order polynomial models is they DON'T do well. They have miserable problems with conditioning, especially when your data is like this. Don't believe me? Don't understand why?

What is 3000^6?

3000^6
ans =
     7.29e+20

Yeah. Its huge. The problem is a double precision number in MATLAB (or almost ANY environment) only has about 16 digits of precision. So when you do arithmetic on numbers with such a wide dynamic range, you end up with ill-conditioned systems of equations. (Why do you think you got that warning message?)

You CAN improve things a bit by transforming the problem, centering and scaling as suggested. See the difference between these two models with your data:

polyfit(x,y,6)
Warning: Polynomial is badly conditioned. Add points with distinct X values, reduce the degree of the polynomial, or try centering and
scaling as described in HELP POLYFIT. 
> In polyfit at 76 
ans =
  1.0486e-20 -1.1933e-16 5.5538e-13 -1.3466e-09 1.7944e-06 -0.001241 0.35052

polyfit((x - 2000)/1000,y,6)
ans =
  0.010486 0.0065015 -0.0087511 0.00106 0.01436 0.016792 0.012122

The latter problem is far better conditioned. If you want evidence...

cond(bsxfun(@power,x,0:6))
ans =
   8.3329e+23

cond(bsxfun(@power,(x-2000)/1000,0:6))
ans =
       91.758

Even with that, the model is simply not that great. It is not even monotone o=ver the data.

P = polyfit((x - 2000)/1000,y,6);
roots(polyder(P))*1000 + 2000
ans =
       2681.7 +     533.47i
       2681.7 -     533.47i
       1047.5 +          0i
       1536.3 +     398.64i
       1536.3 -     398.64i

So the polynomial has a minimum at 1047.5, inside the data range.

Far better is to use a tool designed to deal with curve fitting of data like this, and to do so intelligently. I'd suggest my SLM tools.

http://www.mathworks.com/matlabcentral/fileexchange/24443-slm-shape-language-modeling

[Habib]

Thanks John. your guide was very helpful. I have to go deep through concerns of curve fitting. Otherwise I will some day end up with big problem in computing. I downloaded your tool.