Optimization of many simultaneous dimensions

I am attempting to optimize an objective function across ten dimensions. The objective function calls several systems of ordinary differential equations using some combination of parameters (the parameters used in the ODEs are what the optimization routine is varying), then compares the ODE output to desired data and penalizes differences, so the objective function is a sum of squared errors between ODE output and target data. The results have not been very close to the target data.

I am currently using fmincon as called by multistart or globalsearch (I have tried both). I know ten is a lot of dimensions to handle at once, but I'm not certain if there is a better method available in Matlab. Is this simply too many dimensions or is fmincon still the best method for a large optimization problem like this?

I do consistently get the follow orange text when running the routine:

Warning: Matrix is close to singular or badly scaled. Results may be inaccurate. RCOND = 4.231283e-17.

> In backsolveSys

In solveKKTsystem

In computeTrialStep

In barrier

In fmincon (line 862)

In globaloptim.multistart.fmultistart

In globaloptim.multistart.fmultistart

In MultiStart/run (line 257)

In globalsearchMiddle (line 11)

[Walter Roberson]

A problem with a pure sum of squared errrors, is that it does not take scale into account. For example, a change of 0.001 in a value that is near 0.02 might be an important difference for fitting that component ODE, but a change of 100 in a value that is near 10^18 might be just a change of one bit, just round-off error.

[Matt J]

Ten dimensions is very small. There is nothing that can be concluded from the dimension alone about the difficulties you are having.

Answer 1

[Shubham]

Hi Eric,

The warning you're receiving suggests that during the optimization process, a matrix (likely the Hessian or a matrix related to constraints) is nearly singular, which can lead to numerical instability and inaccurate results. This issue might be due to poorly scaled problem parameters, nearly linearly dependent constraints, or other numerical difficulties.

Here are some strategies and considerations that might help you improve your optimization routine:

1. Ensure all parameters are on a similar scale.
2. Try different algorithms in fmincon ('interior-point', 'sqp', etc.).
3. Use multiple starting points with MultiStart or GlobalSearch.
4. Provide gradients and Hessians analytically if possible.
5. Add a regularization term to the objective function if the problem is ill-conditioned.
6. Consider other optimization methods like ga, particleswarm, or surrogateopt.
7. Adjust fmincon options like tolerances for better stability.

Lastly, ten dimensions is certainly higher-dimensional for optimization problems, but it is not necessarily too many for modern optimization algorithms to handle. The key is to ensure that the problem is formulated and approached in a numerically stable and efficient manner.