5.5 Fitting subject to constraints   5. Least-Squares Methods ... 5.3 Fitting functions linear ...

5.4 Fitting an arbitrary function, not linear in the parameters

Many cases of practical interest do not lead to analytical least-squares solutions. Examples are functions involving exponential or trigonometric functions of the independent variable, where a parameter to be determined enters in the argument to the exponential or trigonometric function. As an example, consider the analysis of the preceding section applied to a function g(x)=a e^bx, where the best-fit value of b is to be determined. The $\chi^2$ function then becomes

$$\chi^2 = {{1}\over{\sigma^2}} \sum_i (y_i - a e^{bx_i})^2$$ (5.35)

 

and the minimum value of the chisquare occurs when

$${{\partial\chi^2}\over{\partial b}} = {{2}\over{\sigma^2}} \sum_i(y_i - a e^{bx_i})(-ax_ie^{bx_i}) = 0 .$$ (5.36)

 

The minimum value thus occurs for

$$\sum_i y_ix_i e^{bx_i} = a\sum_ie^{2bx_i} .$$ (5.37)

 

This is a transcendental equation having no general analytical solution for b. Other examples of transcendental equations are x=e^x and $x=\cos(x)$; equations like these often result from fits to functions where the parameters enter through exponential or trigonometric functions.

Encounters with equations like these should not be daunting, because with modern computing tools their solutions are readily calculated. Some of the methods used to solve such equations are reviewed briefly in Chapter 9. When sets of these equations represent the solution, there are a variety of techniques that can be used to search for that solution. Some options are:

1.

If the equation to be solved is transcendental but easily expressed in analytical form, numerical methods like the Newton-Raphson technique (discussed later, in Chapter 9) can be used to search for solutions.

2.

It is often possible to cover the region of interest with a grid of possible values for the parameters and then simply evaluate the $\chi^2$ function for those grid points and search for the minimum. Once a minimum is found, another grid spanning a subset of the original can be selected and searched again for a minimum. This approach often requires a large number of values in the parameters and is very inefficient in comparison to other methods, but can provide a useful starting value for other methods of minimization. It also helps in cases where there may be multiple extreme values in the solution, because contours of the $\chi^2$ function over grids can indicate how likely it is that a search procedure will find an extreme value of the $\chi^2$ that is not a global minimum.

3.

A "reducing step" technique can be used, as follows. Start with a guess at the best values for the set of parameters. Take steps in one parameter of some size, continuing in a given direction as long as the $\chi^2$ continues to decrease. When it starts to increase, reduce the step size and reverse direction. Continue until the minimum value is found to some desired accuracy. Then repeat the search in the other parameters. After this has been done for all parameters, the procedure must be repeated because the minimum value in the first parameter will have changed when other parameters change. Repeat the loop over all parameters until changes resulting from this iteration are smaller than some tolerance, which must be selected to be small compared to the desired accuracy of the solution. While this approach can cover a larger range of values and requires less knowledge of the likely range of values than does the grid-evaluation approach, it is still usually very inefficient and there is a danger than the procedure will satisfy the selected criteria before a true minimum is found. It is also hard to estimate the accuracy of the solution obtained with confidence.

4.

A "gradient step" technique is usually much more efficient than the preceding approaches. Instead of taking arbitrary steps in each parameter, steps are selected in a "direction" (i.e., with proportional changes in all parameters) that reduces the $\chi^2$function the fastest by selecting the step to be in the direction of maximum decrease in the $\chi^2$ function. That is, steps are taken in a set of parameters $\{a_j\}$ so that the change in each parameter is proportional to the derivative of the $\chi^2$function with respect to that parameter:

$$\Delta a_j = - \lambda {{\partial\chi^2}\over{\partial a_j}} ......ambda {{\chi^2(a_j+\delta a_j)-\chi^2(a_j)}\over{\delta a_j}}$$ (5.38)

 

where the second formula is useful in cases where the derivatives are not expressed in convenient analytical forms. The advantage of this approach is that it rapidly produces a value close to the minimum, but the disadvantage is that once the parameters are near the minimum the differences involved in numerical evaluation of the derivatives are small and the method becomes slow and inaccurate.

5.

Analytical approximations to the $\chi^2$ function can sometimes be used. If a polynomial expansion that approximates the $\chi^2$function can be found, for example, the minimum in the polynomial expansion is usually easy to determine by the analytical methods of the preceding section. This approach is usually used iteratively: Once the minimum in the polynomial expansion is found, a new polynomial expansion is determined that fits the function best at that minimum point, and then the minimum in the new polynomial expansion is found, etc.

6.

Expansion of the approximating functions g_k(x) in Taylor or other series sometimes also proves useful. In this approach, instead of approximating the $\chi^2$ function by simpler forms as in the above approach, we approximate the fitting functions themselves by simpler forms and then determine appropriate coefficients for these simpler forms.

In all these cases, a problem having no general solution is that minimization searches may find a local minimum in the solution that is not the global minimum. There is no general solution to this problem except to consider the expected nature of the solution and be aware that extreme values obtained by search procedures can be true zero-derivative solutions and yet not be the global minimum.

One of the most powerful approaches is to use a gradient-step technique to get a value near the minimum, then switch to a Newton-Raphson approach to refine the estimate of the minimum. This takes advantage of the initial speed of the gradient-step method while using the good convergence properties and generality of the Newton-Raphson technique. Most commercial analysis packages for computers include techniques for the solution of nonlinear equations, often based on this approach.^5.2

   
 5.5 Fitting subject to constraints  5. Least-Squares Methods ...  5.3 Fitting functions linear ...