Skip to content

Line searches, trust regions, learning rates¤

optimistix.AbstractSearch

optimistix.AbstractSearch ¤

The abstract base class for all searches. (Which are our generalisation of line searches, trust regions, and learning rates.)

See this documentation for more information.

¤

optimistix.LearningRate (AbstractSearch) ¤

Move downhill by taking a step of the fixed size learning_rate.

__init__(self, learning_rate: Any) ¤

Arguments:

  • learning_rate: The fixed step-size used at each step.

optimistix.BacktrackingArmijo (AbstractSearch) ¤

Perform a backtracking Armijo line search.

__init__(self, decrease_factor: Union[Array, ndarray, numpy.bool_, numpy.number, bool, int, float, complex] = 0.5, slope: Union[Array, ndarray, numpy.bool_, numpy.number, bool, int, float, complex] = 0.1, step_init: Union[Array, ndarray, numpy.bool_, numpy.number, bool, int, float, complex] = 1.0) ¤

Arguments:

  • decrease_factor: The rate at which to backtrack, i.e. next_stepsize = decrease_factor * current_stepsize. Must be between 0 and 1.
  • slope: The slope of of the linear approximation to f that the backtracking algorithm must exceed to terminate. Larger means stricter termination criteria. Must be between 0 and 1.
  • step_init: The first step_size the backtracking algorithm will try. Must be greater than 0.

optimistix.ClassicalTrustRegion (AbstractSearch) ¤

The classic trust-region update algorithm which uses a quadratic approximation of the objective function to predict reduction.

Building a quadratic approximation requires an approximation to the Hessian of the overall minimisation function. This means that trust region is suitable for use with least-squares algorithms (which make the Gauss--Newton approximation Hessian~Jac^T J) and for quasi-Newton minimisation algorithms like optimistix.BFGS. (An error will be raised if you use this with an incompatible solver.)

__init__(self, high_cutoff: Union[Array, ndarray, numpy.bool_, numpy.number, bool, int, float, complex] = 0.99, low_cutoff: Union[Array, ndarray, numpy.bool_, numpy.number, bool, int, float, complex] = 0.01, high_constant: Union[Array, ndarray, numpy.bool_, numpy.number, bool, int, float, complex] = 3.5, low_constant: Union[Array, ndarray, numpy.bool_, numpy.number, bool, int, float, complex] = 0.25) ¤

In the following, ratio refers to the ratio true_reduction/predicted_reduction.

Arguments:

  • high_cutoff: the cutoff such that ratio > high_cutoff will accept the step and increase the step-size on the next iteration.
  • low_cutoff: the cutoff such that ratio < low_cutoff will reject the step and decrease the step-size on the next iteration.
  • high_constant: when ratio > high_cutoff, multiply the previous step-size by high_constant`.
  • low_constant: when ratio < low_cutoff, multiply the previous step-size by low_constant`.

optimistix.LinearTrustRegion (AbstractSearch) ¤

The trust-region update algorithm which uses a linear approximation of the objective function to predict reduction.

Generally speaking you should prefer optimistix.ClassicalTrustRegion, unless you happen to be using a solver (e.g. a non-quasi-Newton minimiser) with which that is incompatible.

__init__(self, high_cutoff: Union[Array, ndarray, numpy.bool_, numpy.number, bool, int, float, complex] = 0.99, low_cutoff: Union[Array, ndarray, numpy.bool_, numpy.number, bool, int, float, complex] = 0.01, high_constant: Union[Array, ndarray, numpy.bool_, numpy.number, bool, int, float, complex] = 3.5, low_constant: Union[Array, ndarray, numpy.bool_, numpy.number, bool, int, float, complex] = 0.25) ¤

In the following, ratio refers to the ratio true_reduction/predicted_reduction.

Arguments:

  • high_cutoff: the cutoff such that ratio > high_cutoff will accept the step and increase the step-size on the next iteration.
  • low_cutoff: the cutoff such that ratio < low_cutoff will reject the step and decrease the step-size on the next iteration.
  • high_constant: when ratio > high_cutoff, multiply the previous step-size by high_constant`.
  • low_constant: when ratio < low_cutoff, multiply the previous step-size by low_constant`.