Norms

At several points, it is necessary to compute the norm of some arbitrary PyTree. For example, to check that the difference between two adjacent iterates is small: ||y_{n+1} - y_n|| < ε, and thus that an optimisation algorithm has converged.

Optimistix includes the following norms. (But any function PyTree -> non-negative real scalar will suffice.)

Info

For the curious: Optimistix typically uses optimistix.max_norm as its default norm throughout. This is because it is invariant to the size of the problem, i.e. adding extra zero padding will never affect its output. This helps to ensure a consistent experience as the problem size changes. (This property is "sort of" true of optimistix.rms_norm, and not at all true of optimistix.two_norm.)

optimistix.max_norm(x: PyTree[ArrayLike]) -> Array

Compute the L-infinity norm of a PyTree of arrays.

This is the largest absolute elementwise value. Considering the input x as a flat vector (x_1, ..., x_n), then this computes max_i |x_i|.

optimistix.rms_norm(x: PyTree[ArrayLike]) -> Array

Compute the RMS (root-mean-squared) norm of a PyTree of arrays.

This is the same as the L2 norm, averaged by the size of the input x. Considering the input x as a flat vector (x_1, ..., x_n), then this computes sqrt((Σ_i x_i^2)/n)

optimistix.two_norm(x: PyTree[ArrayLike]) -> Array

Computes the L2 norm of a PyTree of arrays.

Considering the input x as a flat vector (x_1, ..., x_n), then this computes sqrt(Σ_i x_i^2)