Kolmogorov–Arnold Network

A Kolmogorov–Arnold Network (or KAN) is a neural network that is trained by learning the activation function at each node, rather than the weight on each edge as in a traditional multilayer perceptron. This is a practical application of the Kolmogorov–Arnold representation theorem.

Description

By the Kolmogorov–Arnold representation theorem, any multivariate continuous function $f$ can be written as

$f(\mathbf {x} )=f(x_{1},\ldots ,x_{n})=\sum _{q=0}^{2n}\Phi _{q}\!\left(\sum _{p=1}^{n}\phi _{q,p}(x_{p})\right)$.

with suitable values of the functions $\phi _{q,p}\colon [0,1]\to \mathbb {R}$ and $\Phi _{q}\colon \mathbb {R} \to \mathbb {R}$. To create a deep network, the KAN extends this to a graph of such representations:

$f(\mathbf {x} )=\sum _{i_{L-1}=1}^{n_{L-1}}\phi _{L-1,i_{L},i_{L-1}}\left(\sum _{i_{L-2}=1}^{n_{L-2}}\cdots \left(\sum _{i_{0}=1}^{n_{0}}\phi _{0,i_{1},i_{0}}(x_{i_{0}})\right)\right)$

where each $\phi _{l,i,j}$ is a learnable B-spline. Since this representation is differentiable, the values can be learned by any standard backpropagation technique.

For regularization, the L1 norm of the activation function is not sufficient by itself. To keep the network sparse and prevent overfitting, an additional entropy term $S$ is introduced for each layer $\Phi$:

$S(\Phi )=-\sum _{i=1}^{n_{in}}\sum _{j=1}^{n_{out}}{\frac {|\phi _{i,j}|_{1}}{|\Phi |_{1}}}\log \left({\frac {|\phi _{i,j}|_{1}}{|\Phi |_{1}}}\right)$

A linear combination of these terms and the L1 norm over all layers produces an effective regularization penalty. This sparse representation helps the deep network to overcome the curse of dimensionality.

Properties

The number of parameters in such a representation is $O(N^{2}LG)$, where $N$ is the output dimension, $L$ is the depth of the network, and $G$ is the number of intervals over which each spline is defined. This might appear to be greater than the $O(N^{2}L)$ parameters needed to train a multilayer perceptron of depth $L$ and output dimension $N$; however, Liu et al. argue that in scientific domains the KAN can achieve equivalent performance with fewer parameters since many natural functions can be decomposed efficiently into splines.

KANs have been shown to perform well on problems from knot theory and physics (such as Anderson localization), although they have not yet been scaled to language models.

History

Computing the optimal Kolmogorov–Arnold representation for a given function has been a research challenge since at least 1993. Early work on applying this representation to a network used a fixed depth of two and did not appear to be a promising alternative to perceptrons for image processing.

More recently, a deep learning algorithm was proposed for constructing these representations using Urysohn operators. In 2021, the representation was successfully applied to a deeper network. In 2022, ExSpliNet brought together the Kolmogorov–Arnold representation with B-splines and probabilistic trees to show good performance on the Iris flower, MNIST, and FMNIST datasets, and argued for the better interpretability of this representation compared to perceptrons.

The term "Kolmogorov–Arnold Network" was introduced by Liu et al. in 2024, which generalizes the networks to arbitrary width and depth, and demonstrated strong performance on a realistic class of multivariate functions although training is inefficient.