Originally published Nov. 2025; technical review Jul. 30, 2026.

Neural networks sometimes memorize data for an extended number of training steps without generalizing. Then they suddenly start to generalize. This is called grokking. The foundational study reviewed here develops a complexity-based account of this behavior in its experiments: the network's measured complexity rises, then falls. The authors describe this as a complexity phase transition.

▶ What is "complexity phase transition"?
A sharp change in a system's state, like water turning to ice. In this paper, it's when the network's complexity suddenly drops from high (memorization) to low (generalization). The network doesn't gradually get better. It stays stuck memorizing, then suddenly flips to understanding the pattern. This sharp transition is what makes grokking special.

The key finding: In grokking with proper regularization, networks get simpler as they learn. Complexity rises during memorization, peaks, then falls as the network finds simpler patterns that work. Without regularization, complexity stays high and generalization does not emerge.

Foundational study reviewed here: submitted Dec. 13, 2024; last revised Aug. 21, 2025. This article explains that study rather than presenting it as current consensus.

Paper by: Branton DeMoss, Silvia Sapora, Jakob Foerster, Nick Hawes, Ingmar Posner
Department of Engineering Science, University of Oxford

Read the full paper: The Complexity Dynamics of Grokking (arXiv:2412.09810)

Current context, reviewed Jul. 30, 2026

Recent work continues to treat grokking as an active research question with more than one proposed mechanism. Ersoy and Wiesner, submitted Jun. 15, 2026, explain delayed convergence in linear deep networks through noise-driven escape from metastable states and present evidence that a related mechanism may operate in nonlinear networks. Pranjić, Roth, and Tutschku, submitted Jul. 9, 2026, observe grokking, double descent, and late generalization decay in a two-qubit quantum neural network, with weight-norm regularization stabilizing the learned state.

These studies broaden the picture, but neither directly replicates the original paper's rate-distortion compression metric. They should be read as complementary evidence, not confirmation that one explanation now accounts for grokking generally.

How to Measure What Networks Learn

Normal methods count parameters, but parameter count is not the same as complexity. This paper uses a different approach: "How many bits does it take to describe what the network learned?" Their metric aims at intrinsic complexity via compressibility.

Here's the idea: measure how small you can compress the network's weights. Random data doesn't compress much. Data with patterns compresses a lot.

The researchers use Kolmogorov complexity and rate-distortion theory to measure network complexity. Their method gets 30-40 times better compression than simple methods. This gives them a tight upper bound on complexity by using lossy compression after coarse-graining, letting them track complexity dynamics during training.

▶ What is "Kolmogorov complexity"?
Kolmogorov complexity measures how many bits it takes to describe something. For example, the string "1111111111" can be described as "print 1 ten times" (short). But random text like "k3j9xz2m" needs the whole text to describe it (long). It's a universal way to measure how complex any data is.
▶ What is "rate-distortion theory"?
Rate-distortion theory is about trading off between compression and accuracy. It asks: how much information do I lose if I compress data this much? In this paper, it helps measure network complexity by finding the smallest description that still works well.

Three Stages of Learning

Stage 1: Memorization
Complexity rises as the network memorizes examples. It stores each one individually. High complexity.

Stage 2: Peak
Complexity reaches its highest point. The network has memorized everything but hasn't found the underlying rule.

Stage 3: Generalization
Complexity falls. The network finds a simpler pattern that works for all data. It stops storing individual examples and learns the rule.

In these experiments, networks without regularization remain in a high-complexity memorization phase and don't generalize. They never make the transition to simplicity.

The Link Between Compression and Generalization

The paper connects complexity to the Minimum Description Length Principle. This principle says simpler models are better.

▶ What is "Minimum Description Length Principle"?
This principle says the best model is the one that compresses your data the most. It combines two costs: how well the model fits the data, plus how complex the model itself is. Pick the model that minimizes the total message length needed to describe both.

A lookup table that memorizes all data drives data entropy to zero, but the model complexity equals the original data entropy. So the total description length doesn't decrease. No compression is achieved. A model that finds patterns can compress the data much more. Under MDL, the model that best compresses the data is expected to generalize best.

Better compression predicts better generalization under MDL. This is a principled link, not an identity. Both involve finding simple patterns in data.

A New Training Method

The researchers created a new training technique called spectral entropy regularization. It penalizes network complexity during training. This pushes networks toward simpler representations and helps them generalize instead of memorizing.

▶ What is "spectral entropy regularization"?
Spectral entropy regularization penalizes the entropy of the singular values of weight matrices, nudging the network toward low-rank, more compressible representations.

Why This Matters

This work investigates a key question: How might lower-complexity solutions relate to generalization in grokking?

The paper argues that simpler, more compressible solutions can generalize better and provides a way to measure and encourage that behavior in its experimental setting. The evidence comes from modular arithmetic grokking tasks, where compressibility tracks with generalization. It does not establish that model size or data scale are generally unimportant.

Within these experiments, the proposed metric makes it possible to track a path from higher measured complexity toward a simpler solution during training. Whether the metric transfers broadly remains an open empirical question.

Note: These results are shown on modular arithmetic grokking tasks using small transformers. Regularization is key to seeing the clean complexity phase transition. The findings may apply more broadly, but the experiments focus on this specific setting.