Feature learning · Research note
Learning changes
what is learnable
Following a weak target through a changing network.
A network learning one pattern is also changing the machinery it will use to learn the next. Some changes may help. Others may make a useful direction harder to reach. Even in a small model, knowing which happens requires more than measuring how much the features move.
Consider two orthogonal patterns, A and B. We give A almost all of the training signal and add a very small amount of B. At initialization the network has equal capacity along both directions. As it learns, that symmetry disappears.
We follow this process in a quadratic population network in five dimensions. The model lets us distinguish three questions. Which changes in prediction become easier to make? Does the interaction between the tasks help B? And can B’s error reach zero without crossing it?
Read the paper
The same network, at five points during training on A. Each contour has radii proportional to the square roots of the two diagonal capacities, on a fixed scale. The cross term is zero along this unperturbed flow. This is a geometric construction from ordinary numerical data, not a picture of the coupling.
Room to move
Here, capacity measures how strongly a small parameter update can change the prediction along a particular target. The starting contour is a circle because the two capacities are equal. Training on A stretches it mainly along A. The smaller vertical expansion shows that the changing features also affect B, despite B being absent from this unperturbed training run.
The analytical result proves growth of A’s capacity throughout the common existence interval. For B, it proves positive curvature at initialization. The later transverse expansion in the figure is a numerical observation. That distinction matters when we ask what the mechanism actually guarantees.
Capacity alone cannot tell us whether an update helps. Training chooses an update to reduce the current loss, and its effect on B depends on the error along A as well as the error along B. In the two-mode model, the relation is explicit.
The diagonal entries are capacities. The cross term, \(\chi\), describes how an update on one task moves the other. When both residuals are negative, a positive cross term pushes B toward zero. The geometry and the current errors have to be read together.
An influence carried forward
To isolate that interaction, we first take the limit of an infinitesimal B. Imagine introducing a small influence along B at one instant, then following it through the subsequent motion of the network. The response depends on the path the features take after that instant.
The paper derives a kernel for this transported response directly from the particle dynamics. Its time derivative, \(M(s,t)\), determines how past weak-task forcing contributes to the coupling at a later time. An individual particle can contribute with the wrong sign. The useful sign appears after pairing the positive and negative initial output weights and averaging over the initialization.
For our specified compact initialization, that paired kernel is nonnegative over the full interval needed by the proof. Along the initialized linear-response trajectory, it generates a strictly positive coupling after time zero. Rotational symmetry extends the statement to equal-norm traceless quadratic probes within the transverse four-dimensional subspace.
The setting, without the shorthand
The input is uniform on the sphere in dimension five. Both layers of the centered quadratic network train in the population limit. The dominant teacher is \(A=uu^\top-I/5\). The transverse probes act on \(u^\perp\), have zero trace, and have the same Frobenius norm as A. The initialization has a particular compact radial law and symmetric output weights. It matches specified Gaussian moments but is not Gaussian.
Physical time is rescaled by a fixed positive factor to give \(\sigma\). The exposure coordinate satisfies \(d\lambda/d\sigma=-\Delta\), where \(\Delta\) is the dominant residual along the unperturbed flow. All finite-perturbation comparisons use this same base clock.
Improving, and still falling behind
A helpful interaction does not mean the weak task catches up. In the normalized linear-response description, both errors start at −1. Both approach zero, but the weak residual remains more negative. Its absolute error decreases while its error relative to A’s increases.
The curves show ordinary numerical integration before the limiting exposure. B is the normalized infinitesimal response, not the raw error of a finite second task. The theorem establishes the ordering and convergence through a separate analytical argument with interval bounds.
Proving this ordering needs more than a positive response kernel. We construct a scalar barrier that keeps the ratio of residual magnitudes increasing. A first-hit argument shows that the ratio cannot turn back while the barrier stays positive.
The barrier is the computer-assisted part of the proof. A separate arbitrary-precision interval checker verifies complete trajectory tubes across the angular and time domains, then checks the derivative and positivity bounds used in the argument. It covers the spaces between sample points. The plotted curves play no role in that certification.
Training also has two different notions of time. Exposure approaches a finite limit as the dominant error vanishes, while physical time continues indefinitely. A comparison argument places that limit inside the interval where the barrier has been checked. The linearized weak residual then converges to zero from below as physical time tends to infinity.
Giving B a finite share
A finite second target changes the trajectory it is probing. We can control that change for sufficiently small mixing on any fixed compact interval before the limiting exposure. On such an interval, the coupling remains helpful and B’s error decreases without changing sign.
The permitted amount of mixing may shrink as the interval grows. We do not prove that one finite choice works all the way to infinite physical time. This is the main boundary of the result.
The two probes respond differently to finite mixing. For the GS probe, the capacity change is first order in ε. For the block probe, symmetry makes it quadratic. An exact moment calculation identifies the leading GS coefficient; the ε-sweep below tests how it appears at finite times.
Dividing the GS capacity change by ε reveals a nonzero leading term. The additional time normalization exposes its early-time coefficient. The reference line is exact algebra; the colored and gray curves are ordinary numerical experiments at finite times. The other probe has an even capacity response and behaves differently.
The finite-mixing theorem uses continuity of capacity and a separate, time-weighted estimate for the coupling. The weighting matters near initialization, where the helpful interaction itself starts at zero. The paper derives the estimates needed to preserve its sign on each compact interval.
The accompanying code includes the full interval evidence, source and artifact hashes, and commands for reproducing the checks. “Independently checked” here means a separate interval implementation of the specified computational obligations. The analytical reductions and the underlying arithmetic software remain part of the trusted argument.
What this example gives us
Feature learning already has a substantial theory. Work on extensive-width quadratic networks, stagewise recovery of quadratic teachers, and dynamical mean-field theory studies how representations and training dynamics evolve under other assumptions. The broad fact that training changes useful features is not our novelty claim.
The contribution here is a sign-controlled transported response, together with a certified barrier for the weak residual, in a particular solvable population model. It makes one mechanism precise enough to follow from the particle equations to convergence. It does not establish the same behavior for arbitrary initialization, finite networks, or general neural architectures.
The lesson we take from the model is a practical question to ask of changing features. Which task directions become more accessible, and does the training trajectory actually use that access to reduce their errors? Answering one does not settle the other.