Curvature Along Gradient Directions

The question

Optimizers interact with the loss landscape through gradients. But the landscape is not flat—some directions are steep, others are nearly level. The relevant quantity is not the gradient alone but the curvature along the directions we actually step in.

Most optimizers treat all layers the same. Adam uses a single recipe. Even second-order methods like K-FAC assume the curvature structure is architecturally uniform. We asked: is it?

It is not. We find that curvature along gradient singular directions follows a simple power law that varies dramatically by layer type. Convolutions, attention layers, MLPs, and output heads each see fundamentally different curvature scaling. A single optimizer recipe is a poor match for all of them.

The empirical law

For each layer, we compute the gradient $ G = U\Sigma V^\top $, take the top-$k$ singular directions $ u_k v_k^\top $, and measure the exact directional curvature $ h_k $ via Hessian-vector products. Plotting $ h_k $ against the gradient singular value $ \sigma_k $ on log–log axes, we find a clean power law:

$$ h_k \;=\; c \cdot \sigma_k^\alpha $$

The exponent $ \alpha $ is not universal. It depends on layer type, architecture, and task:

Architecture	Layer type	Mean α
ResNet-18 / VGG-11	Conv	1.97–2.17
ResNet-50 (Tiny-ImageNet)	Conv (n=49)	1.94
ResNet-50 (ImageNet-1K)	Conv (n=14)	2.10
Pure MLP	Dense	2.18
GPT-2 (6-layer)	Attention	1.30
GPT-2	MLP	1.00
GPT-2	Output head	4.53

Convolutional layers cluster near $ \alpha \approx 2 $. Transformer attention layers sit around $ \alpha \approx 1 $. MLP up-projections dip below 1. Output heads can exceed 4. This is not noise—it is stable across seeds, datasets, and scales. On ResNet-18, the median $ R^2 $ across 21 layers is 0.998.

The law holds at scale: ResNet-50 on ImageNet-1K (pretrained, 14 conv layers) gives median $ \alpha = 2.15 $ with $ R^2 \geq 0.97 $. It is not a small-model artifact.

Why α varies

Under the Gauss–Newton Kronecker curvature model, directional curvature decomposes as:

$$ h_k \;=\; \rho_k^{(\delta)} \cdot \rho_k^{(A)} \cdot [\Lambda_\delta]_{kk} \cdot [\Lambda_A]_{kk} $$

where $ [\Lambda_\delta]_{kk} $ and $ [\Lambda_A]_{kk} $ are eigenvalues of the error and activation covariances, and $ \rho_k^{(\delta)}, \rho_k^{(A)} $ are alignment ratios measuring how gradient singular directions line up with covariance eigenvectors.

This is an identity—it is true by construction given the Kronecker model. Its value comes from what it predicts. Taking log-slopes gives the Spectral Alignment Decomposition:

$$ \alpha \;=\; \underbrace{\frac{d\log \Lambda_k}{d\log \sigma_k}}_{\beta\;\text{(covariance-spectral)}} \;+\; \underbrace{\frac{d\log \Phi_k}{d\log \sigma_k}}_{\phi\;\text{(alignment-slope)}} $$

When Kronecker factor eigenbases align with gradient singular directions, $ \beta = 2 $. Deviations from $ \alpha = 2 $ come entirely from the alignment slope $ \phi $. Each architectural mechanism produces a characteristic signature:

LayerNorm pulls α toward 1. It flattens activation covariance, making $ \rho_k^{(A)} $ independent of $ \sigma_k $. The activation-side contribution to $ \alpha $ vanishes.

Residual connections decouple error alignment. The skip path makes the error covariance independent of the current layer’s gradient structure, so $ \rho_k^{(\delta)} $ contributes no slope.

Softmax concentration pushes α above 2. The error covariance has rank at most $ c $ (the number of classes). Beyond rank $ c $, alignment drops sharply, creating a steep positive slope.

These are not vague intuitions. Each prediction is validated by controlled ablations: switching ViT-Tiny from CIFAR-10 (10 classes) to CIFAR-100 (100 classes) drops $ \alpha $ by 0.5 in attention layers; switching GPT-2 from language modeling to binary classification raises $ \alpha $ by 0.7–0.9 in MLP layers.

The spectral transfer relation

Three exponents characterize a layer’s spectral structure: the curvature exponent $ \alpha $, an effective gradient rank-decay exponent $ \gamma $, and the directional curvature decay $ s $ (the exponent Tang et al. measure as Hessian eigenvalue decay). They are linked by a finite-window relation:

$$ s \;\approx\; \alpha \gamma $$

The proof is one line: if $ \sigma_k \sim k^{-\gamma} $ and $ h_k = c\sigma_k^\alpha $, then $ h_k \sim k^{-\alpha\gamma} $. The algebra is trivial on purpose. The empirical content is that $ \alpha $ and $ \gamma $ are fit on independent data—HVPs vs. SVD—yet their product recovers $ s $ with remarkable precision:

Dataset	Layers	Median error
CIFAR-10 (ResNet-18, VGG-11, GPT-2)	21	1.9%
Tiny-ImageNet (ResNet-50)	54	1.0%
ImageNet-1K (ResNet-50, pretrained)	14	1.6%
All	93	≈1.5%

No free parameters. Five architectures. Three datasets. 93 layers. The relation holds to roughly 1.5% across all of them.

A note on power laws. We do not claim gradient singular values follow a global power law. BIC model selection favors log-normal-in-rank over strict power-law on most layers. The exponent $ \gamma $ is an effective slope on the top-$ k $ measurement window. The spectral transfer relation works because $ \alpha $ absorbs the local curvature scaling regardless of the global rank-profile shape.

Curvature concentration

Under the power-law ansatz, the participation ratio of curvature is bounded by a ratio of Riemann zeta functions: $ \mathrm{PR}_h \leq \zeta(\alpha\gamma)^2 / \zeta(2\alpha\gamma) $.

Empirically, on the measured top-$ k $ window: $ \mathrm{PR}_h \in [1.0, 1.8] $. Despite hundreds of singular directions per layer, curvature concentrates onto one or two. Gradients, by contrast, span moderate-dimensional subspaces ($ \mathrm{PR}_{\mathrm{grad}} \in [1.8, 58] $).

In practical terms: if you are building a second-order optimizer, you can ignore the vast majority of the gradient spectrum. Almost all the curvature information—the signal that tells you how to rescale each direction—lives in the top one or two singular components. The rest is nearly flat. This is not a finite-width artifact: scaling width from $ n = 64 $ to $ n = 512 $ changes $ \mathrm{PR} $ by only 0.7.

What this means for optimizers

If curvature scales as $ h_k \propto \sigma_k^\alpha $, then the optimal step along singular direction $ k $ should scale as:

$$ T(\sigma_k;\, \alpha) \;=\; \frac{\sigma_k}{\sigma_k^\alpha + d} $$

where $ d $ is a damping constant. This is not a uniform spectral flattening (like Muon), and it is not a single scalar per layer (like learning-rate scaling). It is a per-direction, architecture-adaptive correction.

For convolutional layers with $ \alpha \approx 2 $, this reduces to $ T(\sigma) = \sigma / (\sigma^2 + d) $—a damped Newton step in the gradient singular basis. We call this Spectral Newton.

Model	Dataset	AdamW	Spectral Newton
ResNet-18	CIFAR-10	92.7%	93.8%
ResNet-18	CIFAR-100	71.5%	75.0%
ResNet-50	CIFAR-100	72.6%	75.0%

These are single-seed, matched-epoch results—directional evidence, not a comprehensive benchmark. But the signal is consistent: +1.1% on CIFAR-10, +2.4–3.5% on CIFAR-100.

The scalar negative control. We tested a cheaper alternative: Adam with per-layer learning rate $ \eta_\ell \propto (s_\ell / \bar{s})^\beta $ using measured $ s = \alpha\gamma $. Across three seeds and several values of $ \beta $, the improvement over Adam is at most 0.2%—within seed variance. Knowing the eigenvalue profile is not enough. You have to apply the correction directionally, along each gradient singular direction. Curvature is not a single number per layer.

Limitations

Directional curvature, not eigenvalues. Throughout this work, $ h_k $ is a Rayleigh quotient along gradient singular directions, not a true Hessian eigenvalue. Connecting our spectral transfer relation to the true Hessian eigenvalue spectrum requires showing that gradient singular directions approximate Hessian eigenvectors. We verify this empirically ($ R^2 > 0.99 $ for most conv layers) but do not prove it in general.

Scale. Our largest models are GPT-2 (6-layer, 85M parameters) and ResNet-50. The decomposition makes concrete, falsifiable predictions at larger scales—ViT-Large attention layers should exhibit $ \alpha \approx 1 $, LLaMA-7B MLP layers should follow $ \alpha < 1.2 $—but we have not verified these yet.

Spectral Newton. The optimizer results are single-seed on vision benchmarks only. Multi-seed evaluation with wall-clock and FLOP accounting, and comparison against K-FAC, Shampoo, and Muon, is needed before drawing conclusions about practical utility.

What’s next

This note describes the first part of a longer research arc. The decomposition tells us what the curvature structure looks like and why it varies across layer types. But it leaves a deeper question unanswered.

The spectral transfer relation says $ s \approx \alpha\gamma $. We now know what determines $ \alpha $—alignment geometry. But what determines the gradient rank profile itself? Why is $ \sigma_k $ log-normal in most layers but exponential at the boundaries? Why does the shape change during training? The exponent $ \gamma $ is an effective slope on a finite window, not a fundamental quantity. Something deeper must set it.

It turns out the same two Kronecker factor matrices $ C_\delta $ and $ C_A $ that determine $ \alpha $ also determine the full singular value distribution of the gradient—via a connection to the spectral theory of separable random matrices. The entire per-layer structure (curvature exponent, rank-profile shape, effective decay, Hessian exponent) is predicted by two covariance matrices. That is the subject of our next note.

The natural follow-ups beyond that are:

Validating the spectral law at the 1–10B parameter regime (ViT-Large, LLaMA-class models).
Building a production-grade optimizer that uses $ T(\sigma_k;\, \alpha_\ell) $ per layer, benchmarked against the full second-order zoo.

The paper, code, and all frozen experimental data are public:

Code & data: 9D-Labs/9d-spectral-alignment-decomposition

Citation

@article{calvo2026spectral,
  title   = {Spectral Asymptotics of Neural Network Loss Landscapes:
             Decomposing the Curvature Exponent},
  author  = {Calvo, Anherutowa},
  journal = {9D Labs},
  year    = {2026},
  month   = {May},
  url     = {https://9dlabs.xyz/writing/curvature-along-gradient-directions}
}