# The Signal and the Noise

Generalizing AdamW

Source: https://hyperstition.cc/the-signal-and-the-noise

September 10, 2026

Authors: Max Misterka, Nahom Seyoum, Adit Srivastava, Mason Eyler, AJ, Deven Pietrzak

## TL;DR

- AdamW has a privileged basis: it applies its adaptive update separately along each parameter coordinate.

- We explore generalizing these SNR-based updates to arbitrary directions, finding a new route to momentum: under certain conditions, averaging over directions recovers a scaled momentum update.

- For linearly independent gradient histories, we construct a direction whose projection onto every stored gradient is identical, achieving SNR 1 when the stabilizer is zero.

- A variant of this that adds preference toward more recent gradients gives a basis-independent optimizer that matches AdamW.

The Adam optimizer's update for a single parameter is a ratio of two running estimates. For a matrix-valued parameter with gradient $G_t$, Adam's update is

$\mathcal U_{\mathrm{Adam}}
=
\frac{\widehat M_t}{\sqrt{\widehat V_t}+\varepsilon},
\qquad\text{elementwise},
\tag{1}$

where $\widehat M_t$ and $\widehat V_t$ are EMAs of the first and second raw gradient moments.

For a scalar sequence $\{x_i\}_{i=1}^t$, define the signed signal-to-noise ratio (SNR) operator using Adam's decay rates $0\le\beta_1,\beta_2<1$ and stabilizer $\varepsilon\ge0$ (with nonzero denominator):

$\SNR\!\left(\{x_i\}_{i=1}^t\right)
:=\frac{
\displaystyle\frac{1-\beta_1}{1-\beta_1^t}\sum_{i=1}^t\beta_1^{t-i}x_i
}{
\displaystyle\sqrt{\frac{1-\beta_2}{1-\beta_2^t}\sum_{i=1}^t\beta_2^{t-i}x_i^2}+\varepsilon
}.\tag{2}$

We write $\SNR(\{x_i\};\eta)$ when replacing the stabilizer by $\eta$. Here SNR means signed mean divided by RMS, not mean divided by standard deviation. With shared moment weights its magnitude is at most $1$; this bound need not hold when $\beta_1\ne\beta_2$.

AdamW's adaptive update is $\mathcal U_{\mathrm{AdamW}}=\SNR(\{G_i\}_{i=1}^t)$, with the operator applied entrywise. The interpretation of Adam as an SNR is due to Kingma and Ba in their paper introducing Adam [[1]](https://hyperstition.cc/the-signal-and-the-noise#ref-adam).

## Adam Has a Privileged Basis

Plot: Interactive optimizer trajectories on two-dimensional loss landscapes. Rotating the coordinate basis changes the coordinate-wise Adam update while leaving the underlying objective representable in the same plane. Landscape, optimizer, rotation, and starting-point controls compare paths toward minima. The demonstration illustrates basis dependence of coordinate-wise adaptation.

Adam's update acts parameter-wise in the standard basis and does not consider interactions between the parameters. However, there is no reason for the standard basis to be optimal here. For this reason, modern pretraining optimizers such as Shampoo, Muon, and SOAP have no privileged basis. Shampoo preconditions the gradient matrices on both sides using accumulated statistics $L_t=\varepsilon I+\sum_{i=1}^t G_iG_i^\top$ and $R_t=\varepsilon I+\sum_{i=1}^t G_i^\top G_i$:

$\mathcal U_{\mathrm{Shampoo}}
=L_t^{-1/4} G_t R_t^{-1/4}.\tag{3}$

Instead of preconditioning the gradients, Muon orthogonalizes the EMA $\overline{G}_t$ of the gradients: if a compact SVD of $\overline{G}_t$ retaining only its nonzero singular values is

$\overline{G}_t=P\Sigma Q^{\top},\tag{4}$

then the idealized, unscaled polar update is $PQ^{\top}$, which sets the nonzero singular values to $1$ without changing the singular vectors. SOAP uses Shampoo's left and right preconditioners to construct an orthonormal eigenbasis and applies the Adam update formula in this basis. All of these optimizers are equivariant to shared orthogonal transformations of the gradients on both sides: if $\mathcal U(G)$ is the update function,

$\mathcal U(PGQ^{\top})=P \mathcal U(G) Q^{\top}.\tag{5}$

Shampoo and Muon are basis-independent at the cost of completely dropping the concept of SNR that was key to Adam's success. While SOAP takes advantage of the SNR and is a basis-independent optimizer because the Adam basis is chosen in an orthogonal-equivariant way from the gradients, it only computes the SNR in a single basis, leaving the vast majority of directions unprobed. This suggests that we could construct a new optimizer by computing the SNR in every direction, making it a more direct basis-independent adaptation of Adam. The SNR in a non-coordinate direction is defined as follows. For unit vectors $u\in\mathbb S^{p-1}$ and $v\in\mathbb S^{q-1}$, the component of the gradient $G_i$ along the rank-one matrix direction $uv^{\top}$ is

$\langle G_i,uv^{\top}\rangle_F=u^{\top}G_i v.\tag{6}$

This converts the sequence of gradients into a sequence of scalars that depends on the direction. Applying the SNR operator to this scalar sequence gives [Note 1: If we set $u = e_a$ and $v = e_b$, then we recover Adam's update formula for entry $(a,b)$!]

$\SNR\!\left(\{u^\top G_i v\}_{i=1}^t\right).
\tag{7}$

Now that we have a scalar SNR value for each direction in $\mathbb S^{p-1} \times \mathbb S^{q-1}$, we must also decide how to aggregate these values into an update. Adam assembles the coordinates weighted by their SNR into a full matrix:

$\mathcal U_{\mathrm{Adam}} = \sum_{a=1}^p\sum_{b=1}^q \SNR\!\left(\{e_a^\top G_i e_b\}_{i=1}^t\right)\,e_a e_b^{\top}.\tag{8}$

To aggregate the SNR over all unit rank-one directions, we can replace the sum over the basis with an integral over the entire space of directions $(u, v)$, which is the Cartesian product of the unit sphere in $\mathbb{R}^p$ and the unit sphere in $\mathbb{R}^q$: $\mathbb S^{p-1} \times \mathbb S^{q-1}$. The optimizer update then becomes

$\mathcal U_{\mathrm{integral}} = \int_{\mathbb S^{p-1}}\int_{\mathbb S^{q-1}} \SNR\!\left(\{u^\top G_i v\}_{i=1}^t\right)\,uv^\top\,d\sigma_q(v)\,d\sigma_p(u).
\tag{9}$

Here, $\sigma_k$ denotes the uniform probability measure on the sphere $\mathbb S^{k-1}$. Throughout the averaging analysis, the sampled matrix direction is $uv^\top$ with independent uniform unit vectors $u,v$.

## Averaging over All Directions Is Not Ideal

Claim. If the RMS norm in the denominator of the SNR integral is constant, then the integral collapses to a scalar multiple of $\overline G$, where $\overline G$ is the EMA of the gradients.

Here $\E_1$ and $\E_2$ average over the history index $i$ with the numerator's $\beta_1$ weights and denominator's $\beta_2$ weights, respectively, from the opening definition. This implies that $\overline G=\E_1[G_i]$.

Derive the constant-denominator average+

Let $c>0$ be the constant denominator, including the stabilizer: $c=\sqrt{\E_2[(u^\top G_i v)^2]}+\varepsilon$, independent of the unit directions $u$ and $v$.

$\def\arraystretch{0.88}\begin{aligned}
\mathcal U_{\mathrm{integral}}
&=
\frac{1}{c}
\int\!\!\int
(u^{\top}\overline Gv)\,uv^{\top}
\,d\sigma_q(v)\,d\sigma_p(u)\\
&=
\frac{1}{c}
\int\!\!\int
(uu^{\top})\,\overline G\,(vv^{\top})
\,d\sigma_q(v)\,d\sigma_p(u).
\end{aligned}\tag{10}$

By rotational symmetry, both $uu^{\top}$ and $vv^{\top}$ average to scalar multiples of the identity, so

$\def\arraystretch{0.88}\begin{aligned}
\mathcal U_{\mathrm{integral}}
&=
\frac{1}{c}
\left(
\int uu^{\top}\,d\sigma_p(u)
\right)
\overline G
\left(
\int vv^{\top}\,d\sigma_q(v)
\right)\\
&={\frac{1}{cpq}\,\overline G}.

\end{aligned}\tag{11}$

When we write $u^{\top}\overline Gv$ as a function over a product of spheres, we see that this is simply a projection of $\overline G$ and collapses to a linear functional of $\overline G$. This means the only mechanism that can possibly differentiate between directions is the denominator.

### Almost all directions are almost orthogonal to all of the gradients

Let $\mathcal S=\operatorname{span}\{G_i\}_{i=1}^s$ be the span of the gradient history, and let $P_{\mathcal S}$ be the Frobenius-orthogonal projector onto $\mathcal S$. Write $r=\dim\mathcal S\le s$.

Claim. For $H=uv^\top$ sampled as above, $\E\|P_{\mathcal S}H\|_F^2=r/(pq)$. Moreover, for every integer $k\ge1$,

$\Pr\!\left(\|P_{\mathcal S}H\|_F>2ek\sqrt{\frac{r}{pq}}\right)
\le e^{-2k}.\tag{12}$

Thus most rank-one directions are nearly orthogonal to the gradient history when $r\ll pq$, with an exponentially small tail.

Derive the rank-one projection bound+

For any fixed matrix $B\in\mathbb R^{p\times q}$,

$\def\arraystretch{0.88}\begin{aligned}
\E_{u,v}[(u^\top Bv)^2]
&=\E_v\!\left[v^\top B^\top\E_u[uu^\top]Bv\right]\\
&=\frac{1}{p}\E_v[v^\top B^\top Bv]
=\frac{\|B\|_F^2}{pq}.

\end{aligned}\tag{13}$

Choose a Frobenius-orthonormal basis $B_1,\ldots,B_r$ of $\mathcal S$. Then

$\E\|P_{\mathcal S}(uv^\top)\|_F^2
=\sum_{j=1}^r\E[(u^\top B_jv)^2]
=\frac{r}{pq}.\tag{14}$

To obtain an exponential tail, write $\|X\|_{L^{2k}}=(\E|X|^{2k})^{1/(2k)}$. For independent standard Gaussian vectors $g\in\mathbb R^p$ and $h\in\mathbb R^q$, conditioning on $h$ and applying Minkowski's inequality in a singular-vector basis of $B$ gives

$\|g^\top Bh\|_{L^{2k}}
=((2k-1)!!)^{1/(2k)}\,\big\|\|Bh\|\big\|_{L^{2k}}
\le ((2k-1)!!)^{1/k}\|B\|_F
\le 2k\|B\|_F.\tag{15}$

Gaussian radii are independent of their uniform directions, and $\E\|g\|^{2k}\ge p^k$, $\E\|h\|^{2k}\ge q^k$ by Jensen's inequality. Hence $\|u^\top Bv\|_{L^{2k}}\le 2k\|B\|_F/\sqrt{pq}$. Applying Minkowski again to the squared projection yields

$\big\|\|P_{\mathcal S}H\|_F\big\|_{L^{2k}}^2
=\left\|\sum_{j=1}^r(u^\top B_jv)^2\right\|_{L^k}
\le\sum_{j=1}^r\|u^\top B_jv\|_{L^{2k}}^2
\le\frac{(2k)^2r}{pq}.\tag{16}$

Markov's inequality applied to $\|P_{\mathcal S}H\|_F^{2k}$ at the claimed threshold gives $e^{-2k}$ (the case $r=0$ is immediate). Since every $G_i\in\mathcal S$,

$|u^\top G_i v|
=|\langle P_{\mathcal S}(uv^\top),G_i\rangle_F|
\le\|P_{\mathcal S}(uv^\top)\|_F\,\|G_i\|_F.\tag{17}$

### The nullspace can be ignored

Projecting onto $\mathcal S$ measures overlap, but can destroy rank one. To preserve rank-one directions, project the two factors separately onto the left and right gradient spans:

$\mathcal L=\operatorname{span}_i\operatorname{col}(G_i),
\qquad
\mathcal R=\operatorname{span}_i\operatorname{col}(G_i^\top),
\qquad r_L=\dim\mathcal L,\quad r_R=\dim\mathcal R.\tag{18}$

These factor spans may be full even when $\mathcal S$ has low dimension. Let $P_{\mathcal L},P_{\mathcal R}$ be their orthogonal projectors, and write

$\alpha=\|P_{\mathcal L}u\|,\qquad
\gamma=\|P_{\mathcal R}v\|,\qquad
x=\frac{P_{\mathcal L}u}{\alpha},\qquad
y=\frac{P_{\mathcal R}v}{\gamma}.\tag{19}$

Claim. Projecting the two factors and normalizing changes only the effective stabilizer:

$\SNR\!\left(\{u^\top G_i v\}_i\right)
=\SNR\!\left(\{x^\top G_i y\}_i;\frac{\varepsilon}{\alpha\gamma}\right).
\tag{20}$

When $\varepsilon=0$, the score depends only on the rank-one direction $xy^\top$.

Derive the projected sequence score+

Every tracked gradient satisfies $G_i=P_{\mathcal L}G_iP_{\mathcal R}$, so

$u^\top G_i v=\alpha\gamma\,x^\top G_i y
\qquad\text{for every }i.\tag{21}$

For $\alpha\gamma>0$, this scales the weighted mean and RMS by the same amount. Thus

$\def\arraystretch{0.88}\begin{aligned}
\SNR\!\left(\{u^\top G_i v\}_i\right)
&=\frac{\alpha\gamma\,\E_1[x^\top G_i y]}
{\alpha\gamma\sqrt{\E_2[(x^\top G_i y)^2]}+\varepsilon}\\
&=\SNR\!\left(\{x^\top G_i y\}_i;\frac{\varepsilon}{\alpha\gamma}\right).
\end{aligned}\tag{22}$

### Averaging reduces the integral to the gradient spans

Write $\mathbb S(\mathcal L),\mathbb S(\mathcal R)$ for the unit spheres in the two factor spans.

Claim. Averaging over rank-one directions cancels the components outside the left and right gradient spans:

$\mathcal U_{\mathrm{integral}}
=\E_{\alpha,\gamma,x,y}\!\left[
\alpha\gamma\,
\SNR\!\left(\{x^\top G_i y\}_i;\frac{\varepsilon}{\alpha\gamma}\right)xy^\top
\right].
\tag{23}$

For $\varepsilon=0$, the remaining integral is over $\mathbb S(\mathcal L)\times\mathbb S(\mathcal R)$, multiplied by the positive scalar $\E[\alpha]\E[\gamma]$.

Derive the rank-one integral reduction+

Write

$u=\alpha x+u_\perp,\qquad
v=\gamma y+v_\perp,\tag{24}$

where $u_\perp\in\mathcal L^\perp$ and $v_\perp\in\mathcal R^\perp$. Conditional on $\alpha,\gamma,x,y$, reflecting either orthogonal component preserves its distribution and leaves every projected gradient $u^\top G_i v$ unchanged. Thus

$\def\arraystretch{0.88}\begin{align*}
uv^\top&=\alpha\gamma xy^\top
+\alpha xv_\perp^\top+\gamma u_\perp y^\top+u_\perp v_\perp^\top,\tag{25}
\\
\E[uv^\top\mid\alpha,\gamma,x,y]&=\alpha\gamma xy^\top.\tag{26}
\end{align*}$

The same cancellation holds after multiplying by the conditional SNR, giving the reduced integral.

Under independent uniform $u,v$, the normalized projections $x,y$ are independent and uniform on their active spheres, and independent of $\alpha,\gamma$. When $\varepsilon=0$, the score is independent of the projection lengths, so their expectations factor out. A full factor span has projection length $1$ and no orthogonal component to cancel.

For a nonzero gradient history and positive second-moment weights, zero-RMS directions have measure zero, so they do not affect the zero-stabilizer integral.

We can now work within the left and right gradient spans and ask how much the denominator varies across rank-one directions there. We first account for the stabilizer $\varepsilon$.

### The nullspace amplifies the stabilizer

Now suppose $\varepsilon>0$. The nullspace clearly contributes no signal of its own but the stabilizer still acts on a scale set by it. Within the active spans, [(20)](https://hyperstition.cc/the-signal-and-the-noise#eq:matrix-nullspace-ratio) has the effective stabilizer $\varepsilon/(\alpha\gamma)$.

To see why this happens, we can work through a concrete example. Let $N$ be the mean and $D$ be the RMS measured inside the active spans. If $\alpha=\gamma=\frac{1}{4}$, then the ambient direction retains only $\alpha\gamma=\frac{1}{16}$ of every scalar gradient, and its score is

$\frac{\frac{1}{16}N}{\frac{1}{16}D+\varepsilon}
=
\frac{N}{D+16\varepsilon}.\tag{27}$

The mean and RMS clearly recover their original scale after cancellation, but $\varepsilon$ has become sixteen times larger relative to them. If $u$ and $v$ are uniformly random, the lengths of their projections onto the active spans are typically

$\alpha\approx\sqrt{\frac{r_L}{p}},
\qquad
\gamma\approx\sqrt{\frac{r_R}{q}}.\tag{28}$

Substituting these values into $\varepsilon/(\alpha\gamma)$ gives

${
\varepsilon_{\mathrm{eff}}
:=\frac{\varepsilon}{\alpha\gamma}
\approx
\varepsilon\sqrt{\frac{pq}{r_Lr_R}}}.
\tag{29}$

The concentration argument behind these typical projection lengths is included in the panel below.

Show the projection-length concentration bound+

For independent uniform directions $u\in\mathbb S^{p-1}$ and $v\in\mathbb S^{q-1}$, with $0<r_L<p$ and $0<r_R<q$,

$\alpha^2
\sim\operatorname{Beta}\!\left(\frac{r_L}{2},\frac{p-r_L}{2}\right),
\qquad
\gamma^2
\sim\operatorname{Beta}\!\left(\frac{r_R}{2},\frac{q-r_R}{2}\right).\tag{30}$

A full factor span has projection length $1$ exactly. For a proper factor span, Beta concentration gives, for $0<\eta<1$,

$\Pr\!\left(
\left|\frac{\alpha^2}{r_L/p}-1\right|>\eta
\right)
\le
4\exp\!\left(-c\eta^2\min\{r_L,p-r_L\}\right),\tag{31}$

with the analogous bound for $\gamma^2$. We interpret the following estimates asymptotically, with fixed $0<\delta\le1/2$ and $\min\{r_L,p-r_L,r_R,q-r_R\}\to\infty$. Applying the two concentration bounds and taking a union bound shows that, with probability at least $1-\delta$,

$\def\arraystretch{0.88}\begin{align*}
\alpha
&=
\sqrt{\frac{r_L}{p}}
\left[
1+O\!\left(
\sqrt{\frac{\log(1/\delta)}{\min\{r_L,p-r_L\}}}
\right)
\right],\tag{32}
\\
\gamma
&=
\sqrt{\frac{r_R}{q}}
\left[
1+O\!\left(
\sqrt{\frac{\log(1/\delta)}{\min\{r_R,q-r_R\}}}
\right)
\right].\tag{33}
\end{align*}$

Hence, in the common regime $r_L\ll p$ and $r_R\ll q$, the relative errors vanish as the active ranks grow, so taking reciprocals gives

$\frac{1}{\alpha\gamma}
=
\sqrt{\frac{pq}{r_Lr_R}}
\left[
1+O\!\left(
\sqrt{\frac{\log(1/\delta)}{r_L}}
+\sqrt{\frac{\log(1/\delta)}{r_R}}
\right)
\right].\tag{34}$

### When is the RMS nearly constant?

For independent uniform $x\in\mathbb S(\mathcal L)$ and $y\in\mathbb S(\mathcal R)$, define

$Q(x,y):=\E_2[(x^\top G_i y)^2].\tag{35}$

Its mean is

$\mu:=\E_{x,y}[Q(x,y)]
=\frac{\E_2[\|G_i\|_F^2]}{r_Lr_R},\tag{36}$

by [(13)](https://hyperstition.cc/the-signal-and-the-noise#eq:random-rank-one-overlap).

We measure the spread of gradient energy using the effective dimension (participation ratio) of a nonzero positive semidefinite second moment $C$:

$d_{\mathrm{eff}}(C)
:=\frac{(\operatorname{tr}C)^2}{\operatorname{tr}(C^2)}
=\frac{(\sum_j\sigma_j)^2}{\sum_j\sigma_j^2},\tag{37}$

where $\sigma_j$ are the singular values of $C$. This counts the effective number of directions carrying energy: $r$ equal nonzero singular values give $d_{\mathrm{eff}}=r$, while one dominant singular value gives $d_{\mathrm{eff}}\approx1$.

For each gradient in this history, write $G_i=U_i\operatorname{diag}(\sigma_{ij})V_i^\top$, where $U_i$ and $V_i$ are orthogonal matrices whose columns are its left and right singular vectors, and $\operatorname{diag}(\sigma_{ij})$ is the $p\times q$ diagonal matrix containing its singular values. Applying effective dimension to the left/output directions and right/input directions yields

$\def\arraystretch{0.88}\begin{align*}
d_L&=\frac{\bigl(\E_2[\|G_i\|_F^2]\bigr)^2}
{\|\E_2[G_iG_i^\top]\|_F^2}
=\frac{\bigl(\E_2[\sum_j\sigma_{ij}^2]\bigr)^2}
{\|\E_2[U_i\operatorname{diag}(\sigma_{ij}^2)U_i^\top]\|_F^2},\tag{38}
\\
d_R&=\frac{\bigl(\E_2[\|G_i\|_F^2]\bigr)^2}
{\|\E_2[G_i^\top G_i]\|_F^2}
=\frac{\bigl(\E_2[\sum_j\sigma_{ij}^2]\bigr)^2}
{\|\E_2[V_i\operatorname{diag}(\sigma_{ij}^2)V_i^\top]\|_F^2}.\tag{39}
\end{align*}$

If $U_i$ is constant across the history, orthogonal invariance gives

$d_L=\frac{\bigl(\sum_j\E_2[\sigma_{ij}^2]\bigr)^2}
{\sum_j\bigl(\E_2[\sigma_{ij}^2]\bigr)^2},\tag{40}$

with the same formula for $d_R$ when $V_i$ is constant. This counts the fixed singular directions carrying energy; rotation across the history can spread that energy over more directions and increase the effective dimension.

Finally, flatten each gradient to $g_i=\operatorname{vec}(G_i)\in\mathbb R^{pq}$. The effective dimension of the flattened history's second moment is

$d_G=d_{\mathrm{eff}}(\E_2[g_i g_i^\top])
=\frac{\bigl(\E_2[\|g_i\|^2]\bigr)^2}
{\|\E_2[g_i g_i^\top]\|_F^2}.\tag{41}$

This counts effective directions explored by the gradient history. If every $g_i$ is a multiple of the same vector, $d_G=1$, even when the corresponding matrix $G_i$ has large rank.

Claim. The relative variance obeys

$\frac{\operatorname{Var}_{x,y}(Q)}{\mu^2}
\le\frac{2}{d_L}+\frac{2}{d_R}+\frac{4}{d_G}.
\tag{42}$

Thus large $d_L,d_R,d_G$ make $Q/\mu$ concentrate near $1$. Conditional on any $\alpha,\gamma>0$, the denominator then concentrates relative to $\sqrt\mu+\varepsilon/(\alpha\gamma)$.

Derive the variance and conditional denominator bounds+

For this derivation, abbreviate $\mathcal N_2=\E_2[\|G_i\|_F^2]$, $\Sigma_L=\E_2[G_iG_i^\top]$, and $\Sigma_R=\E_2[G_i^\top G_i]$. For the two-gradient expectation below, sample $i$ and $j$ independently from the second-moment EMA distribution. Write $T_{ab,cd}=\E_2[(G_i)_{ab}(G_i)_{cd}]$ in orthonormal bases of the active spaces, and set

$D=\|\E_2[g_i g_i^\top]\|_F^2
=\E_{2,i,j}[\langle G_i,G_j\rangle_F^2]
=\sum_{a,b,c,d}T_{ab,cd}^2,
\qquad
R=\sum_{a,b,c,d}T_{ab,cd}T_{ad,cb}.\tag{43}$

The spherical fourth-moment identity gives

$\E_{x,y}[Q^2]
=\frac{\mathcal N_2^2+2\|\Sigma_L\|_F^2+2\|\Sigma_R\|_F^2+2D+2R}
{r_L(r_L+2)r_R(r_R+2)}.\tag{44}$

The index permutation preserves the sum of squares, so Cauchy–Schwarz gives $|R|\le D$. Using $\mu=\mathcal N_2/(r_Lr_R)$ and subtracting $1$ from $\E[Q^2]/\mu^2$ yields [(42)](https://hyperstition.cc/the-signal-and-the-noise#eq:denominator-relative-variance).

For a failure probability $\delta\in(0,1)$, set

$a_\delta=\sqrt{\frac{2/d_L+2/d_R+4/d_G}{\delta}}.\tag{45}$

Chebyshev's inequality gives $|Q/\mu-1|\le a_\delta$ with probability at least $1-\delta$ over $x,y$. When $a_\delta<1$, conditioning on $\alpha,\gamma$ gives

$\def\arraystretch{0.88}\begin{aligned}
\sqrt{(1-a_\delta)\mu}+\frac{\varepsilon}{\alpha\gamma}
&\le\sqrt Q+\frac{\varepsilon}{\alpha\gamma}\\
&\le\sqrt{(1+a_\delta)\mu}+\frac{\varepsilon}{\alpha\gamma}.

\end{aligned}\tag{46}$

In particular, the relative error of this denominator is at most $|Q/\mu-1|$. This bound holds for every choice of the projection lengths.

### From RMS concentration to momentum

We now use the variance bound to control how far the integral update is from a scalar multiple of momentum. Replacing the active-direction RMS $\sqrt Q$ by $\sqrt\mu$ in [(23)](https://hyperstition.cc/the-signal-and-the-noise#eq:matrix-span-reduction) leaves the projection lengths to average into a scalar:

$\mathcal U_{\mathrm{integral}}\approx\lambda_\varepsilon\overline G,
\qquad
\lambda_\varepsilon=\frac{1}{r_Lr_R}
\E_{\alpha,\gamma}\!\left[
\frac{\alpha\gamma}{\sqrt\mu+\varepsilon/(\alpha\gamma)}
\right].\tag{47}$

To control its error, assume the second-moment EMA assigns positive probability to every stored gradient. Suppose that $0 \le \beta_1^2 < \beta_2 < 1$. This condition is true for standard settings of the parameters: $\beta_1 = 0.9$ and $\beta_2 = 0.999$. Assuming this condition, we can use the following constant to sharply bound $|\E_1[z]|/\sqrt{\E_2[z^2]}$ for every finite sequence of gradients:

$C_{\mathrm{EMA}}
:=\frac{1-\beta_1}
{\sqrt{(1-\beta_2)(1-\beta_1^2/\beta_2)}}
=\sqrt{1+\frac{(\beta_1-\beta_2)^2}
{(1-\beta_2)(\beta_2-\beta_1^2)}}.\tag{48}$

Claim. For $\overline G\ne0$, the relative error of the momentum approximation satisfies

$\frac{\|\mathcal U_{\mathrm{integral}}-\lambda_\varepsilon\overline G\|_F}
{\lambda_\varepsilon\|\overline G\|_F}
\le C_{\mathrm{EMA}}
\frac{\sqrt{\E_2[\|G_i\|_F^2]}}{\|\overline G\|_F}
\sqrt{\frac{2}{d_L}+\frac{2}{d_R}+\frac{4}{d_G}}.
\tag{49}$

The ratio before the square root compares the RMS gradient magnitude with the magnitude of its EMA: it grows when gradients cancel in the mean. When the full right-hand side is much smaller than $1$, the integral update is close to scaled momentum in both magnitude and direction.

Derive the momentum approximation and error bound+

Define EMA weights

$p_i=\frac{1-\beta_1}{1-\beta_1^t}\beta_1^{t-i},
\qquad
q_i=\frac{1-\beta_2}{1-\beta_2^t}\beta_2^{t-i}.\tag{50}$

By Cauchy–Schwarz

$\def\arraystretch{0.88}\begin{aligned}
|\E_1[z]|^2
&=\left|\sum_{i=1}^t p_i z_i\right|^2\\
&=\left|\sum_{i=1}^t\frac{p_i}{\sqrt{q_i}}\left(\sqrt{q_i}z_i\right)\right|^2\\
&\le\left(\sum_{i=1}^t\frac{p_i^2}{q_i}\right)
\left(\sum_{i=1}^t q_i z_i^2\right)\\
&=\left(\sum_{i=1}^t\frac{p_i^2}{q_i}\right)\E_2[z^2].
\end{aligned}
\tag{51}$

Substituting the EMA weights and evaluating the geometric series (since $\beta_1^2/\beta_2<1$ by assumption),

$\def\arraystretch{0.88}\begin{aligned}
\sum_{i=1}^t\frac{p_i^2}{q_i}
&=\frac{(1-\beta_1)^2}{1-\beta_2}
\frac{1-\beta_2^t}{(1-\beta_1^t)^2}
\sum_{i=1}^t\left(\frac{\beta_1^2}{\beta_2}\right)^{t-i}\\
&=\frac{(1-\beta_1)^2}{(1-\beta_2)(1-\beta_1^2/\beta_2)}
\frac{(1-\beta_2^t)(1-\left(\beta_1^2/\beta_2\right)^t)}{(1-\beta_1^t)^2}.
\end{aligned}\tag{52}$

To bound the remaining fraction, use the identity

$\def\arraystretch{0.88}\begin{aligned}
&(1-\beta_1^t)^2-(1-\beta_2^t)(1-\left(\beta_1^2/\beta_2\right)^t)\\
&\quad=\beta_2^t-2\beta_1^t+\left(\frac{\beta_1^2}{\beta_2}\right)^t\\
&\quad=\left(\beta_2^{t/2}-\left(\beta_1^2/\beta_2\right)^{t/2}\right)^2\ge0.
\end{aligned}
\tag{53}$

Rearranging and dividing by $(1-\beta_1^t)^2>0$ gives

$\def\arraystretch{0.88}\begin{aligned}
(1-\beta_2^t)\left(1-\left(\beta_1^2/\beta_2\right)^t\right)
&\le (1-\beta_1^t)^2,\\
\frac{(1-\beta_2^t)\left(1-\left(\beta_1^2/\beta_2\right)^t\right)}
{(1-\beta_1^t)^2}
&\le 1.
\end{aligned}
\tag{54}$

Applying [(54)](https://hyperstition.cc/the-signal-and-the-noise#eq:ema-history-ratio-bound),

$\sum_{i=1}^t\frac{p_i^2}{q_i}
\le\frac{(1-\beta_1)^2}{(1-\beta_2)(1-\beta_1^2/\beta_2)}
=C_{\mathrm{EMA}}^2.\tag{55}$

Combining with [(51)](https://hyperstition.cc/the-signal-and-the-noise#eq:ema-cauchy-schwarz) yields $|\E_1[z]|\le C_{\mathrm{EMA}}\sqrt{\E_2[z^2]}$ for every finite history. $z_i\propto p_i/q_i$ yields equality in [(51)](https://hyperstition.cc/the-signal-and-the-noise#eq:ema-cauchy-schwarz), and $\sum_i p_i^2/q_i\to C_{\mathrm{EMA}}^2$ as $t\to\infty$. Hence, $C_{\mathrm{EMA}}$ is the least upper bound across $\{z_i\}$ and $t$, given $\beta_1$ and $\beta_2$.

We prove the absolute-error bound

$\def\arraystretch{0.88}\begin{aligned}
\|\mathcal U_{\mathrm{integral}}-\lambda_\varepsilon\overline G\|_F
&\le\lambda_\varepsilon C_{\mathrm{EMA}}
\sqrt{\E_2[\|G_i\|_F^2]}
\sqrt{\frac{\operatorname{Var}_{x,y}(Q)}{\mu^2}}\\
&\le\lambda_\varepsilon C_{\mathrm{EMA}}
\sqrt{\E_2[\|G_i\|_F^2]}
\sqrt{\frac{2}{d_L}+\frac{2}{d_R}+\frac{4}{d_G}}.
\end{aligned}\tag{56}$

Dividing by $\lambda_\varepsilon\|\overline G\|_F$ cancels the averaging prefactor and gives the relative-error claim.

First, the constant-RMS part satisfies

$\E_{x,y}[(x^\top\overline G y)xy^\top]
=\frac{\overline G}{r_Lr_R},\tag{57}$

which gives $\lambda_\varepsilon$ after averaging the projection lengths.

For fixed $\alpha,\gamma$, write $c=\varepsilon/(\alpha\gamma)$, $N=x^\top\overline G y$, and

$\Delta_c(x,y)=\frac{N}{\sqrt Q+c}-\frac{N}{\sqrt\mu+c}.\tag{58}$

Since $|N|\le C_{\mathrm{EMA}}\sqrt Q$,

$\def\arraystretch{0.88}\begin{aligned}
|\Delta_c(x,y)|
&\le C_{\mathrm{EMA}}\frac{|\sqrt Q-\sqrt\mu|}{\sqrt\mu+c}\\
&\le C_{\mathrm{EMA}}\frac{\sqrt\mu}{\sqrt\mu+c}
|Q/\mu-1|.
\end{aligned}\tag{59}$

These inequalities hold almost everywhere, including when $\varepsilon=0$.

To bound the averaged matrix, test it against any matrix $H$ with $\|H\|_F=1$. The spherical second-moment identity gives

$\E_{x,y}[(x^\top H y)^2]\le\frac{1}{r_Lr_R}.\tag{60}$

By Cauchy–Schwarz,

$\def\arraystretch{0.88}\begin{aligned}
\left|\left\langle\E_{x,y}[\Delta_c(x,y)xy^\top],H\right\rangle_F\right|
&\le\sqrt{\E_{x,y}[\Delta_c(x,y)^2]}
\sqrt{\E_{x,y}[(x^\top H y)^2]}\\
&\le\frac{C_{\mathrm{EMA}}\sqrt\mu}
{(\sqrt\mu+c)\sqrt{r_Lr_R}}
\sqrt{\frac{\operatorname{Var}_{x,y}(Q)}{\mu^2}}.
\end{aligned}\tag{61}$

Taking the supremum over $H$ bounds the Frobenius norm of the averaged error. Multiplying by $\alpha\gamma$ and averaging the independent projection lengths yields

$\def\arraystretch{0.88}\begin{aligned}
\|\mathcal U_{\mathrm{integral}}-\lambda_\varepsilon\overline G\|_F
&\le\frac{C_{\mathrm{EMA}}\sqrt\mu}{\sqrt{r_Lr_R}}
\E_{\alpha,\gamma}\!\left[
\frac{\alpha\gamma}{\sqrt\mu+\varepsilon/(\alpha\gamma)}
\right]
\sqrt{\frac{\operatorname{Var}_{x,y}(Q)}{\mu^2}}\\
&=\lambda_\varepsilon C_{\mathrm{EMA}}\sqrt{r_Lr_R\mu}
\sqrt{\frac{\operatorname{Var}_{x,y}(Q)}{\mu^2}}.
\end{aligned}\tag{62}$

Since $r_Lr_R\mu=\E_2[\|G_i\|_F^2]$, this proves the bound. At $\varepsilon=0$, the scale factor simplifies to

$\lambda_0=\frac{\E[\alpha]\E[\gamma]}{r_Lr_R\sqrt\mu}.\tag{63}$

### Summary of the results so far

Projection onto the active spans amplifies the stabilizer to $\varepsilon/(\alpha\gamma)$, typically $\varepsilon\sqrt{pq/(r_Lr_R)}$ in the concentration regime above. Thus when the gradient span is much lower rank than the ambient space, $\varepsilon$ can be amplified and push the integral towards a scalar multiple of momentum.

Separately, large effective dimensions of the gradient history make the RMS nearly constant. This also causes the update to be momentum-like, as bounded by [(49)](https://hyperstition.cc/the-signal-and-the-noise#eq:integral-momentum-error).

## The SNR-Maximizing Direction

In the previous section, we identified why the SNR integral fails: most directions in high-dimensional space are almost orthogonal to every gradient, and for these directions the RMS of the projection of the gradients is nearly constant, causing the integral to collapse to momentum. A natural fix is to avoid summing over these almost-orthogonal directions. Instead, we can use the SNR to filter the directions. The strongest possible filter is to take only the direction which maximizes the SNR.

We seek a matrix $M$ that maximizes $\SNR(\{\langle M,G_i\rangle_F\}_i)$, without requiring $M$ to have rank one. Let $S=\operatorname{vec}(\mathcal S)$ be the flattened gradient subspace. Writing $m=\operatorname{vec}(M)$ and $g_i=\operatorname{vec}(G_i)$ gives $\langle M,G_i\rangle_F=m^\top g_i$, so we may use ordinary linear algebra.

Recall that the SNR is $\mu / (\sqrt{\mu^2 + \sigma^2} + \varepsilon)$, where $z_i=m^\top g_i$, $\mu = \E[z_i]$, and $\sigma^2 = \operatorname{Var}(z_i)$. Since scaling $\mu$ and $\sigma$ equally can only increase the SNR due to the $\varepsilon$ term, the maximum SNR direction will live in the span of the gradients to maximize its projection onto each gradient. This causes both the numerator and denominator to be as large as possible given the direction's unit norm constraint. For the following analysis, we set $\varepsilon=0$. In the pretraining regime considered here, we find empirically that $\varepsilon$ has little effect for most directions within the gradient span. This approximates the stabilized score when $\varepsilon$ is small relative to the projected RMS; directions with very small projected RMS may still be affected.

We will also assume that $\beta_1 = \beta_2 = \beta \in (0, 1)$ and will restrict to the span $S$ of the gradients (since the SNR is independent of the direction's nullspace component) unless otherwise stated.

Let $\overline g=\E[g_i]\ne0$ be the mean gradient and $C=\E[g_i g_i^\top]$ be its uncentered second moment. Since we are restricting to $S$, we know that $C$ is positive definite.

Claim. $\SNR(\{g_i^\top m\}_i)$ is maximized by $m=C^{-1}\overline g$, uniquely up to positive rescaling. The maximum is

$\rho:=\sqrt{\overline g^\top C^{-1}\overline g}\le1,
\tag{64}$

with equality when all stored gradients are linearly independent. In this case, $g_i^\top m$ is constant across the history. Linear independence of the gradients is almost always true in pretraining, which implies the surprising conclusion that there is always a direction with SNR equal to 1 in practice.

Derive the maximizing direction and maximum SNR+

Since $C$ is positive definite, we may define the inner product $\langle x,y\rangle_C:=x^\top Cy$. By the Cauchy–Schwarz inequality,

$(\overline g^\top m)^2
=\langle C^{-1}\overline g,m\rangle_C^2
\le(\overline g^\top C^{-1}\overline g)(m^\top Cm).\tag{65}$

$\left|\SNR(\{g_i^\top m\}_i)\right|
=\frac{|\overline g^\top m|}{\sqrt{m^\top Cm}}
\le\sqrt{\overline g^\top C^{-1}\overline g}
=\rho.\tag{66}$

By the equality condition in Cauchy–Schwarz, SNR is maximized exactly at positive multiples of $C^{-1}\overline g$.

The definition of SNR gives $\rho\le1$. Suppose $\{g_i\}_i$ is linearly independent. Then

$\E[g_i(g_i^\top m-1)]
=\E[g_i g_i^\top]m-\E[g_i]
=Cm-\overline g
=CC^{-1}\overline g-\overline g
=0.\tag{67}$

By linear independence of the $g_i$ and positivity of the EMA weights, $g_i^\top m-1=0$ for every $i$, so $g_i^\top m=1$. Thus

$\SNR\!\left(\{g_i^\top m\}_i\right)
=\SNR(\{1\}_i)=1.
\tag{68}$

Maximizing SNR therefore gives a preconditioned mean gradient, $C^{-1}\overline g$. If we have per-example negative-log-likelihood gradients evaluated at fixed parameters, $C$ is a weighted empirical Fisher. In our case, $C$ also reflects changes in the parameters because we use batch gradients from different training steps. Kunstner, Balles, and Hennig [[2]](https://hyperstition.cc/the-signal-and-the-noise#ref-empirical-fisher) argue that the empirical Fisher may help account for gradient noise even when it gives a poor estimate of curvature. We ultimately arrive at the same view by choosing the direction whose gradient projections vary least relative to their mean.

### Critical points of the SNR function

Claim. The only nonzero critical points of $\SNR(\{g_i^\top m\}_i)$ are

$m=\kappa C^{-1}\overline g,
\qquad\kappa\ne0.
\tag{69}$

Positive $\kappa$ gives the global maximum $\rho$; negative $\kappa$ gives the global minimum $-\rho$. On the unit sphere, the only critical points are $\pm C^{-1}\overline g/\lVert C^{-1}\overline g\rVert$.

Derive the critical-point classification+

Differentiating directly gives

$\nabla_m \SNR(\{g_i^\top m\}_i)
=\frac{(m^\top Cm)\overline g-(\overline g^\top m)Cm}
{(m^\top Cm)^{3/2}}.\tag{70}$

Hence

$\nabla_m \SNR(\{g_i^\top m\}_i)=0
\iff(m^\top Cm)\overline g=(\overline g^\top m)Cm.
\tag{71}$

At a critical point, $\overline g^{\top}m$ cannot vanish: otherwise $(m^\top Cm)\overline g=0$, contradicting positive definiteness and $\overline g\ne0$. Multiplying by $C^{-1}$ and rearranging gives

$m=\frac{m^\top Cm}{\overline g^\top m}\,C^{-1}\overline g.
\tag{72}$

Thus $m$ is a nonzero scalar multiple of $C^{-1}\overline g$. Positive rescaling preserves SNR and negation reverses its sign, so all such $m$ are extrema.

Scale invariance also gives $m^\top\nabla_m\SNR(\{g_i^\top m\}_i)=0$, so the gradient is tangent to the sphere. The constrained critical points therefore satisfy the same equation and are exactly the two normalized directions.

Plot: Directional SNR on a torus-shaped gradient geometry. The three-dimensional view lets the reader inspect directions and their EMA-weighted SNR. The selected direction's score is shown alongside the geometry; changing direction changes the projected gradient sequence.

Figure 1. Matrix directions are restricted to $uv^\top$, with $u$ and $v$ moving in two-dimensional subspaces. The visualization uses shared EMA weights and the displayed positive stabilizer, so its SNR remains below one.

### Computing the direction with the Gram matrix

We can compute the same SNR-maximizing direction $m=C^{-1}\overline g$ without forming the full second-moment matrix. In default coordinates, $C$ is a $d\times d$ matrix, where $d=pq$. The Gram formulation instead solves an $s\times s$ system, where $s$ is the number of stored gradients.

Claim. With linearly independent gradient columns $\mathbf G=[g_1,\ldots,g_s]$ and Gram matrix $K=\mathbf G^\top\mathbf G$, the maximum-SNR solution is $m=\mathbf G K^{-1}\mathbf 1$, where $\mathbf 1\in\mathbb R^s$ is the all-ones vector. It satisfies $\mathbf G^\top m=\mathbf 1$ and is independent of the positive moment weights.

Derive the Gram-system solution+

Since $m\in\operatorname{col}(\mathbf G)$, there exists a vector $\mathbf a$ such that $m=\mathbf G\mathbf a$. Then

$Cm-\overline g
=\E[g_i(g_i^\top\mathbf G\mathbf a-1)]
=\E[g_i((K\mathbf a)_i-1)].\tag{73}$

The columns $g_i$ are linearly independent and each has positive probability, so this expectation vanishes exactly when $(K\mathbf a)_i=1$ for every $i$. Thus

$Cm=\overline g
\iff K\mathbf a=\boldsymbol{1}
\iff \mathbf a=K^{-1}\boldsymbol{1}.\tag{74}$

### The naive solution forgets $\beta$

Even though both the numerator and denominator of the SNR depend on the EMA decay $\beta$, the SNR-maximizing direction which solves $C m = \overline g$ is independent of $\beta$:

$m = \mathbf G K^{-1} \boldsymbol{1}.\tag{75}$

This happens because the columns of $\mathbf G K^{-1}$ form a basis of the gradient subspace in which the $i$th coordinate of $m$ is $g_i^\top m$. Therefore we can simply select a point where all coordinates are the same. If the columns of $\mathbf G$ were linearly dependent, we could no longer choose these coordinates independently. When equal nonzero coordinates are unattainable, we must trade off the values of $g_i^\top m$. In that case, $\beta$ governs which values we prioritize keeping close to the same.

This cancellation is undesirable for actual pretraining, where older gradients are stale and thus less useful. The usual EMA remedies this by weighting older gradients exponentially less. Our construction discards the optimizer's intended recency bias and thus produces a poor update.

To remediate this, we can reintroduce recency bias by replacing the target vector $\boldsymbol{1}$ with EMA target values with a separate decay rate $0<\beta_h<1$:

$\mathbf h=(h_1,\ldots,h_s)^\top,
\qquad h_k=\frac{1-\beta_h}{1-\beta_h^s}\beta_h^{s-k}.\tag{76}$

This gives us a weighted variant of the Gram update formula:

$m' = \mathbf G K^{-1}\mathbf h.\tag{77}$

As we will see below, this fix enables the optimizer to match AdamW's performance in a basis-free manner, which was the original goal! The SNR of the update is now a constant.

Derive the SNR after restoring recency weighting+

The target values $h_i=g_i^\top m'$ are the prescribed gradient projections. The EMA weights $\pi_i=\frac{1-\beta}{1-\beta^s}\beta^{s-i}$, with $\sum_i\pi_i=1$, use decay $\beta$ to compute the SNR's mean and second moment.

Since $\mathbf G^\top m'=KK^{-1}\mathbf h=\mathbf h$, the projected history is exactly $\{h_i\}_i$. Hence

$\def\arraystretch{0.88}\begin{aligned}
\SNR\!\left(\{g_i^\top m'\}_i\right)
&=\SNR(\{h_i\}_i)\\
&=\frac{\E[h_i]}{\sqrt{\E[h_i^2]}}\\
&=\frac{\sum_i\pi_i h_i}{\sqrt{\sum_i\pi_i h_i^2}}.

\end{aligned}\tag{78}$

For fixed decay rates $\beta_h$, $\beta$ and history length $s$, the score is independent of the gradients. For positive target values, it is below $1$ unless all $h_i$ are equal.

### Optimizer comparisons

The following figure shows the learning curves measuring the Gram optimizer's performance for pretraining a 286M-parameter model. The parameter $c$ sets $\beta_h$ for the recency weighting: $\beta_h = 1 - c/k$, where $k = 60$ is the number of previous gradients which we save. At $c=0$, we use the uniform-target limit $\beta_h\to1$, giving $h_k=1/s$. The $c = 0$ and $c = 3$ curves are the learning curves for the standard Gram optimizer and the recency-weighted variant, respectively. All curves are shown at their optimal learning rate and weight decay settings. As the curves show, standard Gram loses to momentum, while recency-weighted Gram matches AdamW by the end of the training run. This shows that we have found a basis-independent variant of AdamW which preserves its performance.

The figure below the learning curve plot displays the SNR measured on held-out batches. As the figure indicates, the held-out SNR for both Gram and AdamW is similar to the in-sample SNR, which implies that the Gram optimizer does not overfit to the sampled batches.

Plot: Optimizer comparison curves from the article's benchmark records. Panels compare recorded training behavior of optimizer variants; each panel retains its own benchmark context. The comparison tests whether optimizing directional SNR translates into better optimization performance.

Plot: SNR on independent gradients for Nanochat in 60-gradient windows. Gram and AdamW-style updates are scored both on the gradients used to construct them and on independent-batch gradients along the same training trajectory. Controls change score weights and switch between optimizer-setting sweeps and in-sample-to-held-out comparisons. In-sample optimization is distinguished from held-out performance.

### Alternatives to SNR maximization

So far, we have shown why the SNR integral over all directions recovers momentum and have provided a fix: taking only the maximum SNR direction. However, there is another way to fix the integral: instead of maximizing the SNR, we can replace the integral with a sum over a select set of directions. This recovers Adam if we choose the standard basis. But there is no reason to choose the standard basis. Instead, we can choose the orthonormal eigenbasis of $C = \E[g_i g_i^\top]$ on $S$. Denote the orthonormal eigenvectors as $m_j$ such that $C m_j = \lambda_j m_j$. Since $C$ is positive definite on $S$, $\lambda_j > 0$. Since the SNR denominator in the $m_j$ direction is $\sqrt{m_j^\top C m_j} = \sqrt{\lambda_j}$, the sum becomes

$u_{\mathrm{eig}}
:=\sum_{j=1}^{r}\SNR(\{g_i^\top m_j\}_i)m_j
=\sum_{j=1}^{r}\frac{m_j^\top\overline g}{\sqrt{\lambda_j}}m_j
=C^{-1/2}\overline g.
\tag{79}$

This is identical to the SNR-maximizing direction except we replace the $-1$ exponent on $C$ with $-\frac{1}{2}$. The optimizer $C^{-1/2}\overline g$ is also known as full-matrix AdaGrad [[3]](https://hyperstition.cc/the-signal-and-the-noise#ref-adagrad).

## Tokenwise Gradient Structure

For much of the preceding discussion, we have been considering the gradients as vectors $g_k$ in parameter space with no additional structure. However, there is much more structure to a transformer's parameters than just a vector. We considered matrix structure in the original discussion of the integral, but we ignored it later. There is one additional piece of structure which we haven't discussed yet: the raw gradients split into a weighted sum of tokenwise gradients over all tokens in each batch.

Claim. At a single application of a linear map $y=Wx$, the local weight-gradient contribution is $\delta x^\top$, where $\delta=\nabla_y\mathcal L$ and $\mathcal L$ is the total training loss. It has rank at most one; the full shared-weight gradient sums these contributions over applications.

Derive the local outer-product gradient+

Consider a single matrix $W$ in the model. Fix a single token $t$. For that token's forward pass, let $x$ be the activation entering $W$ and let $y=Wx$ be the output. Let $\mathcal L$ be the total training loss and $\delta=\nabla_y\mathcal L$ be $y$'s gradient, combining contributions from all token losses that depend on this activation. Then, by the chain rule, the local contribution is

$(G_t^{\mathrm{tok}})_{ab}
=\sum_j\frac{\partial\mathcal L}{\partial y_j}\frac{\partial y_j}{\partial W_{ab}}
=\delta_a x_b
\quad\Longrightarrow\quad
G_t^{\mathrm{tok}}=\delta x^\top.
\tag{80}$

Plot: Interactive PCA projections of gradient clouds. The display projects high-dimensional gradient samples into principal-component coordinates to make their geometry visible. Controls choose the displayed geometry and viewpoint. This is a low-dimensional projection, not a depiction of all gradient dimensions.

### Tokenwise structure gives an overdetermined maximum SNR direction

We can reuse the SNR-maximizing direction from Sec. [3](https://hyperstition.cc/the-signal-and-the-noise#sec:snr-max), now applying it to the per-token gradients $g_t=\operatorname{vec}(\delta_t x_t^\top)$ rather than the batch gradients. With $\E$ averaging over tokens in the batch (and across previous batches if we use momentum), define $\overline g=\E[g_t]$ and $C=\E[g_tg_t^\top]$. The same formula gives

$m=C^{-1}\overline g,
\qquad
\max_m\SNR(\{g_t^\top m\}_t)
=\rho=\sqrt{\overline g^\top C^{-1}\overline g}.\tag{81}$

While the formula is unchanged, this can be a different regime. When the number of token contributions exceeds the matrix's parameter dimension $d=pq$, the matrix of flattened token gradients is wide rather than tall. The equal-projection equations $g_t^\top m=1$ are then overdetermined and generally cannot all be satisfied. The maximum SNR $\rho$ is strictly below $1$ unless a direction has the same positive projection on every token gradient. Assuming the token gradients span parameter space, $C$ is full rank and $Cm=\overline g$ still has a unique solution.

The critical-point analysis from the same section also applies: when $\overline g\ne0$, this SNR objective has exactly two critical points on the unit sphere, and they are antipodes. Following its spherical gradient gives a flow with an antipodal source-sink pair.

The following figure shows the structure of this gradient flow on a 3D example. For certain parameter settings, the gradient flow is highly non-geodesic, implying that unlike the directional derivative, SNR does not simply measure the alignment of a direction with a fixed axis like $C^{-1}\overline g$.

Plot: Directional SNR on spherical gradient geometry. A three-dimensional sphere represents candidate directions; selecting a direction shows its EMA-weighted projected-gradient SNR. This complements the torus example with a different geometry.

## Conclusion

Our goal was to generalize AdamW's SNR-based update rule beyond its privileged basis. We saw that integrating over all directions can recover momentum, while the naive maximum-SNR direction loses the preference for recent gradients. The Gram solution makes the update computable without constructing the full gradient second moment. Restoring recency weighting gives a basis-invariant optimizer with final validation loss comparable to AdamW's in our pretraining run and lower loss at the first measured checkpoint.

In the previous section, we introduced tokenwise structure, which gives us a way to decompose the gradient such that we get an overdetermined SNR-maximizing direction. Since the Gram optimizer depends on the EMA decay rate in the overdetermined case, this is a more natural optimizer than the artificially-reweighted Gram $c=3$ case. Tokenwise structure suggests many more open directions which we leave for future research.

## References

- Diederik P. Kingma and Jimmy Ba (2015). [Adam: A Method for Stochastic Optimization](https://arxiv.org/abs/1412.6980).

- Frederik Kunstner, Lukas Balles, and Philipp Hennig (2019). [Limitations of the Empirical Fisher Approximation for Natural Gradient Descent](https://arxiv.org/abs/1905.12558).

- John Duchi, Elad Hazan, and Yoram Singer (2011). [Adaptive Subgradient Methods for Online Learning and Stochastic Optimization](https://jmlr.org/papers/v12/duchi11a.html).

## Cite this article

Max Misterka, Nahom Seyoum, Adit Srivastava, Mason Eyler, AJ, and Deven Pietrzak. (September 10, 2026). The Signal and the Noise: Generalizing AdamW. Hyperstition. https://hyperstition.cc/the-signal-and-the-noise