How Did 2 × 2 Keep Looking Like 3?

The familiar one-dimensional journey

Why did three regimes feel like the whole story?

Because the standard picture assumes we are trying to fit the training data, and turns one knob: model complexity. Turn it from too small, through just right, to too large, and the path naturally visits underfitting, good fitting on both training and test data, and overfitting.

“Isn’t this just Goldilocks? Too simple underfits, too flexible overfits, and somewhere in the middle both training and test fit are just right.”

The classical capacity story, with all its usual knobsdrag any moving point

Expected risk for nested Gaussian linear regression

Fitted coordinates p45 Training examples n118

Training risk0.10

Test risk0.25

Task benchmarkτ = 0.40

Current readingGood fitting

Horizontal: Fitted coordinates pVertical: Risk / predicting zero

expected training riskexpected test risksame path in train/test coordinatesstart: predicting zero

Why the peak at interpolation? Near p = n, the training matrix is nearly square and can have very small nonzero singular values. Least squares divides by these small numbers, amplifying noise into large weights and large fresh errors. Beyond interpolation, the nonzero spectrum improves and this source of error recedes: the second descent in double descent (Belkin, Hsu, Ma & Mandal, 2019). The sensor example shows the same ill-conditioning peak; its later attenuation plateau is a separate effect.

Predicting zero is a starting point, not the standard benchmark for underfitting. In common usage, “good enough” means crossing a task-relevant benchmark τ. Underfitting is the early part of the capacity path where adding useful expressivity improves both train and test. Try every n and alignment setting: the train/test path never enters bad-train/good-test.

The statistical model behind this picture

Where the target signal lives p = 45 reaches 96% of the signal

Horizontal: Feature coordinate jVertical: Signal energy

Signal location j₀1

Moving the gold bump keeps the total signal energy fixed. Blue shading shows the coordinates available to the current p-coordinate fit.

This is the exact expected-risk calculation for x⃗∼N(0,I_d), y=⟨x⃗,β⃗⟩+ε, and ε∼N(0,σ²), after fitting the first p coordinates by ordinary least squares when p<n and by the minimum-norm interpolator when p>n. Let B_p=Σ_j>pβ_j² and V_p=σ²+B_p. The expected risks are

Expected train and test risks on either side of interpolation.

The divergent interpolation peak leaves a gap while it is above the drawing’s risk range. Here d=320, σ²=.12, and ‖β⃗‖²=.88, so predicting zero has risk 1. The presets redistribute that energy across early, middle, or mostly unavailable coordinates; dragging translates the current profile and renormalizes it on coordinates 1–320. The delayed and poor profiles are Gaussian bumps of widths 22 and 30; the well-aligned profile decays as exp(-|j-j₀|/14).

Feature-family alignment with the true pattern decides whether a good-training/good-test “just right” region exists at all. If alignment is poor, the classical path can go from underfitting to overfitting without ever generalizing well. Changing n moves the point where parameter count matches sample count, and with it the variance peak, but it still cannot produce bad training risk with good test risk: along this expected empirical-risk-minimization path, L_train≤L_test. That makes the three-regime story richer—but it still leaves the fourth quadrant off the route.

The familiar repair

What did traditional regularization teach us to expect?

This separate fixed-design example holds the signal and noise model fixed while varying the penalty or sample count. Start at n = 40, where the unregularized predictor interpolates noisy training labels. Positive ridge deliberately raises raw training loss, and by doing so, suppresses the influence of weak, very noisy directions and can thus lower test loss. In the familiar picture, sacrificing training fit narrows the generalization gap but keeps it positive: test risk remains above training risk.

“But regularization has always made training loss worse. Why didn’t that already reveal misfitting?”

Positive ridge spends training fit to remove contaminationdrag any moving point; vary n

Training and test risk along the ridge path

Positive ridge λ0.00 Training examples n40

Test-optimal ridge chooses its penalty using the true test error for this number of examples.

Training risk0.00

Test risk1.57

Test − train1.57

Effective degrees of freedom8.00

Horizontal: Positive ridge λ (log scale)Vertical: Normalized risk

training risestest first fallssame path in train/test coordinatestrain = test

The train/test plot uses the same compressed scale on both axes, including exact zero.

Average fraction retained in each feature group.
S = signal groups; N = noise-only groups.

Eight groups of five features. Only n changes: same target, same feature strengths, same label noise.

More data weakens the case for regularization. Hold λ fixed and increase n: averaging reduces noise in the estimated coefficients, not in fresh labels. The no-ridge test risk improves and the test-optimal ridge setting moves toward zero. The irreducible test noise and predicting zero’s score stay fixed.

Watch which side of the diagonal the path inhabits. In this experiment, positive ridge can narrow the gap all the way toward predicting zero, but it never reverses its sign.

Exact diagonal example behind this picture

The fixed design has 40 orthogonal features in eight groups of five. Each coordinate receives either floor(n/40) or ceil(n/40) independent labels μ_j+ε, with μ_j=s_jθ_j and fixed noise variance v_j. Fresh data choose a coordinate uniformly and draw a new label with that same variance. The coordinates receiving an extra observation are chosen uniformly; the curves average over that allocation and label noise. The empirical loss averages all n original labels, while test data still choose coordinates uniformly. Put m̄=n/40. A coordinate with k labels has filter q_j,k=ks_j²/(ks_j²+m̄λ). Effective degrees of freedom is the sum of the 40 coordinates’ retained fractions, each between 0 and 1. It measures how much of the fit remains after shrinkage: 40 with no ridge, approaching 0 with heavy ridge. When observation counts differ, the fractions are averaged over the allocation of labels. With Z=Σ(μ_j²+v_j), the expected risks are

Exact expected training and test risks.

At multiples of 40 all counts equal m̄, giving q_j=s_j²/(s_j²+λ) and gap 2Σv_jq_j/(m̄Z)>0. The intermediate counts keep the gap positive too. Predicting zero has training and test risk 1 for every n. The test noise floor is fixed at 0.553. Here s²=[12,8,4,.45,.25,.14,.08,.04], θ=[.9,.65,.45,0,0,0,0,0], and v=[.18,.225,.27,2.7,3,3.3,3.6,3.9]. At n=40, the ridgeless fit (λ=0) interpolates; for larger n, extra independent labels make interpolation of all their noise impossible within this fixed family. Ridge minimizes the average raw training loss plus (λ/40)‖θ‖²; this penalty convention stays fixed as n changes. Both ridge train/test plots map risk R to log(1+R/0.1), on identical 0-to-40 ranges.

This familiar sacrifice aims to get closer to the diagonal in the train/test plane. Training fit gets worse while test performance gets better, but the path approaches the diagonal from the positive-gap side. Misfitting will point the other way. Its useful predictor crosses the diagonal and keeps moving until training and test disagree in the opposite direction.

Zhang, Bengio, Hardt, Recht & Vinyals · 2017

What exactly did random labels force us to rethink?

Zhang et al. (2017), Figure 1 showed that the architecture and training procedure could realize unstructured label assignments. The capacity was really there. Generalization on true labels needs an explanation beyond an inability to fit arbitrary labels.

What changed at each step?

Random-label fitting confirmed that good training fit is not sufficient evidence of generalization. Overfitting already warned us of that; these experiments made clear that the same networks could fit perfectly even without a learnable labeling pattern.

Benign overfitting resolved part of the mystery. The ability to fit even random labels perfectly does not rule out generalization on structured data. Perfect training fit can be harmless, not only in practice but in theory too.

Benign misfitting shows that good training fit is not even necessary. In benign misfitting’s sample window, insisting on good training fit rules out good prediction within the training span. Keeping a familiar training fit anyway has a price in real resources: it can require more training examples or stronger features before the same fresh-test target becomes reachable.

Three central landmarks in the interpolation turnthe good-train column gets a happy branch

Hastie, Montanari, Rosset & Tibshirani · 2022Surprises in High-Dimensional Ridgeless Least Squares Interpolation

An exact high-dimensional account of risk around and beyond the interpolation threshold.

Bartlett, Long, Lugosi & Tsigler · 2020Benign Overfitting in Linear Regression

Spectral conditions under which the minimum-norm interpolator predicts well despite fitting noise.

Muthukumar, Vodrahalli, Subramanian & Sahai · 2020Harmless Interpolation of Noisy Data in Regression

The survival-and-contamination view of when interpolation preserves signal without too much nuisance.

The evidentiary and uniform-convergence debatewhy fit needed a new explanation

Zhang, Bengio, Hardt, Recht & Vinyals · 2017Understanding Deep Learning Requires Rethinking Generalization

Random labels can be fit, even after true images are replaced by completely unstructured random noise.

Zhang, Bengio, Hardt, Recht & Vinyals · 2021Understanding Deep Learning (Still) Requires Rethinking Generalization

An updated retrospective after progress on interpolation and implicit bias.

Nagarajan & Kolter · 2019Uniform Convergence May Be Unable to Explain Generalization

A warning about broad two-sided bounds for selected overparameterized predictors.

Koehler, Zhou, Sutherland & Srebro · 2021Uniform Convergence of Interpolators

A useful counterweight: restricted norm-sensitive bounds can explain some benign interpolation.

Now bring back both axes

Which question kept which square offscreen?

The table earns its place once we see how the familiar stories traversed it. Each button below asks a real historical question—and shows the region that question naturally crops away.

The train/test tabletraining error → · test error ↑

Good train

Bad train

Bad test

3good / badOverfitting

Excellent empirical fit does not survive fresh data.

1bad / badUnderfitting

The learned rule captures neither the observations nor the pattern well enough.

Good test

2good / goodGood fitting

Both training and test performance are good. Classical generalization and benign overfitting live here.

bad / goodMisfit Learning

Useful prediction can require poor empirical fit.

Question being asked

What that leaves offscreen

The story is elementary once the right picture is drawn: two binary judgments make four possibilities. The familiar paths and tests had reasons to emphasize three of them. Benign misfitting asks us to remember the fourth.

A blue predictor presents its report card to a puzzled evaluator; the lower-right quadrant highlights bad training fit and good test performance. — One clean linear-regression problem. One very strange report card.

Optional lab: what can norm and training fit certify?Inside: why norm helps, a pair with identical norm and fit, and a learning/relabeling continuation

Continue to the five elephants ↓

Uniform convergence remains useful. Koehler, Zhou, Sutherland & Srebro (2021) show why perfect training fit together with a small enough norm can certify generalization in Gaussian linear regression. Their bounds are uniform over interpolators in a norm ball. The conjunction matters: predicting zero has a wonderfully small norm, but has learned nothing.

Why does that conjunction help? Strong signal features fit cheaply in predictor norm; nuisance-only memorization costs more. Compare that discount among interpolators first, then ask what changes when we allow misfitting.

Beyond interpolation: optimistic rates

Zhou, Koehler, Sutherland & Srebro’s Optimistic Rates extends this approach beyond zero training error. Schematically, the error bound has the form

At interpolation, the training-loss term disappears. The paper covers more than Euclidean norm balls; covariance choices, assumptions and finite-sample terms matter. This schematic expression explains the motivation, not a numerical theorem being evaluated in the plots below.

Live fitting costs

1. How much of the fit came at a discount?

γ = 100, d−1 = 50,000.
y = s₁, Var(y) = 1.
The true label is the standardized signal coordinate.

Make training iid Stylized inputs; label batch 1

Training examples n5,000

How much squared norm does the signal save?

Squared norm saved Still needed

Squared-norm cost: all three fit exactly

True labelsNuisance features only

1

True labelsSignal + nuisance features

Random labelsSignal + nuisance featuresdraw 1

Each row uses its features’ minimum-norm interpolator. The top two fit identical true labels; the bottom fits independent standard Gaussian labels on the same full inputs. Costs are in units of n/(d−1); gray guides mark 1.

Treat norm and training error as budgets. A guarantee that worsens as either grows must cover every predictor inside both budgets. Once zero fits inside, it cannot promise population error below 1. Zero has norm zero and training error L₀ = ||y||²/n, which varies across iid draws. This obstruction works for any norm.

But misfitting can help. What if the guarantee is allowed to reward a larger training error? Let’s hold both numbers exactly fixed.

Stored exact-match comparison · its own fixed inputs

2. Do norm and training error identify the best predictor?

Choose the training span’s best prediction at each training-error cap. Then hold its norm and training error exactly fixed. How badly can another do?

Stored setting

iid draw

Loading the saved comparison…

Training error relative to zero · square-root scale

Both predictors: norm² cost · training error .Fresh error

What this draw shows: cost budgets

This draw: exact summaries

What the asymptotic results show. In the one-spike regime, matching the optimum’s exact norm and training error forces prediction error toward zero, though more slowly. With a much stronger feature and a target relying on weaker ones, the same summaries can admit very poor prediction. These are asymptotic claims under specific Gaussian-model scaling assumptions; this finite draw does not prove them.

Training-error cap

InterpolatorBest in span

Best prediction hereFresh error

The same feature strength that makes a useful response cheap also can make a wrong response cheap, in norm and in the training-loss balance.

How the training-error balance stays unchanged

Change from the useful predictor, with its norm held fixed
Contribution	Training-loss change	Fresh-error change

At best in span, the poor predictor gives up some of the useful response. That frees norm for a nuisance adjustment. The adjustment lowers training loss enough to pay for a wrong response on the stronger feature.

This is nuisance inside the predictor with a new consequence, together with the unequal costs of feature responses.

More features alone do not cause this failure. A much stronger feature can cause trouble even when the task uses it, provided the task also relies on weaker features.

How to read this plot

Best prediction within each training cap Highest compatible error at that reference’s costs

Both curves show population fresh error. The bracket pairs predictors with exactly the same norm and training error; the norm changes along the path. Each upper point matches its own reference. Segments guide the eye between stored states; they do not locate exact crossings or claim every intermediate error is attained.

The reference uses the known population model: it is a benchmark tradeoff, not a training algorithm that knows unseen labels. Here the best predictor always uses its full training-error allowance.

The slider measures residual length. Zero training error fixes the whole residual vector; positive training error fixes only its length. Interpolation is unique within the training span. Does norm recover what the residual length leaves unspecified?

Both curves initially fall as the best predictor improves. Their separation becomes substantial deeper into misfitting. Different positions fix different norm/loss pairs, so the highest compatible error need not increase along the path.

Axes stay fixed across draws within each setting. The slider ends at this draw’s Best-in-span point. Norm² cost is squared Euclidean norm in units of n/d₀. Predicting zero has population fresh error 1; its empirical training error varies across draws.

These named settings have different feature strengths and nuisance dimensions. Switching settings changes more than the number of spikes. The live controls above and below do not change this stored comparison.

Check the nuisance accounting

The shared norm charge is squared norm N multiplied by nuisance dimension per example, τ = d₀/n. The response charge accounts for the chosen signal responses and subtracts the norm they use; residual alignment and nuisance readback supply the balance.

Each predictor’s training error: a shared norm charge plus three signed contributions
Training-error contribution	Best here	Matching predictor

The response charge includes the norm needed to produce those responses. The remaining norm can be used for nuisance. Its alignment with the training residual and its readback strength supply the other two contributions. Compare the two predictors’ entries: their differences balance, not either predictor’s adjustment alone.

A small adjustment can matter because it interacts with a large residual. If r is the training residual and δ its change,

Pointing against the residual makes the first term negative. Differences in nuisance readback strength can also lower the training loss. Neither saving need carry over to fresh examples.

The response costs and exact balance

Write a span predictor as u = Xα. Let S contain standardized signal values (one sample per row), Γ = diag(γⱼ) their strengths, and W the nuisance Gram matrix. Define the signal responses and nuisance variance by a = ΓSᵀα, ν = αᵀWα. Then

Training error reads the stored nuisance back at the same samples:

Separate the minimum-norm way to produce a response from the remaining nuisance freedom. Set

With B = Γ⁻¹ + Γ⁻¹T⁻¹Γ⁻¹, the response requires squared norm aᵀBa; the rest is wᵀWw. Define

The table uses this exact identity:

The response charge is a quadratic: write F(a) = aᵀQa − 2ℓᵀa + c. For a change δ from the reference responses, each feature's row is its change evaluated at that same reference. The coupling gets its own row:

These terms add to the full response-charge change without choosing an order. The individual response changes need not be feasible predictors on their own. Alignment and readback split the training loss, not the fresh error; the fresh-error column therefore reports the total change in nuisance variance separately. A dash means that this decomposition does not assign a separate fresh-error contribution to that row.

The interpolation gain reappears. With even nuisance and orthonormal sampled signals (W = d₀I, SᵀS/n = I),

The interpolation gain is χⱼⁱⁿᵗ = γⱼ/(γⱼ + τ). Features that interpolation preserves more easily also carry a smaller response-error charge in this balance. See those gains on the Function Space mixing board →

Download the accounting for all stored states. The geometry below compares the original draw with even nuisance and with a weaker strong feature, on every two-spike draw.

Where does the room come from?

This experiment compares the selected two-spike draw with even nuisance and with a weaker strong feature, changing one thing at a time. It needs JavaScript to display the saved response regions.

Why don’t extra directions alone suffice? Two idealized examples

These smaller examples explain the geometry under exact symmetry. They are separate from the iid draw above; their best fresh errors are 0.30 and 0.47.

With even nuisance and orthogonal sampled signals, equal strengths make the response constraints touch. When both features carry the target, unequal strengths can permit a separate, smaller effect.

Numerical evidence, exact settings and all stored stops

What persists beyond these draws? Monotone norm/loss budgets must include zero once it fits inside. Exact summaries can do better: in the one-spike theorem, even the largest compatible risk tends to zero, at rate 4R^−2/3 versus the optimum’s R⁻¹, where R = γ²n/d₀ and d₀ is the nuisance dimension. The specified tuned SGD output’s exact summaries also certify vanishing risk. The multispike theorem instead allows arbitrarily poor exact matches when strength disparity grows sufficiently fast and the fixed target has nonzero weaker-feature mass. Comparable strengths and strongest-eigenspace targets remain protected; disparity alone is insufficient. This is not uniform as weaker target mass vanishes, nor a failure theorem for every SGD output.

The browser values are numerical certificates from saved coefficients and matrices, not interval proofs. Each reference minimizes population error within its cap; compatible maxima need a separate feasible witness and global bound. Unresolved maxima remain visible.

How the reference follows the training-cap frontier

Set K = W + SΓSᵀ, H = W + SΓ²Sᵀ, h = SΓβ, y = Sβ. For a nonnegative multiplier, the reference is

The positive-definite quadratic makes this the unique minimizer of population error plus μ times training error. A predictor with no more training error and better population error would contradict that minimum. Thus the reference is best within its cap, and also best at its exact norm and training error. At μ = 0 it is best in span. The interpolation endpoint is evaluated separately as K⁻¹y.

Download displayed bank Download coefficients and certificates Verification report Provenance Training-loss accounting

The one-spike contrast and the two-spike example use different named settings. These finite data do not establish a universal crossing or a theorem about unsampled caps. The two-spike setting is a fixed-task example, separate from the asymptotic theorem’s scaling assumptions.

Stored states for this draw; all errors are population fresh errors except the explicitly labeled training column.
Training / zero	Norm² cost	Best error	Matching error	Upper bound	Status

In the saved two-spike setting, best-in-span fresh error is 0.031–0.052 across 20 iid draws. On every draw, another span predictor has the same Euclidean norm and training error but fresh error above 1. Download all displayed values and the verification report.

In the opening two-spike comparison, both predictors share the entire training dataset. Any diagnostic computed from that dataset alone, including the costs of fitting random labels, is shared too. Such a diagnostic cannot break this tie for a guarantee covering every span predictor. Information about the predictor’s direction, or a justified restriction to a learning rule, asks a different question.

Sometimes the two numbers determine the whole predictor. At a given norm, ridge finds the smallest training error attainable within the span. No other span predictor has that exact pair. This includes negative ridge when its objective still has a unique minimum within the span. Move away from this boundary, and the same two numbers can leave much more room.

Return to the main story: the five elephants ↓

When do the two numbers determine the predictor?Inside: a norm-and-training-error map, with landmarks for best in span, best ridge and one-pass SGD

Preprint landmark map · a separate fixed draw

The Preprint landmark map uses a separate, fixed draw from the preprint. Its colours show the highest fresh error compatible with each norm and training-error pair. With JavaScript, place the three predictors on the map and zoom into the narrow region near the ridge boundary.

Given the training data, a unique pair identifies the predictor. That still does not tell a learner its fresh error. Here we know the data-generating model, so we can calculate that error and ask how much it varies among predictors with the same two numbers.

What else does SGD tell us? Ridge chooses its coefficients jointly; SGD fixes each coefficient when its example arrives. Its triangular coefficient system records this order and learning rule. Can that structure explain why SGD predicts well even when its norm and training error also permit a poor predictor? The landmark makes this a concrete question; it does not yet supply such a guarantee.

Continue: learning and relabelingInside: live inflation and SGD paths, and what random-label fitting might still reveal

Changing labels costs no extra examples. The comparisons below ask whether fitting random labels helps diagnose a representation or distinguish learning rules. Their separation can suggest what to investigate; it does not by itself supply a guarantee for arbitrary predictors.

Return to the live experiment · shared fitting-cost controls

Learning and relabeling

Start with 5,000 examples, then choose 100 examples to see what changes. These live comparisons share the sample count and model controls with the fitting-cost experiment above. The saved exact-match comparison uses its own stated setting.

You know a predictor’s norm and how well it fits the labels it was trained on. How much does that tell you about fresh-test risk? The green path inflates the signal-label interpolator. Each red path inflates an interpolator of different random labels on the same inputs. What if SGD learns those random labels instead? The purple paths sweep its learning rate; the blue path does the same for the signal labels. Each point is a complete pass from zero, on the same inputs in the same order.

Inflation u = c u_int

One-pass SGD learning rate eta

Inflation minimum and SGD minimum are oracle benchmarks, chosen using true test risk.

Match inflation to SGD’s training loss

Compare noise inflations with

What the two numbers reveal

Signal interpolatorSignal inflationZeroNoise inflationsSignal SGDNoise SGD

Inflation test minimumSGD test minimum

The red circles meet the selected predictor at the same training loss. The blue and purple diamonds use the same selected learning rate. Squared norm uses the same units as above. Both axes are log-like, with zero included; a gray-bordered triangle marks a point beyond the fixed range.

What happens on fresh examples?

Each predictor’s own task
Predictor	Norm² cost	Train	Test

There are 64 random-label draws, with every input held fixed. SGD also learns the first 12 of these same label vectors. The red inflation paths start at zero training loss; the purple SGD paths start at predicting zero as the learning rate approaches zero. Their fresh-test labels are independent standard Gaussian noise, so their population test risk can never beat 1. Brackets show the middle 80% over label randomness only.

At n = 100, the interpolator has training loss 0, squared-norm cost 0.833, and test risk 0.696. The best-in-span predictor has training loss 22.224, squared-norm cost 27.211, and test risk 0.0476. A valid bound nondecreasing in norm and training loss must cover the interpolator; it must also cover predicting zero, whose test risk is 1.

The noise-label paths evaluate each predictor on its own labeling task. They are useful comparisons, but they are not witnesses against a guarantee on the original labels. The budget readout above uses only original-task predictors and reports the strongest eligible witness it has actually checked.

What is the math behind this picture, and what survives with iid data?Inside: the ray formulas, two unequal spikes, the noise-label and SGD constructions, the monotone-bound argument, the iid result

The marked minima use the known population test risk. The inflation minimum is analytic; the SGD minimum is found numerically over the displayed learning-rate range 0 ≤ eta ≤ 0.0005, scanning 81 logarithmically spaced positive rates plus zero and refining each detected local minimum.

To match a true-label SGD predictor with training loss t, set c = 1 + sqrt(t/A), A = ||y||²/n. Both predictors then have exactly the same empirical loss on exactly the same labels. The lower-norm SGD example searches this same rate range for a reversal with training loss above predicting zero’s, test risk below one, and both points inside the plot. It is unavailable when none is found on the current inputs.

Here d counts all feature coordinates. With one spike there are d−1 nuisance coordinates; with two spikes there are d−2.

For one spike, write x_i = (√γ s_i, z_i), y_i = s_i. Stylized training sets every s_i = 1 and uses orthogonal nuisances of squared length (d−1). In iid mode, all signal and nuisance coordinates are independent standard Gaussians before the signal coordinate is scaled. Fresh inputs have covariance diag(γ, I) in both modes. These plots show exact realized risks.

Let W = Z^TZ be the nuisance Gram matrix, and define q = s^TW⁻¹s, A = ||s||²/n. Fitting through nuisance alone costs squared norm q; using the signal reduces this to q/(1+γq). Along the inflated ray, the exact identities for either training mode are

The signal supplies γq/(1+γq) of each original label; the nuisance supplies the rest. The cartoon is recovered by setting q = n/(d−1), A = 1. For iid Gaussian data, Koehler et al.’s Corollaries 1–2 relate uniform risk over norm-bounded interpolators to squared norm times covariance trace divided by sample count (with finite-sample terms and appropriate assumptions). That is the connection to the discount picture, not an assertion that these realized risks are their bound.

For the noise-label comparison, draw e ∼ N(0, I) independently and fit u_noise = XK⁻¹e, where K = W + γss^T. Writing b = s^TW⁻¹e, h = e^TW⁻¹e, its signal response is a = γb/(1+γq) and its nuisance squared norm is v = h − 2ab + a²q. For each inflation, the losses are on that noise-label task:

The displayed red circles choose their own inflation factors to match the inspected predictor’s training loss t exactly: c_e = 1 + sqrt(nt/||e||²). Thus their spread measures the range of norm costs and test risks at a fixed empirical loss.

What changes with two unequal spikes? Now x_i = (sqrt(100) s_i1, sqrt(20) s_i2, z_i), y_i = (s_i1 + s_i2)/sqrt(2). Both standardized signals matter equally, and fresh labels still have variance 1. Iid mode draws both signal columns independently. Stylized mode uses a constant first column and an alternating-sign second column; they are orthogonal when n is even. The exact formulas below also handle odd sample counts and their small cross-term.

Let S = [s₁, s₂], Gamma = diag(100, 20), beta = (1, 1)/sqrt(2). For either label vector ell = S beta or ell = e, the interpolator’s signal-response vector and nuisance squared norm are

The best inflation has c_* = max{1, <a,beta>/(||a||² + v)}. It optimizes this one ray, not the whole span. A common scalar generally cannot calibrate both modes. The norm-saving bar compares the interpolator’s squared norm with y^TW⁻¹y, the cost of fitting through nuisance alone; it does not assign a common survival factor to two unequal signals.

Where do the SGD paths come from? At each learning rate, start from zero and take one complete pass in the original sample order, using u_i = u_i−1 + eta(ell_i − <x_i,u_i−1>)x_i, with ell_i = y_i for signal labels or ell_i = e_i for noise labels. Its training loss is evaluated on all examples after that pass. Every SGD marker is an exact evaluated endpoint, together with its paired noise-label endpoints; the connecting curves sample rates from zero to 5 × 10⁻⁴. With signal responses a_j = sqrt(gamma_j) u[j] and nuisance squared norm v, the signal-task risk is ||a − beta||² + v and the noise-task risk is 1 + ||a||² + v. The reference rate is the predetermined choice eta = log(1+2n gamma²/(d−k))/(2n gamma), where k is the number of spikes, using gamma = 100 in either mode.

What stays fixed across the noise-label comparisons? All signal columns and the nuisance Gram matrix W. We draw 64 independent label vectors, keep every one, and solve against that same matrix for every fit. “Fresh labels” changes only those vectors. “Fresh inputs” draws a new signal and nuisance matrix, keeping the random-label vectors for comparison.

How can this be fast at 5,000 examples? The iid nuisance Gram is sampled exactly by Bartlett’s decomposition: W = LL^T, where L_ii ∼ χ_d−k−i+1, L_ij ∼ N(0,1) below the diagonal independently. Triangular solves avoid forming an inverse. No rotation mixes the training examples: that would preserve batch-fit metrics but change SGD.

For the obstruction, fix one training dataset D. Suppose C_D(r, ℓ) is nondecreasing in both arguments and bounds test risk for every span predictor on that dataset. Whenever a predictor has at least the interpolator’s norm, monotonicity and validity at the interpolator give

If its training loss is at least predicting zero’s realized loss A, predicting zero gives the lower limit 1. These are lower limits on an upper bound, not proposed bounds. The noise-label paths serve a different purpose: they compare the information in norm and empirical loss across labeling tasks. They are not additional witnesses for a data-dependent bound on the signal-labeled dataset.

The single-spike iid result is stronger than one ray: in the paper’s misfitting window, with high probability, every accurate span predictor has larger norm and larger training loss than the interpolator. To see why the norm follows the loss, put K = X^TX, let z be the training predictions, and let y be the labels. The squared norm is z^TK⁻¹z. When K/(d−k) ≈ I, a large residual forces large predictions and hence a large norm. More precisely, writing L₀ = ||y||²/n,

Thus the same ordering obstruction holds on that high-probability event. A high-probability certificate is compared on the intersection with its own validity event. This is a limitation of a particular family of certificates, not of uniform convergence as a whole.

Fit points are not the whole shape

Five ways to draw an elephant.

“With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.” Attributed to John von Neumann by Enrico Fermi, as reported by Freeman Dyson.

A pink-elephant intermission

Treat the outline as the population target and a few dots on it as training observations. Interpolation only tells us what happened at those dots. Generalization asks what happened to the entire animal.

One spectral model sits behind all five drawings. The fixed target is split into an orthogonal circle mode and an elephant-shaped correction. Each important mode can survive by a different amount, while high-frequency Fourier modes repair what is missing at the observed points. The atlas changes the feature spectrum; the scale control then changes one predictor within those features. Every curve is computed from this model.

What exactly survived, and what contaminated it? Inside: a useful vocabulary for telling deflation from wild oscillation

The Stylized page uses this ledger to expose the training-cost reversal. Muthukumar and collaborators gave interpolation a particularly visual accounting language. Survival is how much of the shared target pattern remains in the learned rule. Contamination is the sample-specific nuisance that rides along with it. The two numbers answer different questions, which is why two interpolators can both hit every dot and look completely different between them.

Too little survivalThe important low-frequency body deflates between observations.

Too much contaminationCheap high-frequency repairs turn into violent off-sample wiggles.

Benign misfittingInflation restores the shared body, while the sample-specific repairs overshoot their own dots.

This vocabulary was developed to understand harmless interpolation. The new twist is calibration under squared loss: a predictor can have little contamination relative to its surviving signal and still be badly under-scaled. Pushing survival to the right level then exposes the training-cost reversal. The live ledger below lets you change circle survival, elephant-correction survival, and contamination separately.

Muthukumar et al. (2021): survival and contamination in classification and regression

How can two interpolators fail so differently?

same quadrant, two very different-looking failures

One target, different spectra

Benign misfitting

Train error / predicting zero0.00

Shape error / predicting zero0.00

fixed targetsurviving shapewith nuisance tailtraining residual

Survival bars run from 0 to 200%; the black tick marks 100%.

circle survival46%

correction survival46%

Contamination: squared-error contribution0.04

For these same features, what does changing the scale repair? Try Both shrink → Best shape. Then Not enough correction → Best shape. Why one scale cannot repair unequal modes →

Predictor scale c1.00

Number of fit points21

More fit points make the observation grid denser; the feature bank and target stay fixed.

Circle strength30

Elephant-correction strength30

Try differential survival

White rings are observed fit points; a blue center means the prediction coincides at this plot scale. Black is surviving shared shape. Blue adds the nuisance tail and shows empirical residuals.

Survival is the important low-frequency shape left between dots. Contamination is the high-frequency residue used to hit the dots.

Change the nuisance feature bank

Include the nuisance completion, not just the surviving signal

Tail blocks M18

Tail strength ν1

Each block adds 21 Fourier frequencies to each output coordinate.

Exact curve formula behind this picture

T = C + E, ⟨C,E⟩ = 0, F_c = cI

C is the circle; E is its orthogonal correction using harmonics 0–8. Normalize these vector-valued features by their uniform-measure L² norms and give them strengths γ_C and γ_E. Each output coordinate also receives sine/cosine features at frequencies 41 through 40+21M, each of strength ν. This entire feature bank stays fixed when n changes. The displayed predictor solves the exact 2n-by-2n interpolation Gram system, without ridge, on the n-point regular grid. Its signal part is s_CC+s_EE; its Fourier tail supplies the rest. Current survival is cs_C, cs_E, not just the interpolator's s values. Normalized training error is (c-1)². Shape error is [(cs_C-1)²‖C‖²+(cs_E-1)²‖E‖²+‖cN‖²]/‖T‖², where N is the interpolator's Fourier tail; cN is its current scaled tail. At n=21 the alias blocks are orthogonal and s_j=γ_j/(γ_j+2Mν); at other n the full Gram is used. The default γ_C=γ_E=30, M=18, ν=1, c*=2.115 gives survival 96.2%, training risk 1.244, and shape risk 0.0385. Predicting zero, c=0, is the baseline. This is an exact vector-valued Fourier example, not an additional preprint theorem. Switching off the nuisance completion shows only the surviving signal, without refitting. In that view the training error is computed directly on the displayed sample predictions, rather than using (c-1)².

Too little survival leaves a deflated body with sample spikes. Too much contamination makes the whole curve thrash. Differential survival distorts which parts of the elephant remain. And moving beyond interpolation can restore the shared shape while pulling every fit-point correction past its label.

Have I seen deflated overfitting in an ordinary kernel method?Inside: a lab with a ridge-free Gaussian kernel, sliders for its lengthscale and the sample count

From isolated bumps to a globally coupled interpolant

Horizontal: Input xVertical: Function value

target shapekernel interpolant

Yes: overly localized interpolation.

With a ridge-free Gaussian kernel fit, a very short lengthscale builds isolated bumps around the observations. Every dot is fit, but the curve sags back toward its zero baseline between them. Increase the lengthscale and neighboring observations begin to hold up the shape together. But how many observations does that take? Try a wide kernel with just two or three points: the resulting curve can miss the target’s shape, even while hitting all its training dots.

Kernel lengthscale0.021 Training points n9

The short-lengthscale case is often described as small-bandwidth or spiky interpolation. It is adjacent to, but not identical with, the successful spiky-smooth construction of Haas, Holzmüller, von Luxburg, and Steinwart (2023), which deliberately combines a broad surviving backbone with narrow spikes.

Exact formula behind this picture

There are n equally spaced observations on [0.055,0.945], with n adjustable from 2 to 12. The target and the plotted interval [0,1] stay fixed. Observations are interpolated with k(x,x')=exp(-(x-x')²/(2ℓ²)). The displayed coefficients solve Ka=y, with no diagonal term. For wide kernels, direct solution becomes numerically ill-conditioned even though the interpolant exists. On this equally spaced grid, k translates are a common exponential prefactor times powers of exp(hx/ℓ²). We evaluate that exact polynomial in its barycentric form, using expm1 to keep differences accurate as ℓ grows. No modes are truncated. The isolated-spike caption requires a label-independent bound: at every interval midpoint, the prediction magnitude is at most 10% of the largest label magnitude. Cancellation between opposite-sign labels alone does not trigger it. The flat limit is a degree-(n-1) polynomial interpolant; with two observations it is a line. Platte and Driscoll (2005) develop the Gaussian-interpolation connection with polynomials.

The opposite repair

What if the useful direction was already shrunk too much?

Positive ridge suppresses contamination. But the deflated elephant has the opposite problem: too little signal survived. Along the stylized span path, anti-shrinkage restores the shared shape while the sample-specific corrections overshoot their own labels.

“So after decades of being told to shrink, are you really asking me to turn the ridge dial through zero?”

Here c is the predictor's scale relative to interpolation. The SGD page draws the same outward motion along the one-spike span ray as a negative-ridge path.

Anti-shrinkage restores survivalexact one-sample stylized path

Training risk27.56

Test risk0.18

Compare Interpolator → Best in span: better fresh prediction comes with worse training fit.

Predictor scale c6.25

Horizontal: Scale relative to the interpolator, cVertical: Normalized risk (log scale)

Regularization path

Fresh-data view

Retrospective view

Numbers used in this example

With γ=30 and d−1=199, set α=c/(γ+(d−1)). Then training risk is (c-1)^2 and test risk is (γα-1)^2+(d−1)α^2. Predicting zero itself is c=0, where (training risk, test risk)=(1,1). Interpolation is c=1; the exact gap reversal is c=1.775. At c=2, a different, nonzero predictor has training risk 1 and test risk 0.560. The best-in-span predictor is at c*=6.251. The vertical axis is logarithmic. Curve segments below the displayed range are omitted rather than flattened; in particular, the exact zero training risk at c=1 is off-chart. The train/test plot retains that zero exactly. The SGD page’s exact formula labels its stylized negative-ridge path by a decreasing parameter m and calls the scale relative to interpolation a rather than c.

training risktest risksame path in train/test coordinates

The gray dot is predicting zero; gray shading marks test < train.

The minus sign was already classical

Golub and Van Loan (1980) derived the subtraction in total least squares (TLS). In standard TLS with one response and the usual singular-value gap, the normal equation subtracts s_min², the squared smallest singular value of the augmented data matrix. They observed that the resulting solution is the ridge formula evaluated at −s_min². They called TLS “a deregularizing procedure, a kind of ‘reverse’ ridge regression.”

For one training example the connection goes beyond the sign. Applying the Stylized paper’s test-error calculation to the sensor model gives the same leading subtraction as the classical known-noise correction in the limit of many equally noisy sensors. The test-tuned Stylized rule and known-noise EIV use different finite-sample subtractions. Golub, Hansen, and O’Leary (1999) studied Tikhonov regularization within TLS. Markovsky and Van Huffel (2007) review both its geometry and its errors-in-variables interpretation.

The sensor example makes the subtraction concrete. The whole reference record M₀ is one labeled training example, with known label 1; M is the incoming record of sensor readings. The noisy reference record has squared length P, and ordinary least squares divides by it. Correct that denominator by subtracting either the supplied noise contribution across p sensors (the known-noise errors-in-variables, or EIV, correction), or the total squared perpendicular residual of the reference and incoming records:

Known-noise EIV

P − pσ²

The supplied sensor-noise variance fixes the subtraction before a fresh record arrives.

TLS on the pair

P − δ_TLS(M₀, M)

Here δ_TLS = s_min²([M₀ M]): the squared smallest singular value, or the narrow direction’s squared thickness.

Both formulas subtract from the same observed denominator, but TLS selects δ_TLS using the incoming record and is therefore not one fixed reference-trained vector. Feed each rule its own reference record: TLS refits the identical pair, subtracts zero and returns the known label 1. For the opening noise draw, EIV retains its subtraction and returns 6.40.

Errors-in-variables names the measurement-error setting; the known-noise correction and TLS are approaches to it. Negative ridge names an algebraic predictor family. Misfitting concerns training and fresh-prediction errors.

Optional lab: compare EIV and TLSInside: scatter, replay, and a denominator stress test

Continue to the earlier clues ↓

After the denominator correction

What if the correction must be learned from the geometry?

The sensor page explains the physical attenuation argument: ordinary least squares divides by the inflated reference length P, while known-noise EIV divides by P − pσ². Here we begin with that correction already understood and ask two questions it leaves open.

Can the subtraction be inferred when the noise variance is not supplied? And what happens when a finite random denominator approaches or crosses zero? Choose Orthogonal fit (TLS) to see the perpendicular residuals that answer the first question. The optional denominator stress test below explores the second.

p = 100 sensors
Noise variance σ² = 4 on both measurements
Fresh physical quantity Y = 1.6

The interactive figure has not loaded. You can still read its explanation and equations below.

What should the scatter show?

Known-noise errors-in-variables correction (EIV) Orthogonal fit (total least squares, TLS)

Which sensor readings are supplied now?

Fresh sensor readings Use the same recorded reference readings again

Each dot is one sensor. The single labeled training example is the whole reference vector, with known label 1.

Vertical: fresh sensor reading M_j Horizontal: reference sensor reading M_0j

Physical relation: reference onlyOLS — before correctionSelected fitted lineTLS perpendicular residuals

The faint gray line is ordinary least squares. With fresh readings, noise in the reference can flatten its slope. EIV corrects using the supplied noise variance; TLS uses the cloud’s perpendicular spread. Select TLS to see those residuals.
The dashed line is the true relation, Y = 1.6; the fitted rules never see it.

Begin with the supplied-noise correction

Known-noise EIV subtracts the expected reference-noise contribution from the observed denominator. The sensor lesson explains why this reverses attenuation.

Ŷ_EIV = J/(P − pσ²).

Now compare that supplied subtraction with the amount inferred from the two-record geometry.

Known-noise EIV gives slope 1.94; the true value is Y = 1.6.

Squared reference length P = ∑ M_0j²: —
Product sum J = ∑ M_0jM_j: —
Expected reference-noise contribution pσ²: 400

Outputs from the noisy sensor records

Each output is a slope in the sensor-coordinate scatter and an estimate of the single physical quantity. The squared errors below concern this one supplied record, not expected test risk.

Known-noise EIV

Denominator P − pσ²: —
Output: —
Squared error: —

TLS on this pair

Denominator P − δ_TLS: —
Output: —
Squared error: —

We picked this noise draw because it shows attenuation clearly while both corrected denominators stay safely positive. Test error, and which correction did better, played no part.

Both unregularized corrections can have infinite expected squared error in this Gaussian model. These single examples show the mechanisms, not average performance.

When can the correction go too far?Inside: compare two noise draws, including a sign reversal

The expected noise contribution pσ² can exceed the reference record’s observed squared length P. The denominator then becomes negative. If the product sum J is positive, the corrected slope reverses direction. It is the stationary quotient of a quadratic whose curvature is now negative, not a least-squares minimum.

Compare the two saved noise draws using fresh sensor readings.

Noise realization

Default noise Alternate noise — chosen for EIV reversal

EIV subtracts 400 from P = 474.1, leaving 74.1. J = 143.8 is positive, so its slope is 1.94.

Default noise makes OLS attenuation visible, with both corrected denominators at least 5% of P.

Alternate noise keeps the same sensor responses and redraws only the measurement noise; we picked it because EIV reverses direction there while TLS stays clear of its small-denominator region. These examples show two mechanisms, not their frequency; the selection procedure is recorded below.

The cloud’s thickness tells TLS about the noise

What if the noise variance is not given? Total least squares allows both plotted coordinates to move. With the same units and equal, independent noise variance on the two axes, the shortest moves to the fitted line are perpendicular. Its long direction tells us the relation between the measurements; its perpendicular thickness tells us about the noise. In the TLS overlay, the residual segments show that narrow direction.

Write δ_TLS for the minimum total squared perpendicular residual to a through-origin line. It equals the squared smallest singular value of the two-column matrix of sensor readings:

δ_TLS = s_min²([M₀ M]),Ŷ_TLS = J/(P − δ_TLS).

As equally noisy sensors are added, δ_TLS/p approaches σ². EIV is given pσ²; TLS estimates the corresponding subtraction from the two-record geometry. This is another small single-spike picture: one signal direction surrounded by isotropic noise. Different units or unequal noise variances require appropriate weighting or whitening.

Now use the control to reuse the recorded reference readings. Fixed EIV overshoots on its positive-denominator branch. But TLS fits [M₀ M₀] anew: the points lie exactly on the diagonal, δ_TLS = 0, and TLS returns one. TLS changes its correction for the incoming pair; EIV keeps the correction fixed by the reference and specified variance. Sharing a subtractive equation does not force the same training behavior.

The sensor model, its targets, and the domains of the two corrections

M₀ = a + N₀ and M = aY + N, with ‖a‖² = p. Across sensors and across records, N₀ and N have independent Gaussian entries of mean zero and variance σ² > 0. They are independent of Y. The experiment shown conditions on Y = 1.6. When taking ensemble averages, the model uses Y ∼ N(0, 1).

P = ‖M₀‖², J = M₀^TM, and Q = ‖M‖². The scalar expected-noise identity is E‖N₀‖² = pσ². EIV requires P ≠ pσ². The finite TLS slope requires p ≥ 2 and the strict singular-value gap s_min([M₀ M]) < ‖M₀‖. The gap holds almost surely for fresh records in this model; the one-sensor case is degenerate.

The scatter contains one point per sensor. The single labeled learning example is the entire reference vector with label one. The physical quantity Y here is not the noiseless normalized label in the preprints. The Sensors page explains that distinction.

Why perpendicular thickness estimates the noise variance

At fixed Y, as sensors are added with σ² fixed, the matrix of uncentered sums of squares and products converges in probability:

[M₀ M]^T[M₀ M]/p → [1, Y]^T[1, Y] + σ²I₂.

The clean matrix has rank one. Isotropic noise adds the same amount in both directions without changing the limiting principal direction. The smaller eigenvalue tends to σ², giving δ_TLS/p → σ². This is the uncentered matrix appropriate to a through-origin fit, not an empirically centered covariance matrix. The limiting TLS line recovers Y in probability; this does not guarantee finite expected squared risk at finite width.

On reuse, [M₀ M₀] has rank one, δ_TLS = 0, and J = P. Query dependence comes from putting M in the response column here; it is not an assertion about every use of TLS.

The exact ridge connection and the earlier inflation example

The one-reference Gram is the scalar matrix [P]. Its ridge predictor is M₀/(P + λ), so the positive-denominator family P − δ is exactly the ridge formula with λ = −δ and 0 ≤ δ < P.

The anti-shrinkage plot above instead has G = γ + d−1 = 229. Its inflation parameter satisfies G + λ = G/c. Thus c > 1 corresponds to −G < λ < 0, and c = 1 to interpolation. Shifts at or beyond λ = −G are not on that positive-denominator path. Its test-optimal shift is λ = G(1/c* − 1).

In raw sensor units, that Stylized one-sample calculation gives Δ_tuned = pσ²(p − 1)/(p + σ²), whereas the known-noise correction gives Δ_EIV = pσ². Their ratio (p − 1)/(p + σ²) tends to one at fixed noise variance. This leading agreement does not identify their finite-sample experiments or establish finite expected risk for a ratio transplanted into random sensor data.

Subtracting δI moves every Gram eigenvalue downward. While the corrected matrix remains positive definite, that reduces its positive-definiteness margin. At zero it is singular; beyond zero it is no longer positive definite. The TLS rank-gap condition keeps its shifted Gram positive definite. A specified EIV subtraction can cross zero.

Vanishing denominators and the two different risk warnings

EIV crosses a pole at P = pσ². The record-averaged squared risk is infinite because records can come arbitrarily close to that pole. Its consistency in probability does not imply convergence of expected squared errors. TLS has a different failure: its denominator stays positive on the strict-gap domain, but can become arbitrarily small for nearly orthogonal queries. Even conditional on a nonzero reference record, the one-record TLS prediction has infinite expected fresh squared error in this Gaussian model.

The law P/σ² is noncentral chi-square with p degrees of freedom and noncentrality p/σ². At p = 16 and σ² = 4, its lower-tail probability at 16 is about 0.307: the full correction has a nonpositive denominator. Using the smaller transplanted tuned subtraction instead gives threshold 12 and probability about 0.109. These are denominator sign probabilities, not the frequency of near-pole events.

For the full known-noise subtraction, the probability of a nonpositive denominator first falls below one percent at p = 71 when σ² = 2, p = 232 when σ² = 4, and p = 1013 when σ² = 9. In the 100-sensor example with σ² = 4, the probability that (P − 400)/P ≤ −0.05 is about 0.034. That condition alone does not guarantee a visible sign-reversed slope or a regular TLS fit.

The plot always reports the denominator. At zero the slope is undefined. Within 10⁻⁹P of zero, it also suppresses the slope because division is numerically unreliable. Outside that region it reports the slope, but draws the line only when |denominator|/P ≥ 0.05.

Sources, statistical assumptions, and the two regression orientations

Fuller (1987), §1.1.1, equation (1.1.6), printed p. 3, gives the classical scalar attenuation factor. The known-noise correction is in §2.2, equation (2.2.20) and Exercise 9(a), printed pp. 107 and 124. Frost and Thompson (2000) discuss regression dilution and correction using repeated measurements.

Golub and Van Loan (1980), Theorem 4.1 and equation (4.9), p. 888, derive the negative identity shift under the singular-value gap. Page 889 gives the ridge identity and the quoted deregularizing interpretation. Golub, Hansen, and O’Leary (1999) later analyze Tikhonov regularization within TLS.

For statistical consistency, assume fixed dimension, augmented error vectors independent and identically distributed across observations with zero mean and covariance proportional to the identity, and a fixed clean design whose normalized Gram matrix has a positive-definite limit. Under these sufficient assumptions the subtraction estimates the measurement-noise contribution, and TLS coefficients converge in probability to the true regression coefficients as observations accumulate. Markovsky and Van Huffel (2007), §§2.3–2.4, review this interpretation. It is not a finite-sample unbiasedness or expected-risk claim.

Modern high-dimensional errors-in-variables methods likewise subtract the predictor-error covariance from a feature moment matrix. Loh and Wainwright (2012), equation (2.8), and Sørensen, Frigessi, and Thoresen (2015), equation (4.1), retain that generally indefinite correction while restricting the coefficients to an ℓ₁-ball. Datta and Zou (2017) instead replace the corrected matrix by a nearby positive-semidefinite one.

Here the p channels are the observations and the one reference record is the regressor: P − pσ² is the overdetermined, one-regressor known-noise correction. With more features than observations, the corresponding corrected quadratic is generally indefinite and unbounded without an additional restriction. Transposing the regression changes which Gram matrix is corrected.

How the two noise draws were chosen

Both records come from a seeded generator: an FNV-1a hash of each stream’s name, combined with the draw number, seeds mulberry32, and Box–Muller turns its output into Gaussians. We tried noise draws in order, starting from draw 0, and kept the first that met each list. The default list is 0 < J/P < 1.6 and EIV and TLS denominators at least 5% of P; draw 0, the first tried, already meets it. The alternate keeps the same responses; its list is an EIV denominator at most −5% of P, JY > 0, a reversed EIV slope of magnitude 1 to 8, and a TLS denominator at least 5% of P; draw 15, the sixteenth tried, is the first to meet it.

Exact settings and numerical values

The machine-readable record preserves the seeds, selection criteria and unrounded sums for reproduction. Its field tau stores the TLS subtraction δ_TLS.

Numerical realization audit will appear when the interaction initializes.

When exact fits cost TLS nothing

Now use twenty labeled examples, each a vector of a thousand sensor readings: the design matrix has 20 example rows and 1,000 feature columns. The training equations already admit exact fits without altering any measured entry. Ordinary TLS therefore reaches zero measurement-correction cost at an interpolant. Its minimum-norm choice is the same minimum-norm interpolator shown in the sensor page’s point-by-point experiment.

In the negative-ridge interpretation, the TLS subtraction has fallen to zero—not because the sensors are noiseless, but because the underdetermined data already admit an exact fit. Tuned negative ridge makes a nonzero subtraction and improves fresh prediction in that experiment by giving up exact training fit.

Why the underdetermined TLS subtraction is zero

For A = Xᵀ, ordinary unweighted TLS minimizes ‖E‖²F + ‖e‖² subject to (A + E)w = y + e. Since Aw = y is already solvable, E = 0 and e = 0 attain the minimum. The full augmented Gram matrix has structural zero eigenvalues; a thin SVD’s smallest positive singular value is not the subtraction here.

Negative ridge and inflation were real clues.

Earlier work showed that the optimal ridge penalty can be zero or negative in high-dimensional regression. More recent work studies when simply inflating the minimum-norm interpolator improves expected test error. These are genuine antecedents.

The retrospective axis was not the headline.

One plausible reason the quadrant stayed unnoticed is that these works ask about estimator risk, regularization, or new-task postprocessing. LaBonte, Lai, and Muthukumar (2025) make that last lens especially clear: structured attenuation in a classification interpolator can support few-shot regression recovery. The old points belong to the classification task, so retrospective regression loss on them is not the natural headline.

Kobak, Lomond & Sanchez · 2020The Optimal Ridge Penalty Can Be Zero or Negative

Implicit regularization can make additional positive ridge harmful.

Tsigler & Bartlett · 2023Benign Overfitting in Ridge Regression

General conditions include regimes with negative optimal regularization.

Freeman · 2025Shrinkage to Infinity

Scalar inflation can improve the minimum-norm interpolator.

LaBonte, Lai & Muthukumar · 2025Task Shift: From Classification to Regression

Structured attenuation can be decoded by a few-shot regression postprocessor.

Earlier spectral ingredients, a different question

With enough independent and identically distributed (iid) examples, the training span can contain a useful predictor. More examples may be needed before the minimum-norm interpolator predicts well. Bartlett, Long, Lugosi, and Tsigler (2020) supplied two effective ranks: one compares total spectral mass to its largest eigenvalue; the other measures how widely that mass is shared. Those summaries help explain the separation. Keeping training loss visible asks a further question: what does useful prediction cost on the old examples?

In the iid comparison below, each input has 21 independent Gaussian coordinates: one signal coordinate with variance γ = 20 and twenty nuisance coordinates. Its label is the signal coordinate divided by √γ. Both spectra keep all twenty nuisance modes and total nuisance variance 180; only the concentration changes.

Call the useful-prediction threshold n_useful: the sample count at which the span begins to support good fresh predictions. Call the useful-interpolation threshold n_interp: the sample count at which the minimum-norm interpolator begins to predict well. These thresholds describe a gradual improvement as examples accumulate, not exact minimum sample counts.

Reference sample-count thresholds for the two iid spectra above.
Nuisance spectrum	Useful prediction n_useful	Useful interpolation n_interp
Spread out20 modes of variance 9	4.05	9
Concentrated10 modes of variance 17; 10 of variance 1	7.25	9

Concentrating the same nuisance variance moves the useful-prediction threshold from 4.05 to 7.25, while the interpolation threshold stays at 9: the interval narrows. Function Space shows what concentrating nuisance does to an actual learned curve →

Where do these sample-count thresholds come from?Inside: the iid setup, effective ranks, and the earlier papers

2020When can minimum-norm interpolation generalize?

Bartlett, Long, Lugosi, and Tsigler use mass-to-peak and participation effective ranks to describe a benign tail.

2023What changes with bias and ridge?

Tsigler and Bartlett keep the head-tail geometry, add ridge, and identify settings where the optimal penalty is negative.

This seriesWhat is the empirical price before interpolation works?

Between the two thresholds, ask how a useful predictor performs on its own training examples.

The ranks below describe the whole input covariance, including the signal. Both examples have total variance 20 + 180 = 200. The signal variance 20 is the largest eigenvalue in each. Function Space reports the ranks of the nuisance alone, r₁ and R₁ (Bartlett et al.’s k = 1); nuisance participation gives n_useful = n_interp²/R₁.

Mass-to-peak rank r₀Total spectral mass / largest eigenvalue

Participation rank R₀(Total mass)² / sum of squared eigenvalues

Effective ranks for each spectrum in the comparison.
Nuisance spectrum	Mass to peak r₀	Participation R₀
Spread out	10.00	19.80
Concentrated	10.00	12.12

Write ν_j for the variance of nuisance mode j. In this single-spike iid model, the reference thresholds are

n_useful = Σ nuisance variance² / γ² = r₀²/R₀ − 1.

n_interp = Σ nuisance variance / γ = r₀ − 1.

For the spread-out spectrum, the squared nuisance variances add to 20 × 9² = 1620, giving n_useful = 1620/20² = 4.05. For the concentrated spectrum, they add to 10 × 17² + 10 × 1² = 2900, giving n_useful = 2900/20² = 7.25. The total nuisance variance stays 180, so both have n_interp = 180/20 = 9.

Concentration lowers participation and brings the useful-prediction threshold closer to the interpolation threshold. The interval in which we can seek good prediction before interpolation works becomes smaller.

Bartlett, Long, Lugosi, and Tsigler (2020) · Tsigler and Bartlett (2023)

Same sign, different mechanism

Haven’t negative generalization gaps happened before?

Certainly. The sign alone is not the contribution. The revealing question is what made training harder than test, or whether anything about the problem changed at all.

Traditional regularization appears in the first column as a contrast: in its familiar role, it narrows a positive gap. SCO is stochastic convex optimization. Click any column heading to inspect a mechanism, or any row heading to compare one question across mechanisms.

Choose a mechanism; its answers appear beside the same questions.

positive → smaller

negative by design

negative

negative; can grow

same train/test distribution

training inputs are augmented or selected to be harder

same train/test distribution

ordinary loss plus a penalty, or a compressed model

usually the ordinary loss, evaluated on altered inputs

constructed nonsmooth convex loss

ordinary squared loss

ridge: same class, different rule; compression may restrict the class

usually the same predictor class

one algorithm: one-pass SGD

linear rules in the training span

not the mechanism

algorithm-specific

forced by span geometry

Selected mechanism

Why focus on combinations of the training inputs?

With no preferred signal direction, projecting a linear predictor into their span preserves its training predictions and never raises fresh error averaged over the signal directions still compatible with the data.

Why the Span? Read the full argument →

Selected comparison question

A negative gap is unsurprising when training is deliberately harder. Here the same true rule fits train and test perfectly. The training points look difficult only to a predictor carrying the nuisance residue they helped create.

Is this just benign underfitting in stochastic convex optimization?Inside: a table comparing model, magnitude and scope

The family resemblance is real. Here is where the model, magnitude, and scope differ.

Axis	SCO benign underfitting	Benign misfitting
Model	Constructed nonsmooth convex loss and oracle.	Ordinary squared-loss regression with iid Gaussian train and test data in the same-distribution result.
Magnitude	Constant-order empirical error and gap in the canonical construction.	Vanishing test risk with a training-error lower bound that can diverge relative to the zero baseline.
Scope	One algorithm: one-pass, without-replacement SGD.	A training-error lower bound for every proper span predictor reaching useful test error; SGD shows that useful prediction is achievable.

Koren, Livni, Mansour & Sherman · 2022Benign Underfitting of Stochastic Gradient Descent

One-pass SGD generalizes on a constructed instance despite large empirical risk.

Zhang, Cisse, Dauphin & Lopez-Paz · 2018Mixup

An example where the effective training problem is deliberately altered.

Yun et al. · 2019CutMix

Another reason raw train and ordinary test losses need not be comparable.

“But surely a learning rate is either stable or unstable. How can the answer depend on which example comes next?”

One update rule, two intellectual homes

How can the same SGD step be useful and optimization-unstable?

Large compared with what? The rulers below show why different stability questions have different answers.

The Robbins–Monro (1951) stochastic-approximation and online-learning heritage treats each example as a new draw from a population process. One pass is natural, the covariance Σ controls the average dynamics, and pre-update loss is a meaningful fresh-data diagnostic.

The empirical-optimization heritage usually fixes a finite training set, reuses it, and asks whether the empirical objective descends. The full Gram matrix governs batch-gradient dynamics; individual-example Hessians govern single-point updates.

A rate below 2/γ makes the mean signal response converge. Controlling the expected squared test error across stochastic runs is a different question. The rulers below distinguish these questions; the optional reuse experiment compares points with new nuisance directions against a returning point.

One-pass SGD skis through fresh powder: shared signal reinforces while each example brings new nuisance. A returning point instead meets its own nuisance stored in the predictor. Which rate is safe depends on the quantity being controlled, whether examples return, and how long learning continues.

Want the algorithm story before the intellectual history? Follow one exact pass, inspect which training points carry the iterate, and watch reuse turn quiet updates into gradient spikes. Open the SGD page →

One rate, five different questions

Averaging before or after scoring makes a difference. At one input, two learned predictors might have errors +10 and −10. Their mean predictor is exact there, but their mean squared error is 100. The mean predictor averages across independent training runs; it need not resemble any one trained predictor.

Use a single signal of strength γ = 100, d−1 = 50,000 unit-variance nuisance coordinates, and clean labels y = x[1]/√γ. Every predictor starts at zero. GD is full gradient descent. The first three rulers use the Gaussian population. The last two use the SGD page’s 100 stylized training examples, each with its own orthogonal nuisance direction. The examples share the signal direction.

Learning rate η = 2 × 10⁻⁴

Each ruler has the same logarithmic range, 10⁻⁶ to 0.03. Blue marks rates that satisfy the stated guarantee; the dark marker is your rate. The first four questions concern indefinite repetition. The last concerns one immediate update. None of these rulers marks the boundary of misfitting.

Every update uses half squared loss. Full-batch GD averages over its 100 examples: one batch step uses all 100, while one SGD step uses one. The One-step matched-filter rate, η = 1/γ, calibrates the signal in one full-batch step for this single-spike, fixed-signal training set: average each input times its label, then scale. For iid Gaussian training, it calibrates the mean response across datasets. The SGD page shows what happens when that batch step is repeated.

The mean predictor can hide growing errors: inspect the fresh-stream momentsInside: the risk of the mean predictor against the mean risk, out to 5,000 fresh examples

Now draw a fresh independent Gaussian example at every SGD update. Imagine running this many times. Averaging the predictors first cancels random errors; scoring each run first does not. The gold curve is the risk of the mean predictor. The green curve is the mean test risk of the actual predictors. The shaded gap between them is the cost of variation across runs.

A useful first pass can still fail later. Even with continually fresh data, the larger rate can eventually let accumulated nuisance overwhelm the signal.

Risk of the mean predictor Mean test risk Cost of variation across runs

Exact expectations, not a sampled run or a median. Predicting zero has risk 1. Curves leave the frame outside 0.001–10; the readouts retain their values in scientific notation even beyond ordinary floating-point range. The dotted vertical line marks 100 updates. Every later example is still fresh.

These buttons search the first 5,000 fresh updates at the selected rate.

Inspect an update 100 fresh updates

Risk of the mean predictor
Mean test risk
Cost of variation

Which loss does each update descend, and where do the boundaries come from?Inside: the two report cards, and the boundaries for population GD, fresh iid SGD, full-batch GD and one selected-point update

Two report cards. Population risk R(u) = E(uᵀx−y)² scores fresh examples. Empirical loss L(u) = (1/n) Σ(uᵀxᵢ−yᵢ)² scores the fixed training set. The rate multiplies the gradient of half of the chosen loss.

Population GD uses the exact population gradient. Along covariance eigenvalue λ, each error coordinate is multiplied by 1−ηλ. Here the largest eigenvalue is γ, so risk converges for 0 < η < 2/γ. This is an oracle reference: it already knows the population.
Fresh iid SGD has that same gradient on average. Independence of the next example makes the mean iterate follow population GD. From zero, its signed signal error is E hₜ = (1−ηγ)ᵗ. Squaring after taking that average loses the variability between runs; expected risk has the smaller boundary derived in the next drawer.
Full-batch GD uses all rows of X at once. With K = XXᵀ and training residual r = Xu−y,

u⁺ = u − (η/n)Xᵀr, r⁺ = (I−ηK/n)r.

Thus 0 < η < 2n/λmax(K) contracts the training residual. In the displayed model with orthogonal nuisance components, K = (d−1)I + γ11ᵀ, giving 2/(γ+(d−1)/n). With iid training examples, use their actual Gram matrix. If the loss is summed instead of averaged, divide the quoted rate by n to get the same update.

One selected-point SGD update uses only its point’s gradient:

u⁺ = u + η(yᵢ−uᵀxᵢ)xᵢ, rᵢ⁺ = (1−η‖xᵢ‖²)rᵢ.

The point’s nonzero residual shrinks precisely when 0 < η < 2/‖xᵢ‖². Here every training point has squared norm γ+d−1. An iid point has its own realized norm. To protect every point with one rate, use the each-point stability threshold 2/maxⱼ‖xⱼ‖². The two thresholds coincide here because the training points have equal norms. The loss on the whole training set can still rise.

Why can the first pass help if the fresh stream eventually fails?

Write the signal calibration error as hₜ = 1−√γ uₜ[1]. Let Bₜ = E hₜ² and Nₜ = E‖uₜ,⊥‖². The latter is the nuisance contribution to test error. With d₀ = d−1, independent Gaussian inputs give the exact two-number recurrence

(Bₜ₊₁, Nₜ₊₁) follow the displayed two-by-two Gaussian moment recurrence, starting from (1,0).

Every finite update is a polynomial in finitely many Gaussian examples, so these moments are finite. The calculation rescales the positive recurrence to retain very large or small values; it tracks the variation separately instead of subtracting nearly equal risks.

Expected risk is Bₜ+Nₜ. Signal error creates nuisance error, which feeds back into later signal updates: the off-diagonal entries quantify that exchange. In contrast, the mean predictor’s risk is just (E hₜ)² = (1−ηγ)²ᵗ. Their difference is exactly the prediction variance across runs:

E R(uₜ) = R(E uₜ) + E‖uₜ−E uₜ‖²Σ.

Here ‖v‖²Σ = vᵀΣv weights each coordinate by its fresh-input variance, with Σ = diag(γ,1,…,1).

There are two contraction tests: |1−ηγ| < 1 for the mean predictor, and a largest eigenvalue below 1 for the two-moment recurrence matrix above. The latter controls expected risk, including variation across runs. The boundary is the smaller positive root ηms of

γ(d₀+3)ηms² − (d₀+3γ+2)ηms + 2 = 0.

The expected-risk boundary is about 4 × 10⁻⁵, just below the selected-point boundary. At the Useful first pass rate 2 × 10⁻⁴, mean test error over fresh Gaussian streams falls to 0.0768 after 100 updates, reaches about 0.0662 at update 149, then returns above 1 at update 1,852. The mean predictor keeps improving throughout. Stopping after a useful finite pass and converging forever answer different questions.

The successful growing single-spike families accumulate enough signal correction over one pass while keeping the accumulated nuisance cost small. At the displayed Useful first pass rate, continuing forever on this fixed-dimensional problem eventually allows the feedback to dominate. Several unequal signals add another obstruction: even their desired calibrations disagree.

At the same predictor, what changes when a point returns?Inside: two next-point residuals at one fixed iterate, with their contraction rulers

Wasn’t early stopping supposed to act like positive ridge?Inside: why the sign can change within a single fresh pass

For small steps, stopping early leaves the shared signal short of the minimum-norm interpolator’s response: the familiar positive-ridge-like shrinkage. Raise the step size while still using each point only once, and that response can cross the interpolator’s. A signed-ridge predictor with the same signal response would now need a negative penalty. One-pass SGD reaches that anti-shrinkage with chronological, unequal point weights, so its test and training costs differ from ridge’s.

Move the learning-rate dial across the exact crossover on the SGD page. Its earlier causal Gram comparison shows why the two rules still differ.

One predictor, two possible next pointsbefore update 1 of 200

At one fixed predictor, would the next point be fresh or familiar? Follow the complete first-pass and return trajectory on the SGD page; here compare the two possible next residuals.

Choose the predictor after this update0 / 200

The 100 observations share signal energy γ=100. Each point also carries its own orthogonal nuisance with energy d−1=50,000.

Point with a new nuisance direction Shared-signal residual

A new orthogonal nuisance direction has no stored coefficient. Its shared-signal residual is multiplied by 1−ηγ, so that component contracts when |1−ηγ|<1.

One selected point Single-example update x⃗_ix⃗_i^T

Immediately updating on the same point multiplies its residual by 1−η(γ+(d−1)). That one-point update contracts only when |1−η(γ+(d−1))|<1.

This experiment has its own learning-rate control; it does not change the five rulers above.

Learning rate η (log scale)2.00×10^-4

one-point reuse edge
2/(γ+(d−1)) best after one pass signal-contraction edge
2/γ

Fresh-test error (stylized training set)0.067

Training risk23.676

Arriving-point loss0.018

Gradient norm40.6

Same iterate, two possible next points

A new orthogonal nuisance direction and training point 1 share the same signal. These bars show residual magnitudes, not population test risk.

point with new nuisance0.13

point 1 returns9.87

Second epoch · shuffle 1: points 8, 81, 21, 75, 99, 66, 97, 83, ...

Why do two communities put different rulers on the same update? Inside: the different quantities they track, and why here their contraction edges lie far apart

Stochastic approximation

Empirical optimization

Data

Fresh stream

Frozen finite training set

Passes

One is natural

Reuse is natural

Object here

Shared-signal multiplier

Single-example residual multiplier

Diagnostic

Pre-update / fresh loss

Full empirical training loss

Contraction condition

|1−ηγ|<1

|1−η(γ+(d−1))|<1

In low dimension these viewpoints often tell compatible qualitative stories. Here (d−1)/γ is large, so these contraction edges are far apart. That open interval is where one-pass SGD can acquire shared signal while storing unstable sample-specific residue.

Exact formulas and numbers behind this experiment

The displayed training vectors satisfy x⃗_i=(√γ,√(d−1) e⃗_i), y_i=1, with n=100, γ=100, and d−1=50,000. Starting from zero, SGD uses w⃗←w⃗+η(1-⟨w⃗,x⃗_i⟩)x⃗_i. Every point, loss, and gradient in the plot is recomputed from the 100 exact span coefficients. The first pass is fixed; the second is either cyclic or a displayed deterministic shuffle generated from a numbered seed. The default η=2×10^-4 is the useful first-pass rate (the stylized preprint’s large-step trace); the gentle rate is η=10^-5. The first-pass test risk is minimized numerically from its exact finite-n formula at ; the best possible span risk is exactly (d−1)/(γ²n+(d−1))=. The minimum-norm interpolator has test risk (d−1)((d−1)+n)/(γn+(d−1))²=. During the fresh pass, α_i=η(1−ηγ)ⁱ⁻¹.

Tutorial bridgeStochastic approximation

The fresh-information tradition in which an expected objective is the natural object.

Nakkiran, Neyshabur & Sedghi · 2021The Deep Bootstrap Framework

A modern coupling between ideal online and empirical offline training.

Koren, Livni, Mansour & Sherman · 2022Benign Underfitting of SGD

Without- and with-replacement behavior separate on the constructed SCO instance.

The synthesis is the contribution

What does this series add?

The ingredients were not all new. The series makes the fourth-quadrant tradeoff itself the object: first exactly, then under iid sampling, then across unequal signals, and finally without labels.

What forces the reversal?

Stylized deterministic model

The exact frontier shows how good prediction can require bad empirical fit. One-pass SGD makes the tradeoff tangible.

Was the clean geometry essential?

Same distribution

iid Gaussian train and test data preserve the reversal. For each realized dataset, scaling its interpolator can recover the best-in-span predictor.

What if useful patterns compete?

Several unequal spikes

Scalar attenuation becomes matrix-valued distortion. Different channels want different corrections, and one learning rate must negotiate among them. One-pass SGD can still succeed, even when a single scalar correction cannot.

What if most labels disappear?

Oja

A stream learns a spectral direction; a few labels can then calibrate its predictions. Explore the interval between spectral detection and consistent recovery.

What becomes central

The frontier, not one estimator.

Earlier work often asks whether one algorithm or one regularization value generalizes. The necessity results instead fix a useful level of test error and ask how much training error every linear predictor in the training span must incur.

The naming move

Why give the fourth quadrant a name?

Earlier work may already have produced predictors in this quadrant. The contribution is to make the quadrant coherent, necessary in a natural sample window, and connected to algorithmic dynamics, spectral geometry, robustness, and nonlinear predictors that can avoid the training-loss penalty.