A physical route through double descent

Noisy measurements, good predictions

and — UC Berkeley

If you landed here first: benign misfitting means predicting fresh examples well while fitting the training examples worse than predicting zero.

Imperfect sensors measure one shared physical quantity. More sensors give more coordinates, not more labeled examples: a physical model of growing feature dimension.

“With enough sensors, shouldn’t the noise average away?”

It can—but the way we learn to combine the readings matters. Minimum-norm interpolation can leave the shared signal shrunken even after fresh noise has largely averaged away. The correlator, which weights each sensor by how its readings correlate with the known calibration labels, and negative ridge recover more of that signal, and the price is a worse fit to the very readings that taught them.

One object. Many imperfect scales.

Imagine measuring an object's weight relative to a reference weight. Put each object on the same rack of scales. Each scale’s unknown response stays fixed; each measurement adds new noise. The list of readings is one feature vector.

  1. CalibrateMeasure objects whose true weights you know.
  2. LearnUse those labeled rows to choose how to combine the readings.
  3. PredictMeasure a new object. Its weight is the missing entry.
Each row is one object; each sensor supplies a column.

Scroll sideways to inspect all the sensor columns.

Illustrative measurements: negative values mean below the reference. A sensor can also respond with the opposite sign. Adding a sensor measures every object again with that new sensor; it does not give us another known weight.

A rack of p sensors reads M = aY + N. Sensor i responds by an unknown amount a[i] and adds independent Gaussian noise of variance σ2. In our units the rack’s total response energy, the squared length ‖a‖2 = p, is known; the individual responses may have arbitrary unequal magnitudes and signs. In the random racks below, each calibration object is drawn independently from the same distribution and measured with fresh noise: the examples are independent and identically distributed (iid).

How does this produce the one-spike covariance?

The shared term a⃗Y contributes one rank-one direction; independent sensor noise contributes an isotropic floor. After standardizing and rotating, this is the Core Story’s one-spike model. Build the covariance interactively or open the exact variable dictionary. The Stylized page’s Gram-matrix drawer shows why wide random nuisance becomes identity-like.

Part 1 · The plateau

Why do more sensors stop helping least squares?

More sensors keep helping an oracle that knows their responses. Least squares settles onto a plateau. Step by step shows that more calibration examples lower that plateau while more sensors do not.

A predictor multiplies each sensor reading by a learned coefficient, then adds the results; the sum is its prediction of the object’s weight. Interpolation means fitting every calibration label exactly. With more sensors than examples, least squares can do that in many ways; minimum norm picks the exact fit with the shortest coefficient vector.

Quick overview: a new rack at each sensor count, 20 calibration examples and noise variance 40. The calibration labels stay fixed across the sweep.

Step by step: each rack grows by retaining earlier sensor columns and labels.

Least squares peaks when the number of sensors meets the number of examples, then descends to a nonzero plateau while an oracle that knows the sensor responses keeps improving.

Where does the information go?

Why the peak? Near equal numbers of sensors and calibration examples, least squares can divide by tiny singular values and amplify measurement noise. The second descent removes that instability but leaves a signal-attenuation plateau. The rack is a physical relative of the weak-feature double-descent models of Belkin, Hsu and Xu.

In Step by step, use Back and Next beside the plot.

More options and how the sweep is scored
Regime

These regime buttons also update the ridge-repair figures in Part 5.

How to read this sweep

Both learned rules use the same 1,000 Gaussian racks at every sensor count. Thick charcoal and green curves show medians; both bands contain the middle half (25th–75th percentiles), not confidence intervals. Denser counts around p = n resolve the interpolation peak. Open circles follow one additional paired draw. Each count draws fresh sensor responses and noise, then normalizes it so ‖a‖² = p; the Gaussian calibration labels stay fixed across counts within each rack. The optional dashed green reference is the exact correlator mean conditional on ‖Y‖² = n; the median and bands include ordinary random label energies. The oracle knows the responses; both learned rules do not. Data and sampling details.

Build a rule from what the labels tell us.

A scale whose readings tend to rise with the known object weights should get a positive coefficient. A scale that moves the other way should get a negative one. Multiply each reading by its calibration label, then average: the common response reinforces; independent measurement noise tends to cancel. This estimates how each sensor responds.

For one reference known to equal 1, that estimate is simply its recorded vector M₀. On a fresh object with readings M, compare two ways to normalize exactly the same sum of products:

The interpolator: fit the reference
Ŷint = M₀ᵀM / ‖M₀‖²

Divide by the recorded reference's squared length, including its noise. Replaying that reference returns exactly 1.

The correlator: calibrate the shared response
Ŷcorr = M₀ᵀM / p

Divide by the known total response energy. Fresh noise averages away; replaying the stored noise can make the training prediction much too large.

The correlator is also one full-batch gradient step from zero for squared loss, with the step size choosing this normalization. The SGD page follows what changes when examples arrive one at a time.

The same construction with several calibration examples

Put the recorded vectors in the rows of X and the physical calibration labels in Y. Without noise, X = Ya⊤, so the label-weighted average recovers the response pattern:

aest = XᵀY / ‖Y‖² = a; wcorr = XᵀY / (p‖Y‖²).

The second expression plugs that estimate into the unbiased known-direction average aᵀM/p.

The noisy estimate need not identify the true response direction to predict well. The Why the Span? page compares two sensor-response patterns that explain the same calibration data.

So the gap above the oracle is not a shortage of information. The plateau comes from the denominator choice. Avoiding the inverse of the noisy Gram matrix (Part 4) avoids the interpolation peak; using the clean response energy avoids the attenuation behind the plateau. The correlator follows the oracle’s 1/p rate, but does not interpolate.

The fourth quadrant means worse training error than predicting zero, but better fresh error than predicting zero. The stylized model’s frontier experiment shows the best train–test tradeoff.

Part 2 · Misfitting

The correlator learns while misfitting.

Its expected fresh error is below zero’s baseline, while its training error is above it.

These buttons load different experiments, with their own calibration sets and noise levels.

Expected comparison: 1 calibration example, noise variance 4. Calibration labels have mean square 1.

Add sensors to lower expected fresh error; expected training error approaches 16.

Drag a predictor marker along its path to change the sensor count; every predictor shown is scored at that count.

Training and fresh-test errors for the sensor rulesAs sensors are added, the expected correlator enters the region with training error above one and fresh-test error below one. Green dots compare its expected and realized errors; charcoal dots do the same for minimum norm.
Predicting zero scores 1 on both axes.
Plot key, uncertainty, and calculation details

Here the training labels are normalized to have mean square 1, so raw and relative training error coincide. The smooth green curve is the correlator’s exact expected path, and purple whiskers show the correlator’s separate 10–90% ranges of training and fresh-test error at the selected sensor count, crossing at their marginal medians. They are not a joint probability region. The charcoal expectation uses an exact Gaussian calculation, evaluated as a bounded series rather than a simulated average.

Why the expected error is trustworthy near interpolation

Fix the labels and sensor responses with squared lengths n and p. Rotate the labels onto one sample axis. The remaining n − 1 Gaussian rows leave a random nullspace of dimension k = p − n + 1. The minimum-norm interpolator lies along the first row projected into that space. If its standardized squared length is Q and its mean vector is μ, its fresh risk satisfies

n/Q ≤ R ≤ 2 + (2‖μ‖² + n)/Q, with ‖μ‖² ≤ np/σ².

The density of Q near zero is proportional to Qk/2−1. Thus the mean risk is infinite at p = n and n + 1. At p = n + 2 and n + 3, the mean exists but the variance is infinite. A conventional simulation error bar would be unjustified there.

Instead, the Gaussian and random-subspace integrals give this exact expression for p > n + 1:

E R = (p−2)/(p−n−1) E g(J), where J is Poisson with mean np/(2σ²) and g(0) = (p−1)/(p−2).
g(j) = [(p−1)(p−4) + 2j(n+1)] / [(p+2j−4)(p+2j−2)] for j ≥ 1.

Every scaled summand lies between zero and (p−1)/(p−n−1). We sum the Poisson weights deterministically; a Chernoff bound on omitted weight bounds the truncation error. No sampled heavy tail is used to estimate the mean. Full derivation, moment theorem, and numerical error control.

Here the correlator predicts fresh measurements well while fitting its calibration readings worse than zero.

Why can the correlator enter the fourth quadrant? The next lab returns to the one-reference experiment, using a separate draw. Let J = M₀ᵀM be the cross-product between the stored reference readings and a new sensor record, and P = ‖M₀‖² the squared length of that stored record. Both rules predict with J; the correlator divides it by the known p, the interpolator by the noisy P. So the correlator’s prediction is c = P/p times the interpolator’s, and we call c the output multiplier.

Part 3 · Fresh vs replay

The correlator’s mechanism

Fresh noise enters J as signed products N0[i]N[i], which average away across sensors. Reusing the reference turns those products into noise squares, which accumulate instead.

Connection: the same fresh-versus-reused distinction drives the SGD page.

Why do one-reference racks differ so much?Inside: Part 1’s first saved one-reference rack, its noise products and squares, and all five saved racks’ scores

This uses the first saved rack from Part 1’s one-example, noise-variance-1 setting; the two curves average products and squares of its actual sensor noises. This rack’s single target is unusually large, so its training error stays below predicting zero’s even though replay inflates its prediction; most one-reference racks at this noise score worse than predicting zero.

Open this comparison to load the saved rack.

The interactive figure has not loaded. You can still read the explanation, formulas, and expected-error comparisons below.

Part 4 · The denominator

Why does the interpolator settle onto a plateau?

Attenuation. The cross-product J is a healthy numerator: fresh products average toward pY. But the interpolator divides by P, the stored reference record’s squared length. That denominator retains about pσ2 of noise energy, so J/P shrinks the signal.

With one reference of quantity 1, J/P → Y/(1+σ2) as sensors accumulate. For fresh quantities with variance 1, this attenuation leaves a limiting fresh-test error of (σ2/(1+σ2))2: 0.64 at noise variance 4.

Noisy inputs cause attenuation, or regression dilution: a classical errors-in-variables (EIV) problem dating back to Spearman (1904). Here each sensor is an observation; its reference reading is the noisy regressor. Noise inflates its measured second moment. Classical corrections subtract that contribution (Fuller, 1987); total least squares instead determines its subtraction from the data.

One hundred sensors · reference quantity 1 · fresh physical quantity Y = 1.6 · noise variance σ2 = 4

The interactive comparison has not loaded. The key limits are J/p → Y and P/p → 1+σ2, so J/P → Y/(1+σ2).

From one denominator to Gram matrices and SGD

With n calibrations, the scalar denominator becomes a Gram matrix. Each row of X is one calibration record of all the sensor readings, and Y⃗ holds their known calibration targets. The entries of XX⊤ are cross-products between pairs of those records; its diagonal contains each record’s squared length.

In XX⊤/p ≈ Y⃗Y⃗⊤ + σ2In, stored reference noise adds the identity term. Subtracting pσ2In removes its expected contribution from each diagonal entry.

XX⊤  →  XX⊤ − pσ2In.

Known-noise EIV subtracts that expected noise directly. The correlator instead divides by the known response energy p; the test-tuned ridge keeps its coefficient vector in the training span—linear combinations of the training sensor records—while choosing a negative shift. These corrections use different information and carry different finite-sample risks.

From diagonal correction to one-pass SGD

Known-noise EIV Gram correctionA symmetric Gram matrix with its large diagonal contribution subtracted.− noise I Known-noise EIVSubtract the known expected noise energy pσ² from each diagonal entry.
Signed ridge Gram shiftA symmetric Gram matrix with one tunable shift applied to every diagonal entry.+ nλI Signed ridgeChoose one uniform shift. Positive values stabilize; negative values undo attenuation.
One-pass SGD causal Gram systemOnly the strictly lower triangle is visible, with an algorithmic identity shift on the diagonal.causal mask One-pass SGDOnly earlier examples enter each coefficient (the triangular reading of Elfving and Nikazad, 2009). The learning rate chooses the replacement diagonal; a high rate can give negative-ridge-like anti-shrinkage.

Watch the learning-rate dial cross ridge zero, then compare the two Gram systems →

Known-noise subtraction, data-dependent TLS subtraction, and span-restricted negative ridge are distinct rules. A corrected denominator can cross zero; the Context comparison exposes their poles and sign reversals. The questions here concern when misfitting is necessary, what resources it saves, and which learners reach it.

Part 5 · Ridge repair

The test-tuned ridge shift changes sign.

Near interpolation, stabilize. Far beyond it, undo attenuation.

Test-tuned ridge is a benchmark: at each sensor count, it chooses the shift multiplier with the lowest median test error across racks, using the known model. Each rack’s test error averages over fresh readings; the plotted curve takes the median over racks.

Same regime as the Quick overview in Part 1, using a separate Monte Carlo run: 20 calibration examples, noise variance 40.

At the largest sensor count: minimum norm → test-tuned ridge (signed) Fresh error: loading Training error: loading × predicting zero Tuned shift: loading × smallest nonzero Gram eigenvalue

Drag the dotted cursor in any panel to inspect the same sensor count in all three. The shift shown is chosen to minimize median fresh-test error.

A Fresh prediction

Signed ridge closes the large-sensor gapAllowing negative shifts makes the test-tuned ridge curve continue downward while minimum norm plateaus.

B Training cost

Training error across sensor countMinimum norm has exactly zero training error after interpolation. Test-tuned ridge eventually accepts training error above predicting zero’s while continuing to improve fresh prediction.

C Test-tuned ridge shift (signed)

Positive: shrink Negative: undo attenuation −1: first Gram pole
The test-tuned ridge parameter changes signThe tuned parameter is positive near the interpolation peak and negative in the wide sensor regime.

One sweep, three views. A: fresh-test error. B: the training error the tuned shift costs. C: the signed Gram shift itself; positive shifts use log spacing, negative shifts linear. Around the sign change, ridge nearly interpolates before accepting more training error again.

Kobak, Lomond and Sanchez (2020) and Tsigler and Bartlett (2023) found that the optimal ridge penalty can be negative. For p > n, a negative shift is applied inside the training span only; unrestricted negative ridge would be unbounded along the directions no training record touches.

How is the shift tuned, and why does its sign reverse?

Training curves show medians of the per-rack ratios, rather than a ratio of medians. Fresh-test curves show the median, across racks, of each rack’s expected fresh squared error.

For mean squared training loss plus λ‖u‖², ridge adds nλI to the Gram matrix. The plots report this shift as a multiple of each rack’s smallest nonzero Gram eigenvalue, with one multiplier chosen at each p to minimize median test error. The tuning uses the analytic risk along the ridge path, including all positive shifts. Negative shifts stay above −1, the first Gram pole.

Near p ≈ n, positive ridge damps an almost-singular inverse; far beyond interpolation, identity-like nuisance acts as an implicit positive penalty on signal, so a negative shift removes some of the resulting attenuation. See the singular values and Gram matrix behind this approximation.

Part 4’s full stored-noise subtraction, XXᵀ − pσ2I, can cross the first Gram pole. This ridge path stops before it; the correlator avoids that inverse entirely and can do better.

Part 6 · Labels needed

Keep the sensors. How many labels do we need?

Best in span chooses the weights with the lowest true test error. This benchmark measures what the training records can support.

So far we added sensors to a fixed calibration set. Start with a rack of 50,000 sensors, with noise variance 500, and buy more calibration objects with known weights. How soon can the training span support good prediction? How many labels does insisting on interpolation cost?

At fresh error 0.1, the stylized model’s best-in-span predictor needs 45 examples; its interpolator needs 1,088. In the physical sensor model the median curves reach that target at roughly 50 and 1,140. Near that first physical crossing, the best-in-span predictor’s median training error is still around 80× the error from predicting zero. Choose Median & bands to see these medians; the opening view shows one dataset, whose crossing counts and training error vary. Try switching models below: the sample-count advantage survives.

More sensors and quieter sensors buy different things. In the explorer’s units the rack has d = p coordinates and spike strength γ = 1+p/σ², so when p ≫ σ² the two reference thresholds are approximately (d−1)/γ² ≈ σ⁴/p examples for useful prediction and (d−1)/γ ≈ σ² for useful interpolation. Part 2 shows what adding sensors at fixed noise buys: the correlator improves while interpolation levels off. Adding sensors lowers the first threshold without removing the second. Quieter sensors lower both.

Or keep the calibration objects and buy quieter sensors, holding their count p fixed. The excess strength is γ−1=p/σ². Making γ−1 five times larger means one fifth of the noise variance, or 1/√5 of the noise amplitude. Fix 50 calibration examples and compare quieter sensors for Train = test →

Add examples. Watch prediction improve.

The sensor count is p = d. Compare stylized, iid, and physical sensor data on the same axes.

Draw 1
Move the dots along the curves
Lower error asks more of the predictor
Strength of the shared signal
Dimension outside the signal direction

From the opening rack:
Misfitting required Target reachable without misfitting; interpolation still misses Together: prediction before interpolation.

Compare prediction with and without a training-error limit as calibration objects are added.

Best in span needs—training examples
Interpolator needs—training examples
Exact fit: price—the examples best in span needs
Best-in-span test error — Interpolator test error — Best test error under the cap — Training at best-in-span —
How to read this plotInside: dragging the plot, what each data model shows, the training cap

Drag the sample-count line or a colored dot sideways to change n. Drag the dashed target line vertically to choose an error level, or the violet diamond vertically to change the training cap. The controls above do the same.Drag the sample-count line or a colored dot sideways to change n. Drag the blue ring vertically to choose the target error, or the violet diamond vertically to change the training cap. Numeric inputs and sliders do the same.

The same iid features, now predicting the physical quantity behind noisy sensors. Moving n keeps this dataset; Fresh draw replaces it. Faint curves and 80% bands summarize other draws. The dotted gray line is the irreducible test-error floor.

Require raw training MSE to equal fresh-test MSE, before dividing training error by predicting zero’s. The teal curve shows the best prediction available under that requirement. This comparison uses one dataset; selecting it returns to One draw.

Logarithmic axes · horizontal: training examples n · vertical: squared error. Predicting zero has fresh error 1.

Useful-prediction threshold (d−1)/γ²4.9Useful-interpolation threshold (d−1)/γ495

Full sample range · comparisons ready.

Model, formulas, and sampling calculation

The stylized curves are exact. In One draw, each sample count uses a longer prefix of the same Gaussian or sensor dataset. Selected scores and verified crossing counts come from solving the two quadratic error formulas; line segments interpolate solved points only for drawing. Pale Gaussian curves instead use a typical-spectrum approximation. Median curves and their bands summarize independent datasets.

Full numerical method and reproducible values

          

          
The starting settings give all three models matching covariance geometry, γ = 101, d−1 = 49,999, but different targets. Physical weights retain an irreducible error floor, 500/(50,000 + 500) ≈ 0.0099. The cartoon uses noiseless signal labels and orthogonal nuisance. Explore the matching cartoon on the Stylized page, or open the target dictionary.

          
Best in the span, for any set of observed vectors: a short derivation
            
Let S be the span of the recorded sensor vectors, and let v = PSa be the projection of the true response onto that span. Fresh physical quantities have variance 1; each sensor adds independent noise of variance σ² > 0. For any linear predictor w ∈ S,

            
R(w) = (1 − aᵀw)² + σ²‖w‖² = (1 − vᵀw)² + σ²‖w‖².

            
A component perpendicular to v adds noise without helping the signal. So set w = tv and minimize this one-variable quadratic:

            
w* = v/(σ² + ‖v‖²), R* = σ²/(σ² + ‖v‖²).

            
That is the entire optimization. It works for arbitrary recorded vectors, including dependent ones. This benchmark uses the true response for analysis; a learner is not given that response.

            
For computation, put the vectors in the rows of X, collect their known physical targets in Y, form G = XXᵀ and b = Xa. Then ‖v‖² = bᵀG†b and

            
α* = G†b/(σ² + bᵀG†b), w* = Xᵀα*, Rtrain = ‖Gα* − Y‖²/n.

            
The dagger is the pseudoinverse; it is the ordinary inverse when the rows are independent. Training error is the mean squared residual, in the same units as test error. Predicting zero’s population error is 1; its training error fluctuates with the sampled targets. To get the displayed median curves, repeat over iid calibration sets. Gaussian symmetry lets us sample the same scores from small matrices without storing all 50,000 coordinates. The widget also follows one nested dataset when you choose One draw; its formulas are in the drawer above. See the sampling details and reproducible calculation.

          

          
The physical experiment is complete. The one-reference lab above compares errors in physical-Y units. More sensors reduce fresh noise, yet interpolation can keep shrinking the signal. The correlator and tuned ridge can recover signal by accepting poorer calibration fit. More labels make the span useful before interpolation. Why focus on that span?

        


        

          
Optional bridge · Part 7

          
Twenty examples, one thousand sensors: see misfitting point by point.

          
A different target: the normalized optimal sensor score, the best linear combination of readings scaled to variance one. The earlier experiment learned the physical quantity Y; this bridge learns the score. Multiplying a score prediction by √(1−1/γ) (about 0.981 here) converts its units, but does not turn it into a predictor trained on physical labels. Irreducible uncertainty about Y remains. How do the two targets relate?

          
The calibration count falls between the useful-prediction and useful-interpolation thresholds: σ4/p = 1.6 < n = 20 < σ2 = 40. This finite example is only loosely inside the window. Fresh prediction can already be accurate while interpolation remains attenuated.

          
1,000 sensors · 20 labeled examples · independent Gaussian sensor noise, variance 40

          
The earlier correlator estimated the sensor response pattern directly. Here, test-optimal rescaling keeps the interpolator’s direction and chooses its multiplier to minimize conditional fresh risk, using the training data and supplied spike strength γ. Both it and negative ridge recover signal that interpolation left too small; the negative-ridge button tunes its shift with a plug-in risk estimate from that same γ, so it need not beat rescaling on this training set.

          
          
The interactive figure has not loaded. Its static claim is that test-optimal rescaling and negative ridge accept large training errors while sharply improving fresh prediction over minimum-norm interpolation.

          

          
          
          

          

            The model, the three rules, and numerical audit
            
The sensor count is p = 1,000 and the training count is n = 20. Each example has M⃗=a⃗Y+N⃗, with ‖a⃗‖2=p, independent Y ∼ N(0,1), and N⃗ ∼ N(0,40I). Here every sensor has the same response, a⃗=1⃗.

            
The target is the normalized optimal sensor average defined here. Test-optimal rescaling chooses the multiplier that minimizes conditional fresh risk along the minimum-norm predictor’s ray. Negative ridge instead chooses a span-restricted shift before the first Gram pole. Both use the observed training data and supplied spike strength; neither uses fresh labels.

            
To estimate the hidden physical quantity Y, rescale the selected rule’s score prediction by √(1−1/γ). Its expected squared error, including irreducible sensor uncertainty, is available when the interaction loads.

            
Exact values and reproducibility details

          

          
Why does ordinary TLS still select an interpolant when p > n? The Context continuation explains why exact fits cost it no measurement correction.

        



        

          Optional: explore other possible predictorsInside: a lab of jointly attainable training and test errors, with sensor, reference, noise and target controls, and the geometry behind each point
          
Beyond these two rules

          
Same test error. How different can training error be?

          
The correlator and interpolator take different paths. What else is possible? Measure the reference quantity 1 on
            n occasions. Now let the predictor and its training noises align in whatever way helps or hurts the fit.
            With many sensors and relatively few references, approximate independent sensor noises by mutually orthogonal vectors,
            each with length at most sigma sqrt(p).
            Fresh measurements still have independent Gaussian sensor noise of variance sigma squared.

          
With one reference, 100 sensors and noise variance 4, predictors with target fresh-test error 0.1
            can have training error anywhere from 0 to 10.1 under this noise budget. These are jointly chosen predictors and training noises, not independent draws.

          
          
Capacity leaves room for both fitting and misfitting. Fixing fresh-test error does not fix training error.
            Adding orthogonal references limits how much noise can align with any one predictor, pulling the high-training-error edge inward.
            The zero-training-error edge can stay put until interpolation itself becomes impossible at the chosen target.
            This envelope includes all linear predictors, including ones that know the true sensor responses.
            The correlator shows that a rule using only the available calibration data can actually reach the misfitting region.

          
What is the math behind this possibility region?
            
For 1 ≤ n ≤ p, fix ||a|| squared = p
              and all reference labels at 1. We jointly choose w and training noises subject to

            

            
The test constraint is an ellipsoid centered at w star = a/(p + sigma squared).
              Its minimum risk is g star = sigma squared/(p + sigma squared).
              The noise ball uses the root-mean-square length of Gaussian sensor noise as a budget.

            
For a fixed predictor, put b = signal error.
              Orthogonality gives sum of squared noise projections ≤ B squared ||w|| squared.
              The exact attainable interval t min ≤ t ≤ t max is therefore

            

            
The upper edge aligns noise to reinforce the existing signal error. The lower edge cancels as much of that error as the budget permits.
              Both edges can be attained with the same noise projection on every reference, even though the noise vectors themselves are orthogonal.

            
Each horizontal slice uses exactly test risk g. Write
              R = sigma ||w||. Then

            

            
For p ≥ 2, every bias in this interval is attainable; for one sensor only its two endpoints are.
              The smallest training error comes from the feasible bias nearest zero. The largest comes from an endpoint or a feasible stationary point
              b = plus or minus sqrt(g/(1 + p/n)).
              In particular, t ≤ (1 + p/n)g, with equality whenever one of those stationary points is feasible.
              The maximum under the whole test-risk ball lies on its boundary, so the upper edge also solves the budget-constrained maximization.

            
“First possible interpolation” marks the lowest test risk permitting training error zero over these joint choices.
              It is not the typical interpolator from the earlier sensor sweep. Writing mean M,
              the witnessing minimum-norm interpolator is w = mean M/||mean M|| squared.
              With one reference, its norm is exactly the reciprocal of that reference’s length.

            
At p = 0, the predictor has no coordinates. Only (t, g) = (1, 1) is attainable.

          

          
How can orthogonal noises all reinforce the same error?
            
Orthogonal vectors can still have projections with the same sign. Choose an orthonormal frame
              q one through q n whose vectors each make projection
              1/sqrt(n) onto the unit predictor direction. Their sum points along the predictor, while their perpendicular components cancel.

            
For a nonzero predictor, choose a residual r with
              r squared = t and the same sign as b. Set

            

            
The blue vector is this mean noise. The smaller blue circle shows why its maximum length falls as
              1/sqrt(n). Without orthogonality, all noises could be identical and their mean would not shrink.

            
This is exact geometry for the orthogonal-noise cartoon, not a probability statement about iid data.
              Near-orthogonality motivates it when n much smaller than p.
              It does not cover the opposite regime n much larger than p; there are only
              p nonzero mutually orthogonal directions.

          

        


        
For the careful reader
Dictionaries and derivations


        
What do several physical quantities look like?
For q latent physical quantities mixed through many sensors, M⃗ = AY + N⃗ and the shared covariance term A Cov(Y) A⊤ has rank at most q: q quantities create at most q spikes. The Function Space mixing board explores what unequal spikes then do to the estimator.

        
Exact errors and fourth-quadrant thresholds
Expected errors average over both noise records and Y ∼ N(0,1). Predicting zero has fresh and replay squared error 1.
Correlator fresh error: σ2(2+σ2)/p. Correlator replay error: σ4+(4σ2+2σ4)/p.
At σ2 = 4, replay error exceeds 1 for every p, while fresh error falls below 1 from p = 25. At p = 10,000 the two errors are 0.0024 and 16.0048; the interpolator’s fresh error is about 0.6401 by numerical quadrature and approaches 0.64.
Equivalently, the finite fresh-error condition is σ2 < √(p+1)−1. At the self-comparison boundary σ2 = 1, replay error is 1+6/p, slightly above 1. These are expected-loss thresholds, not promises about every draw. The numerical calculation reproduces the quoted quadrature.

        
Derive the one-reference window
        
With d = p sensor readings per input and γ = 1+p/σ2, the one-reference window 1 ≪ σ² ≪ √p corresponds to the single-spike conditions (d−1)/γ2 ≪ 1 ≪ (d−1)/γ in physical terms.

        
The fresh-comparison boundary

        
The correlator's error on a new measurement separates into three terms:

        
J/p − Y = a⃗⊤N⃗/p + Ya⃗⊤N⃗0/p + N⃗0⊤N⃗/p.

        
The terms have zero pairwise expected products, so their squared sizes add. If a⃗ were known, only the first term would remain and p ≫ σ2 would suffice. Learning the response pattern from one noisy record adds the other two; for σ2 ≫ 1, cross-noise dominates and tightens the requirement to p ≫ σ4.

        
The self-comparison boundary

        
At σ2 = 1, expected noise energy equals clean response energy and the limiting replay error σ4 equals the zero rule’s error. Replay loss diverges only when per-sensor noise also grows; the window requires p to grow faster still.

        

        
What information made one-example prediction possible?
        
We did not identify each sensor's true response a⃗[i]. We used the known total response energy: ∑a⃗[i]2 = p tells us to divide the sum of products by p, giving coefficients M⃗0[i]/p for combining new sensor readings.

        
The exact finite-width calculation additionally needs the spike strength γ = 1+p/σ2, or equivalently the per-sensor noise variance.

        

        
What does the normalized label mean?
          
In M⃗ = a⃗Y+N⃗, Y is the physical quantity, a⃗ the response vector, and N⃗ the noise vector. Assume Y ∼ N(0,1), independently of N⃗ ∼ N(0,σ2Ip), with σ2 > 0 and ‖a⃗‖2 = p. The noiseless signal label is the optimal sensor average normalized to variance one:

          
y = a⃗⊤M⃗/√[p(p+σ2)].

          
It is not the inaccessible physical quantity Y. What would be the best mean-square estimate of Y itself? If the responses a⃗ were known, the answer would be

          
E[Y | M⃗] = a⃗⊤M⃗/(p+σ2) = √(1−1/γ)y.

          
Here γ = 1+p/σ2. The label y is this optimal prediction divided by its standard deviation. Even this estimate leaves mean squared error 1/γ = σ2/(p+σ2) (notice that it decreases as equally noisy sensors are added). The normalized error in predicting y does not include this irreducible uncertainty about Y.

          
Noise parallel to a⃗ contributes to y; perpendicular noise supplies the nuisance coordinates. The rotated Gaussian test distribution is exactly the Core Story’s. Fixing the physical reference value at 1 does not, however, fix the normalized label or nuisance length as the stylized training point does.

          
The covariance construction allows arbitrary unequal response strengths and signs, provided ‖a⃗‖2 = p: Cov(M⃗) = a⃗a⃗⊤ + σ2I has eigenvalues p+σ2 along a⃗ and σ2 perpendicular to it.