This can happen in ordinary linear regression.
Predict a target with a weighted sum of features, scored by squared error. Training and test examples come from the same distribution, and every predictor gets two report cards.
Score the learned predictor on the training examples used to learn it.
Score that same predictor on new examples from the same distribution.
Shared signal. Sample-specific nuisance.
Among the d feature coordinates, one loud direction carries the shared signal : the spike. The other d−1 coordinates carry nuisance. The spike strength γ is the signal direction’s variance; each nuisance direction has variance 1. Increasing γ makes the signal louder without changing the target’s scale.
Draw each example independently from the same distribution (iid): the target is the Gaussian signal itself, with no label noise; the nuisance is independent of it. We show the signal in its own coordinate only to explain the geometry; the learner is not told which direction carries it.
Weighted sums of the training inputs form the training span—the first place we will look for a useful predictor.
Full regression setup, notation, and assumptionsInside: the three-step workflow, why one signal direction, the error ruler, span predictors
xi ∈ ℝd comes with one real target yi.
u predicts ⟨u,x⟩ on any input.
Training reuses those n points. Test draws a fresh one.
For clarity, start with one shared signal direction.
In this nonsingular linear squared-loss model, there is one best coefficient vector. The assumption here is about the inputs: they vary most along that same direction. It is a linear cousin of a noisy manifold hypothesis: one direction carries shared, meaningful variation while many others carry weaker, irrelevant variation. We call that one loud direction the spike and everything else the nuisance.
The feature coordinates have no intrinsic meaning here. Rotating them preserves inner products: minimum-norm least squares, ridge and zero-initialized SGD rotate their predictors with the training data. Rotating fresh inputs by the same amount leaves their predictions unchanged. By contrast, Lasso favors sparsity along the supplied axes—additional structure that we do not assume. We display the unknown signal direction as the first coordinate simply for clarity of exposition: the learner in practice does not know these coordinates.
Since x[1]=√s, this is the same as y=x[1]/√. The division keeps Var(y)=1: γ changes how loudly the representation carries the target, not the target's own scale.
Gold is the shared signal, the spike. Blue is the d−1 dimensional nuisance.
Because Var(y)=1, predicting zero has fresh-test error 1. On a realized training set, its error is the mean squared label.
Every example, old or fresh, follows the same sampling rule.
Every predictor here is a span predictor, i.e. a weighted sum of the training examples, one weight per example. Minimum-norm least squares, ridge, regression on principal components of these training inputs, and zero-initialized gradient descent or its stochastic version all learn here. Why the Span? asks what, if anything, leaving the span can buy.
How can noisy measurements create a loud signal and lots of nuisance directions?Inside: a lab with one object and many noisy sensors, sliders for p and σ, the exact covariance
The object does not get heavier when we add sensors.
Let Y measure how much heavier or lighter an object is than average, in units of the population’s standard deviation. Each sensor measures that same quantity with its own independent noise: Mⱼ = Y + Nⱼ. More measurements can make the shared signal louder without changing the quantity we want to predict.
With identical independent sensor noise of variance σ², dividing the readings by σ makes the noise variance 1. The full covariance Cov(M/σ) has one eigenvalue γ = 1 + p/σ² and p−1 eigenvalues equal to 1.
Nuisance arrives as snowflakes.
The blue fraction of the leading bar is 1/γ. In this Gaussian sensor model, that is also the error left in predicting the physical weight even when the sensor responses and noise variances are known.
The noiseless label in the regression examples below is the normalized signal coordinate y = x_spike/√γ, with variance 1. The physical weight also has variance 1, but is only partly predictable from the readings: Y = √(1−1/γ)y + ε, where Var(ε) = 1/γ. Follow the physical target in Noisy Sensors.
For several physical quantities, visit the multispike sensor model; for learned representations, explore the pretraining model. The Function Space page’s nuisance-spectrum experiment explores what happens when nuisance is not isotropic and has colors of its own.
Exact formula behind this sensor picture
Here Y ∼ N(0,1) and every sensor has independent noise of the same variance σ². Let b = 1/σ, so ‖b‖² = p/σ². Then
Project the normalized readings onto b/‖b‖ to obtain x_spike. Gaussian conditioning gives E[Y|M] = √(1−1/γ)y and conditional variance 1/γ. The noise contribution to that coordinate is 1 out of total variance γ; the target remains in its original units.
Yes. Good prediction can come with a terrible training score.
The best in span (green) is the weighted combination of these training inputs with the lowest fresh-test error, computed from the model rather than learned. The minimum-norm interpolator (black) is the shortest weight vector that fits every training label exactly.
The red curve scales one dataset’s interpolator. Choose another with Inspect one dataset; Dataset cloud, parameters and exact frontier offers . Each contributes its own curve and endpoints.
Fresh-test error averages squared error on new examples; predicting zero scores 1. Training error is divided by predicting zero’s error on this same dataset. Both axes are logarithmic except for a short linear segment next to zero.
Drag the green point to change the number of examples n; γ changes only when the point needs it. Drag the black point up or down to change γ.
The predictor that fits
The black endpoint is this dataset’s minimum-norm interpolator. Compare its perfect training fit with its fresh-test error.
The predictor that predicts best
The green endpoint is best in this training span. Training error above one means worse fit than predicting zero.
Why scaling outward can help
Scaling outward changes both errors. This compares predictors, not successive learning updates.
Misfitting means scoring worse than predicting zero on the training examples. It is benign when fresh prediction is good. Misfit learning accepts that bad report card: the training residuals take one for the team. Why did underfitting, good fitting, and overfitting feel like the whole story? →
Why the difference? Training and fresh inputs are drawn the same way, but only the training inputs helped build the predictor, and each left its own nuisance in it. A fresh input’s nuisance is independent of the predictor; a training input lines up with the nuisance it put there. Fitting gives that nuisance credit for doing the signal’s job, so the interpolator carries too little signal. Scaling it outward restores signal for fresh inputs while amplifying each training input’s own nuisance, so its prediction overshoots.
Scaling the interpolator by c makes each training prediction c times its label, so training error is (c−1)² times predicting zero’s. On the opening dataset, best in span scales the interpolator by 5.152, giving training error 17.2 times predicting zero’s. See the two evaluations in the Stylized model, or start with one training point.
Yes. SGD can learn while it climbs the training-loss curve.
Watch SGD’s predictor change, one update at a time
On the opening Gaussian dataset, SGD makes one pass over these 100 examples. Compare the completed passes, or explore the rate slider. To vary the spike strength γ, the number of examples n, and feature dimension d, explore the adjustable SGD experiment in Stylized.
Data model and fresh drawsInside: iid Gaussian or Stylized data, a Fresh iid draw button
Opening Gaussian dataset shared with Questions 1 and 3: 100 examples, spike strength 100, 50,000 nuisance coordinates
Current predictor · update 100 / 100
| Learning rate | Training error / predicting zero | Fresh-test error |
|---|---|---|
| — | — | |
| — | — |
Both rates use this same dataset and order. Only the learning rate differs.
Gentle on each point: no update increases the loss on the example that supplied it.
Training scores use all 100 examples, including those not yet visited. Predicting zero scores 1 on both curves.
The experiment starts after one pass. Play it to watch the two report cards separate.
Why does training error climb before any example returns in Useful first pass? Large updates overshoot each label and store that point’s nuisance. Visited points meet their own stored nuisance, as in Question 1. An iid arrival has not contributed yet: its before-update loss samples fresh-test error, the fresh-stream view of Robbins and Monro (1951). Select to compare.
What happens when the points return? →
Raw losses and the displayed ratiosInside: the raw training, fresh-test and arriving-point losses behind the plotted ratios
Values outside 0.001–10,000 are drawn at the plot’s edge; the scorecards give the actual values. Arriving-point losses are raw squared errors before each update, not ratios to predicting zero.
Update rule and what is computedInside: the update, training error over all 100 points, the exact population error
Start at u₀ = 0. At update i, predict on the arriving input, measure its residual, and move in that input’s direction:
This is a gradient step on ℓᵢ(u) = ½(yᵢ − ⟨u,xᵢ⟩)². Each purple dot is the raw arriving-point loss eᵢ², before that point’s update.
Score every iterate on the same fixed training set, including points it has not yet visited:
The rust curve displays L_train(u)/L_train(0). For the Gaussian model x = (√γ y,z), with independent unit-variance signal and nuisance, the green curve is the exact population error:
Here γ = 100, n = 100, and there are 50,000 nuisance coordinates. The Stylized button instead gives the training points mutually orthogonal nuisance vectors and labels of 1, keeping the same fresh-risk formula. Explore what happens when training points return on the SGD page.
Yes. Good learned prediction can require misfitting the training data.
Perhaps SGD and the scaled-up interpolator just fit badly. Can a training-error requirement rule out every useful predictor in the span? Why restrict to the span? First compare two minima on the same data. Then choose a target, and ask what fitting costs in examples or feature quality. Each constrained minimum searches every qualifying span predictor. Methods & checks.
What does a fitting requirement rule out?
Across datasets: median fresh-test error; bands: 10th–90th percentiles. The quoted example counts are where this median curve crosses the target, not medians of each dataset’s own crossing.
On this saved draw at target 0.1, SGD reaches the target at 65 examples, inside the 38–238 interval where doing so requires misfitting.
Train = test compares the two mean squared errors before training error is divided by predicting zero’s error. The custom cap uses these same undivided units.
Training error must be no worse than predicting zero on these examples.
Hold the data fixed
Green is best in span: it permits every span predictor. The other minimum adds the selected fitting requirement. Both use the same inputs and labels; neither is one learner’s failed search.
Compare the target with both minima
A target between these minima rules out every span predictor meeting the selected fitting requirement. Move the target to see when that conclusion changes.
Compare example budgets
Compare the budgets that reach the target, with and without the selected fitting requirement, using prefixes of the same dataset. Yellow marks where reaching the target requires misfitting.
Compare matching errors
The guide temporarily selects Train = test. Choose Exact fit to require interpolation in earlier steps and feature quality. Yellow still marks the weaker no-worse-than-zero question.
When examples cannot increase
Return to your chosen example budget and improve feature quality instead. The selected target and fitting requirement stay the same. The view labels its Gaussian reference or exact orthogonal model.
SGD captures much of the saving from permitting misfitting. In the worked example above,On the saved opening draw, counting training examples at this feature strength,In the worked example (), best in span reaches test error 0.1 with 38 examples. Fit no worse than zero takes 239—6.3× as many—and Train = test takes 531. One pass of SGD, at a rate set in advance, succeeds with 65 examples, inside the interval where success requires misfitting; its training error is about 41× predicting zero. How the count was computed.
The same tradeoff, with feature quality as the resourceInside: why feature quality has two resource scales, reading the target map
Fix n and write the nuisance-to-sample aspect ratio as rasp=(d−1)/n (asp stands for aspect). The Stylized page uses ρ=γ/rasp for signal relative to this nuisance load. In the stylized model the three errors are
For a fixed small target error, useful prediction therefore asks for feature quality of order √rasp, while interpolation and zero gap ask for feature quality of order rasp. The target map turns these same inequalities around: each lower boundary shows the least examples needed at that feature quality; dashed upper boundaries show where the target is lost again. Yellow marks where the target is reachable in the training span but no predictor fitting no worse than zero can reach it. Yellow-green continues to interpolation: fitting no worse than zero can reach the target there, but exact fitting cannot. Together the two shades show where the span reaches the target before interpolation. Only yellow requires misfitting. The selected readout prices each fitting standard separately.
A fixed finite training cap also demands feature quality proportional to rasp. For target error 0.1 and cap 1, the leading requirement is about 0.52rasp, versus 3√rasp for best in span. Choose “Fit no worse than zero” to measure that cost.
The stylized curves are exact. In the two feature-quality views, iid curves use the typical Gaussian nuisance spectrum, including its finite-sample correction; these are Gaussian approximations, shown as smooth reference curves. For this approximation, write qtyp=n/(d−n). With noiseless signal labels the errors are 1/(1+γ²qtyp) and (1+qtyp)/(1+γqtyp)². The cap and zero-gap curves solve the same two quadratics using that spectrum and the population signal and label energies. Sensor targets also have irreducible error 1/γ, which falls as sensor quality improves.
Two convex quadratics: why the frontier is the best tradeoffInside: both errors as quadratics, the frontier’s weights, why nothing beats it, applying a training cap
The entire training set is drawn iid from the Gaussian model. Its targets are the clean signal coordinates.
Put the inputs in the columns of X=[√γ sᵀ; Z]. Here s contains their signal coordinates, Z contains their nuisance coordinates, and y contains their targets. Every predictor in the training span is u=Xα: the entries of α weight the training inputs. Write
The SGD and Function Space pages put examples in rows instead: their data matrix is this Xᵀ, and their XXᵀ is the same sample Gram matrix K.
For a fresh point, write its target as Y = βs + ε, where ε is independent of its inputs, has mean zero, and has expected square v.
For noiseless signal labels, β = 1, v = 0, y = s.
For physical sensor targets, β = √(1−1/γ), v = 1/γ. The realized targets y include the uncertainty left after observing the readings.
The fresh-test error and training error of this same predictor are exactly
Both are strictly convex quadratics in α. For convenience, name their quadratic matrices
Trace the entire frontier. Minimize L + λ̃R, with Lagrange multiplier λ̃ ≥ 0. Differentiating gives the explicit sample weights
At λ̃ = 0, this is the interpolator K⁻¹y. As λ̃ increases, fresh-test error cannot rise and training error cannot fall; the weights approach the best-in-span solution α* = H⁻¹βγs. For noiseless signal labels, Sherman–Morrison makes both endpoints multiples of W⁻¹s: with q = sᵀW⁻¹s, best in span is the interpolator scaled by γ(1+γq)/(1+γ²q), which is why Question 1’s red curve reaches it. Between the endpoints the frontier can leave that ray, only slightly in the Gaussian examples shown here.
Why is it the best tradeoff? Set g = R(αλ̃) and δ = α − αλ̃. Completing the square gives, for any competing predictor with R(α) ≤ g,
Both terms are nonnegative. No other combination of these training inputs can attain the same fresh-test error with lower training error.
Apply your training cap. A cap Lmax requires training squared error L ≤ Lmax, in the same units as fresh squared error. Increase λ̃ along the frontier until the cap binds or the best-in-span solution is allowed. This improves fresh prediction while using more of the training-error allowance. A zero cap gives the interpolator; a cap that permits best in span keeps it. A one-dimensional search therefore finds the best test error under each cap, while the formulas search all α—not just the interpolator’s ray.
Return to the sample-count comparison. The stylized derivation plugs simple Gram matrices into this same equation and solves it explicitly.
Learning can succeed where familiar explanations fail.
Suppose we try to explain generalization by adding a nonnegative complexity penalty to training error. For our useful misfitting predictor, the training error alone is already worse than predicting zero’s. Adding a penalty cannot explain its success.
A norm budget does not rescue this explanation. Predicting zero has both smaller norm and lower training error, so any guarantee covering every predictor within both budgets also covers predicting zero, and so must allow fresh error of at least 1.
In Question 3’s example, such a training-error bound cannot certify fresh error below 0.1 until training error is below 0.1 too. Getting both errors that low takes 531 examples—roughly 14× the 38 that best in span needs.
These explanations miss the first useful regime. Of course, a fresh validation set can still judge the predictor.
Handcrafted nonlinear predictors can escape misfitting.
Learn three predictors on separate piles of data, then take the median of their predictions. The predictors are learned; the nonlinear combining rule is handcrafted. At a training point, the two predictors that never trained on it can outvote the one that misfits it. Explore the median escape →
The training set is not always the judge.
- Can this actually happen? Yes. A predictor can test well while fitting its own training examples worse than predicting zero.
- Can learning find it? Yes. One pass of SGD can improve fresh prediction while training error climbs.
- Must fitting cost us? Yes. With few examples, a good training fit can exclude every useful predictor in the span. At target test error 0.1, matching training and test errors costs about 14× the examples on the opening Gaussian example, or, with 100 examples fixed, about 8× stronger features according to an approximate calculation for the Gaussian model.
A way out. A median changes the predictor class and can shield the training points that its three predictors misfit. For the historical comparison: How does this differ from benign underfitting and earlier negative gaps?