28 Regression Diagnostics and Model Evaluation
Least squares never refuses. Hand it any column of numbers against any other and it returns a slope, a standard error, a \(p\)-value and an \(R^2\), with no indication anywhere in that output of whether the exercise made sense. Chapter 22 fitted a line to forty stores. Chapter 23 fitted a plane, then a model with a categorical predictor, then one with an interaction, and each time the software produced a tidy table that looked equally trustworthy.
Some of those models were fine. Some were not, and the output looked identical either way.
This chapter is about the gap between a model that fits and a model that can be believed. It covers what the residuals look like when each assumption fails, which single observations are quietly deciding the answer, what to do when the spread of the errors is not constant, and the problem that only appears once a model can be made arbitrarily complicated: a model that fits its own data beautifully is almost no evidence that it will fit anything else.
28.1 The Assumptions, Revisited
Chapter 22 listed four conditions behind the inference in a regression table. They are worth restating as things that can be checked, rather than things that are assumed.
| Assumption | What it says | What breaks if it fails |
|---|---|---|
| Linearity | The relationship really is a straight line, or a plane | The coefficients themselves, which now estimate nothing in particular |
| Independence | Each observation carries its own information | Standard errors, usually far too small |
| Constant variance | The spread of the errors is the same everywhere | Standard errors, \(p\)-values and every interval |
| Normality | The errors follow a Normal distribution | Prediction intervals, and little else at large \(n\) |
Two things are worth noticing before any technique. The failures are not equally serious: a bent relationship damages the estimate itself, while non-Normal errors at \(n = 200\) damage almost nothing. And only one of the four is visible in the raw scatterplot. Independence lives in how the data were collected, and once there are three or more predictors the scatterplot that made linearity obvious no longer exists.
What replaces it is the residual.
28.2 The Residual Plot
Plot the residuals \(e_i = y_i - \hat{y}_i\) against the fitted values \(\hat{y}_i\). This single picture works for one predictor or for twenty, because the fitted value collapses the whole right-hand side of the model into one number.
What it should look like is nothing. A formless horizontal band, centred on zero, of the same thickness from left to right. Any structure at all is the model telling you about something it failed to capture, because a well specified model leaves behind only what it could not have known.
Three patterns account for most of what goes wrong.
A curve. Residuals negative at both ends and positive in the middle, or the reverse. A straight line was fitted to something bent. The remedy is a transformation of \(x\), a quadratic term, or a different model; not a better line, since no line fits a curve.
A funnel. The band widens as the fitted values grow. The variance is not constant. The coefficients survive this, the standard errors do not.
A point on its own. One residual far from the rest, which inflates the residual standard error and so widens every interval in the model.
Why plot against \(\hat{y}\) rather than \(y\)? Because residuals are correlated with \(y\) by construction, so a plot against \(y\) always slopes upward and tells you nothing. The residuals are orthogonal to the fitted values, so any slope in that plot is real.
Example
Four datasets, sixty observations each, one predictor. Their \(R^2\) values are 0.892, 0.878, 0.899 and 0.872. By that measure they are the same dataset four times.
| Dataset | \(R^2\) | Residual plot | Verdict |
|---|---|---|---|
| First | 0.892 | Formless band | Sound |
| Second | 0.878 | Clear arch | The relationship is curved |
| Third | 0.899 | Funnel opening right | Variance is not constant |
| Fourth | 0.872 | One residual near 90, rest within 20 | One observation is not like the others |
Three of the four models are unsound, and \(R^2\) gives no hint of which. This is the same lesson Anscombe’s quartet taught in Chapter 21, now arriving one level further into the analysis: a summary statistic cannot report a shape.
28.3 Non-Constant Variance
Heteroscedasticity, the ungainly word for a funnel, is the most common of the three failures in business data, because scale is the most common driver of spread. Large stores, large accounts and large invoices vary more in absolute terms than small ones do, and revenue measured in thousands inherits that.
Be precise about the damage. The coefficients remain unbiased: the slope is still right on average, and the model still predicts about as well. What fails is \(\widehat{\text{SE}}(b_1)\), because the formula behind it assumes one common \(\sigma^2\) that applies everywhere. When the true variance is larger where the leverage is higher, that formula returns a number too small, and everything built on it, the \(t\) statistic, the \(p\)-value and the confidence interval, is too confident.
Three responses, roughly in order of preference.
Robust standard errors. Keep the coefficients, replace the standard error formula with one that does not assume constant variance. The usual choice is the HC3 sandwich estimator. This is the cheapest fix and it costs nothing when the variance was constant after all, so it is increasingly the default in applied work.
Transform the response. If the spread grows in proportion to the level, modelling \(\log y\) often produces constant variance and a coefficient that reads as a percentage change, which is frequently what was wanted anyway.
Model the variance. Weighted least squares, when the pattern is understood well enough to specify.
Example
A simulated dataset of 120 observations where the true slope is 4 and the spread of the errors grows sharply with \(x\).
| Quantity | Ordinary | Robust (HC3) |
|---|---|---|
| Slope estimate | 3.949 | 3.949 |
| Standard error | 0.342 | 0.448 |
| \(t\) statistic | 11.56 | 8.81 |
The ordinary standard error is 24 per cent too small. One dataset proves nothing, so repeat the whole experiment 1000 times and count how often each 95 per cent interval contains the true value of 4:
| Interval | Coverage over 1000 datasets |
|---|---|
| Ordinary | 87.2 per cent |
| Robust (HC3) | 94.6 per cent |
A 95 per cent interval that is right 87 per cent of the time is not a 95 per cent interval. Here the slope was significant on either calculation, so nothing changed in the conclusion, which is exactly the situation in which the habit is worth forming: the correction is cheap when it does not matter, and you cannot know in advance when it will.
28.4 Leverage
Not every observation gets an equal vote. An observation far from the centre of the predictors pulls harder on the fit, for the same reason that a longer lever moves more weight: the line must pivot around \((\bar{x}, \bar{y})\), so a point far out on \(x\) swings the far end a long way for a small change in its own position.
The measure is the leverage \(h_i\), the \(i\)th diagonal element of the hat matrix \(H = X(X'X)^{-1}X'\), so called because it puts the hat on \(y\) in \(\hat{y} = Hy\). For simple regression it has a readable form:
\[h_i = \frac{1}{n} + \frac{(x_i - \bar{x})^2}{\sum (x_j - \bar{x})^2}\]
Three facts follow. Leverage lies between \(1/n\) and 1. It sums to \(k + 1\), the number of parameters, so the average leverage is exactly \((k+1)/n\), which is \(2/n\) in simple regression. And it depends only on the predictors: leverage is fixed before \(y\) is even looked at.
The usual flag is \(h_i > 2(k+1)/n\), twice the average. It is a prompt to look, not a verdict.
High leverage is not a fault. A point far out on \(x\) that lies exactly where the rest of the data says it should is the most informative observation in the set. It is the one that pins down the slope. Leverage is opportunity, and what is done with the opportunity is a separate question.
28.5 Influence and Cook’s Distance
That separate question is influence: how much would the fitted model change if this observation were removed? Influence requires two things at once, leverage and a residual. Either alone is harmless.
Cook’s distance combines them:
\[D_i = \frac{e_i^2}{(k+1)\,s^2} \cdot \frac{h_i}{(1 - h_i)^2}\]
Read the two factors separately. The first is the squared standardised residual, how badly the model missed this point. The second rises steeply with leverage, and the \((1-h_i)^2\) in the denominator means it grows without bound as \(h_i\) approaches 1. Both factors must be non-trivial for \(D_i\) to be large.
An equivalent and more intuitive definition: \(D_i\) measures how far every fitted value moves when observation \(i\) is dropped, scaled so the numbers are comparable across models. The common thresholds are \(D_i > 0.5\) for a look and \(D_i > 1\) for concern, though with many observations any \(D_i\) far above the rest deserves attention regardless of its absolute size.
Example
The forty stores of Chapters 21 to 23 fit a slope of 4.2715 with \(R^2 = 0.3845\). Average leverage is \(2/40 = 0.05\); the highest is store 11 at \(h = 0.1514\), the store with the lowest spend in the set; the largest Cook’s distance is 0.2051. Nothing is running the show.
Now add a forty-first store in three different positions.
| The forty-first store | Slope | \(h\) | Cook’s \(D\) | \(R^2\) |
|---|---|---|---|---|
| None, the original forty | 4.271 | 0.384 | ||
| Spend 18, revenue 330: odd in \(y\), ordinary in \(x\) | 4.272 | 0.024 | 0.177 | 0.285 |
| Spend 40, revenue 308: extreme in \(x\), on the line | 4.271 | 0.389 | 0.000 | 0.499 |
| Spend 40, revenue 180: extreme in \(x\), off the line | 2.095 | 0.389 | 3.761 | 0.143 |
Read the middle two rows against each other, because they are the entire point of this section. Both are wildly unusual. The second has a large residual and no leverage, so it damages \(R^2\) badly and moves the slope by 0.001. The third has the highest leverage in the set and no residual, so Cook’s \(D\) is zero to three decimals and it improves \(R^2\) from 0.384 to 0.499 by widening the range of \(x\).
Only the last row is influential, and it is influential because it has both. One store in forty-one cuts the estimated return on marketing spend in half.
Finding an influential point is not permission to remove it. Deleting observations because they change the answer is how a result becomes a foregone conclusion, and a dataset from which every inconvenient point has been removed will produce a model that fits perfectly and predicts nothing.
The honest sequence is:
Check it. A great many influential points are data entry errors, a decimal in the wrong place, a figure in the wrong currency, an annual number in a monthly column. If it is an error, correct it or drop it and say so.
Ask what it is. An influential point is frequently a real case that does not belong in the population being modelled: the flagship store among branches, the enterprise contract among small accounts, the month containing the acquisition. If so, the fix is to narrow the stated scope of the model, not to hide the observation.
Report both. If the point is real and in scope, fit the model with and without it and report both. “The slope is 4.27, or 2.10 if the outlet at Riverside is excluded” is a more useful sentence than either number alone, and it is the only version that lets a reader judge.
28.6 Normality, and What It Is Actually For
Normality of the errors is the assumption most often tested and least often important. The check is a Q-Q plot of the residuals: quantiles of the residuals against quantiles of a Normal distribution, straight if the assumption holds, curved at the ends if the tails are heavier or lighter.
What it buys, and what it does not:
Confidence intervals for the coefficients survive its failure. \(b_1\) is a weighted sum of the \(y_i\), so the Central Limit Theorem of Chapter 13 applies to it directly. At moderate \(n\) the sampling distribution of the slope is close to Normal whatever the errors look like.
Prediction intervals do not. A prediction interval is a statement about a single future observation, and a single observation gets no help from the Central Limit Theorem. If the errors are skewed, the prediction interval is wrong in shape no matter how large the sample.
So the Q-Q plot matters most when the model’s purpose is prediction, and matters least when its purpose is estimating an effect. As with Chapter 17’s advice on Levene’s test and Chapter 20’s on Mauchly’s, do not run a formal test of Normality in order to decide which procedure to use. At small \(n\) it has no power to detect the departures that matter; at large \(n\) it rejects departures that do not.
28.7 Overfitting
Everything so far has been about whether the model describes the data in hand. This section is about a different and harder question: whether it describes anything else.
Chapter 23 established that \(R^2\) can only rise as predictors are added, and that adjusted \(R^2\) applies a mild penalty for the cost in degrees of freedom. Neither fact prepares you for how far this goes. With \(n\) observations and \(n\) parameters, least squares fits the data exactly: residuals of zero, \(R^2\) of 1, and a model that has learned nothing except the particular noise in those particular rows.
The general shape of the problem:
Training error falls monotonically with complexity. Adding a term can never make the fit to the data it was fitted on worse, because the simpler model is always available as a special case with that coefficient set to zero.
Test error falls, then rises. Up to a point, extra complexity captures real structure. Past that point it captures the noise, which the next sample will not share, so the model carries the wrong pattern forward.
The gap between the two curves is the overfitting. It is not a small effect, and no statistic computed on the training data alone can detect it reliably, because from inside the training data the overfit model genuinely does look better.
Example
Twenty points drawn from \(y = 20 + 3x - 0.22x^2\) plus Normal noise, so the true relationship is a gentle curve. Polynomials of rising degree are fitted to those twenty points and then scored on 400 fresh points from the same process.
| Degree | Training RMSE | Test RMSE | Training \(R^2\) |
|---|---|---|---|
| 1 | 3.039 | 3.97 | 0.118 |
| 2 | 2.567 | 3.54 | 0.371 |
| 3 | 2.515 | 3.68 | 0.396 |
| 4 | 2.380 | 3.74 | 0.459 |
| 6 | 2.110 | 7.04 | 0.575 |
| 8 | 1.906 | 44.84 | 0.653 |
| 12 | 1.465 | 997.58 | 0.795 |
| 14 | 1.365 | 4425.79 | 0.822 |
The training column falls the whole way and \(R^2\) climbs from 0.12 to 0.82, which by the standards of the previous three chapters looks like steady improvement. The test column reaches its minimum at degree 2, which is the degree the data was actually generated from, and then runs away to a root mean squared error of 4425 on a scale where \(y\) itself never exceeds 45.
At degree 14 this model has an \(R^2\) of 0.82 and is worthless.
28.8 Splitting the Data
The example above had the luxury of 400 fresh observations. Real analysis does not, so the fresh data has to be manufactured by withholding some.
The holdout split. Set aside a random fraction, commonly 20 or 30 per cent, before anything else happens. Fit on the rest. Score on the held-out part. The score is an honest estimate of performance on new data, and it is honest only for as long as the held-out part is untouched: every time a model is adjusted after seeing that score, some of the test set leaks into the training, and the estimate creeps upward.
The weakness is variance. With 100 observations a 25-observation test set gives a noisy estimate, and the answer depends on which 25 happened to be chosen.
\(k\)-fold cross-validation solves that. Split the data into \(k\) parts, commonly 5 or 10. Fit on \(k-1\) of them and score on the one left out. Repeat until each part has been the test set exactly once, then average the \(k\) scores. Every observation is used for training \(k-1\) times and for testing once, which removes both the waste and most of the arbitrariness.
The cost is \(k\) fits rather than one, which for regression is nothing. The usual summary is the cross-validated root mean squared error,
\[\text{RMSE}_{\text{CV}} = \sqrt{\frac{1}{n}\sum_{i=1}^{n} \left(y_i - \hat{y}_{-k(i)}\right)^2}\]
where \(\hat{y}_{-k(i)}\) is the prediction for observation \(i\) from the model fitted without the fold containing \(i\). It is directly comparable across models in a way \(R^2\) never was.
Example
The same twenty points, now with no test set at all. Five-fold cross-validation, using only those twenty:
| Degree | 5-fold RMSE |
|---|---|
| 1 | 3.606 |
| 2 | 3.361 |
| 3 | 3.627 |
| 4 | 3.618 |
| 6 | 4.557 |
| 8 | 23.679 |
Cross-validation picks degree 2, which is the truth, from twenty points and no outside information. Compare that to what the training statistics said about the same twenty points: \(R^2\) rising monotonically to 0.82, recommending degree 14.
The difference between those two answers is the entire argument for evaluating a model on data that did not choose it.
28.9 Selection on the Data That Chose It
Overfitting is usually described as a problem of model complexity. It is at least as often a problem of model search, and that version is more dangerous because the final model looks simple.
The procedure is familiar and it appears in a great deal of published work. Measure many candidate predictors. Screen them, keeping the ones significant at 0.05 on their own. Fit a model with the survivors. Report that model.
Every step is defensible and the combination is not, because the screen used the outcome. The survivors were selected precisely because they happened to correlate with \(y\) in this sample, and their \(p\)-values in the final model are then computed as though the model had been specified in advance. It was not. Chapter 19’s warning about the family-wise error rate applies here with a vengeance: screening 30 predictors at 0.05 gives roughly \(30 \times 0.05 = 1.5\) false positives per dataset, by construction.
Example
Fifty observations, 30 predictors, and a \(y\) drawn independently of every one of them. There is nothing to find.
Put all 30 in at once and the model behaves: \(R^2 = 0.855\), which sounds impressive until adjusted \(R^2\) brings it to 0.625 and the overall \(F\) test returns \(p = 0.0020\), which is a false positive but at least an honest one. Now run the screen instead. Four of the 30 pass at \(p < 0.05\), and the model of those four reads:
| Term | Estimate | Std. Error | \(t\) | \(p\) |
|---|---|---|---|---|
| (Intercept) | -0.110 | 0.113 | -0.97 | 0.336 |
| x10 | -0.460 | 0.133 | -3.45 | 0.0012 |
| x13 | 0.289 | 0.101 | 2.87 | 0.0062 |
| x15 | 0.368 | 0.123 | 2.99 | 0.0045 |
| x26 | -0.303 | 0.102 | -2.98 | 0.0046 |
\(R^2 = 0.434\), adjusted \(R^2 = 0.384\), \(F(4,45) = 8.624\), \(p = 0.00003\). Four predictors, all significant, a highly significant overall test and a parsimonious model. Every number in that table is describing noise.
Apply that model to fresh data from the same process and \(R^2\) is \(-0.416\), meaning it predicts worse than using the mean of \(y\).
Repeat the whole exercise 1000 times, so that no individual run can be blamed:
| Over 1000 datasets of pure noise | |
|---|---|
| Average number of predictors surviving the screen | 1.52 |
| Selected model significant at 0.05 | 79.1 per cent |
| Average adjusted \(R^2\) of the selected model | 0.157 |
A procedure meant to produce a false positive 5 per cent of the time produces one 79 per cent of the time. Stepwise selection, forward, backward or both, has the same defect and hides it behind more machinery.
What to do instead, in rough order of how often it is available.
Specify the model before looking. The strongest defence and the one most often possible. Subject knowledge, not data, chooses the predictors.
Select inside the cross-validation, not before it. If a screen is unavoidable, it must be redone within every training fold. Screening once on the whole dataset and then cross-validating the survivors leaks the outcome into every fold and returns the flattering answer again.
Split before you search. Do all the selection on one part of the data and report the final model’s performance on a part that was sealed until the end.
Report the search. A model arrived at after trying forty is not the same evidence as a model specified once, and readers can only discount what they are told about.
28.10 Choosing Among Models
Several criteria are in common use, and they answer different questions.
| Criterion | What it rewards | Use it for |
|---|---|---|
| \(R^2\) | Fit to the data in hand | Describing this sample, nothing else |
| Adjusted \(R^2\) | Fit, minus a mild cost per predictor | A quick comparison of nested models |
| AIC | Fit, minus 2 per parameter | Comparing non-nested models on the same data |
| BIC | Fit, minus \(\log(n)\) per parameter | The same, penalising size harder |
| Cross-validated RMSE | Prediction on data the model did not see | Anything that will be used on new data |
AIC and BIC are penalised likelihood measures: lower is better, differences of less than about 2 are not worth acting on, and both are comparable only across models fitted to the same observations, which quietly rules out comparing a model that dropped rows with missing values against one that did not.
Underneath the arithmetic sits a question the criteria cannot answer, which is what the model is for. A model built to explain needs defensible predictors, honest standard errors and a coefficient that means something; it may be worth keeping a variable with \(p = 0.30\) because leaving it out biases the rest. A model built to predict does not care what the coefficients mean, only what the held-out error is; there it may be worth keeping a predictor nobody can interpret. Reporting a model chosen on one criterion as though it had been chosen on the other is among the most common failures in applied work, and it is not a statistical error so much as a failure to say what was being attempted.
Three numbers from that output are worth carrying out of the chapter. One store in forty-one halves the estimated return on marketing spend, and it does so while sitting at the same leverage as a store that changes nothing. A 95 per cent interval built on the wrong variance assumption is right 87 per cent of the time, silently. And a screening procedure applied to data containing no signal whatsoever returns a significant model in 79 runs out of a hundred.
Recap
Chapters 23 and 24 are two halves of one argument. Chapter 23 built the model: several predictors, each coefficient meaning something only in the company of the others, with categorical variables entering as dummies and interactions letting one effect depend on another. Chapter 24 asked whether the result deserves belief. The answer turns on things no coefficient table displays. Residual plots show what the model failed to capture, and four datasets with the same \(R^2\) can be sound, curved, funnelled or dominated by a single point. Leverage and Cook’s distance separate the observations that could move the answer from the ones that do, and the store at the far right that sits on the line improves the fit while changing nothing, whereas its twin a hundred thousand lower halves the slope. Robust standard errors cost nothing and repair the interval when the spread is not constant. And overfitting turns the whole enterprise around: training error can only fall, so a model can never be judged on the data that chose it, which is why the degree that fits twenty points best is the one that predicts new points worst, and why a screen run over thirty meaningless predictors returns a significant model four times in five. Cross-validation is the discipline that resists all of this, because it scores every model on observations it has never met. Module VI has now taken the relationship between two measured quantities from a single number, through a line, to a model with many parts and a way of telling whether the parts are real. What none of it has done is show anybody the answer. Every chapter so far has ended in a table, and tables are where findings go to be ignored. The remainder of the book turns to making the analysis visible and to carrying it out end to end in R and in Python, which is where the work stops being an exercise and starts being an argument that somebody else can follow.
Summary
| Concept | Description |
|---|---|
| The Assumptions | |
| The Four Assumptions | Linearity, independence, constant variance, Normality of the errors |
| Unequal Consequences | A bent relationship damages the estimate; non-Normal errors at large n damage little |
| Why Residuals, Not the Scatter | With three or more predictors the scatterplot that showed linearity no longer exists |
| Reading the Residuals | |
| The Residual Plot | The single most useful diagnostic, and it works for any number of predictors |
| Residuals Against Fitted Values | Plotted against y it always slopes; against y-hat any slope is real |
| A Curve in the Residuals | A straight line was fitted to something bent, so there is no single slope |
| A Funnel in the Residuals | The spread grows with the fitted value, so the standard errors are wrong |
| An Isolated Residual | One point far from the band, inflating s and widening every interval |
| What R Squared Cannot See | Four datasets between 0.87 and 0.90, and only one of the models is sound |
| Non-Constant Variance | |
| Heteroscedasticity | Non-constant error variance, the commonest failure in business data |
| What Non-Constant Variance Breaks | Standard errors, p-values and intervals; not the coefficients themselves |
| Robust Standard Errors | The HC3 sandwich estimator, which costs nothing when the variance was constant |
| Coverage | An ordinary 95 per cent interval that contains the truth 87 per cent of the time |
| Transforming the Response | Modelling log y often stabilises the spread and reads as a percentage change |
| Leverage | |
| Leverage | How far an observation sits from the centre of the predictors |
| The Hat Matrix | H equals X inverse of X prime X times X prime, and h is its diagonal |
| Average Leverage | Exactly k plus one over n, so 2 over n in simple regression; flag at twice that |
| Leverage Ignores y | It is fixed by the predictors before the outcome is examined at all |
| High Leverage Is Not a Fault | A far point that lies on the trend is the most informative observation in the set |
| Influence | |
| Influence | How much the fitted model would change if this observation were removed |
| Cook's Distance | The squared standardised residual times a term rising steeply with leverage |
| Two Factors at Once | A large residual and high leverage; either one alone is harmless |
| Thresholds for Cook's D | Above 0.5 for a look, above 1 for concern, and any value far above the rest |
| Never Simply Delete | Removing points because they change the answer guarantees the answer |
| Report Both Fits | Fit with and without, and say so; one number alone hides the question |
| Normality | |
| Normality of the Errors | The least important of the four, and the most often formally tested |
| The Q-Q Plot | Residual quantiles against Normal quantiles, straight if the assumption holds |
| Where Normality Matters | Prediction intervals need it; coefficient intervals are protected by the CLT |
| Overfitting | |
| Overfitting | A model that fits its own data beautifully is weak evidence about any other data |
| Training Error Falls Monotonically | Adding a term can never worsen the fit to the data it was fitted on |
| The Test Error Turns | Test error falls while complexity captures structure, then rises as it captures noise |
| Exact Fit | With n parameters and n observations the residuals are zero and nothing was learned |
| Honest Evaluation | |
| The Holdout Split | Seal a fraction of the data before anything else happens, and do not reopen it |
| Leakage | Every adjustment made after seeing the test score moves it back toward the training score |
| k-Fold Cross-Validation | Each fold is the test set exactly once, so nothing is wasted and nothing is arbitrary |
| Cross-Validated RMSE | Comparable across models in a way that R squared never was |
| Selection and Choice | |
| Selection Is Overfitting Too | Searching over many models overfits even when each model is small |
| Screening on the Outcome | Keeping the predictors that correlate with y, then testing them as if specified in advance |
| Seventy-Nine Per Cent | How often a screen over thirty meaningless predictors returns a significant model |
| Select Inside the Folds | A screen run once on the whole dataset leaks the outcome into every fold |
| AIC and BIC | Penalised likelihood, lower is better, comparable only on identical observations |
| Explaining Against Predicting | One needs defensible coefficients, the other needs held-out error; they choose differently |