Error estimation

By error estimation I mean point and interval estimation of the generalization/prediction/test error, preferably without data splitting.

Methods

  • Cross-validation (CV): k-fold and leave-one-out (LOO)
  • \(C_p\) / AIC / SURE (Stein’s unbiased risk estimate) [see Stats 300C notes]

Literature

Textbook

  • Hastie, Tibshirani, Friedman, 2008: ESL, 2nd ed., Ch 7: Model Assessment and Selection
  • Efron & Hastie, 2016: Computer-age Statistical Inference, Ch. 12: CV and \(C_p\) Estimates of Prediction Error
  • Efron & Tibshirani, 1993: Introduction to the Bootstrap, Ch. 17: CV and Other Estimates of Prediction Error
  • Braga Neto & Dougherty, 2015: Error Estimation for Pattern Recognition
  • Devroye, Gyorfi, Lugosi, 1996 (“The Hungarian book”), Ch.8: Error Estimation

Empirical studies

  • Kohavi, 1995: A study of CV and bootstrap for accuracy estimation and model selection (pdf)
    • The classic study, highly cited (6000+)
  • Martin & Hirschberg, 1996: Small sample statistics for classification error rates
    • I: Error rate measurements (pdf)
    • II: Confidence intervals and significance tests (pdf)
  • Molinaro, Simon, Pfeiffer, 2005: Prediction error estimation: a comparison of resampling methods (doi)
    • Studies error estimates after model selection in high-dimensions (e.g., microarray)
  • Kim, 2009: Estimating classification error rate (doi)
    • Unlike previous studies, compares repeated CV and repeated holdout to bootstrap

Methodology: point estimation

  • Efron & Gong, 1983: A leisurely look at the bootstrap, the jackknife, and CV (doi, pdf)
  • Efron, 1983: Estimating the error rate of a prediction rule: improvement on CV (doi, jstor )
    • Introduces the “0.632 estimator” and other bootstrap competitors to CV
    • Refined in Efron & Tibshirani, 1997: Improvements on CV: the 0.632+ bootstrap method (doi)
  • Efron, 1986: How biased is the apparent error rate of a prediction rule? (doi, jstor )
  • Efron, 2004: The estimation of prediction error: covariance penalties and CV (doi, pdf, figures )
    • Covariance penalty as Rao-Blackwellized version of cross-validation
  • Tibshirani & Tibshirani, 2009: A bias correction for the minimum error in cross-validation (doi)

Methodology: interval estimation

  • Jiang, Varma, Simon, 2008: Calculating CIs for prediction error in microarray classification under resampling (doi)
  • Laber & Murphy, 2012: Adaptive CIs for the test error in classification (doi)
  • Efron, 2014: Estimation and accuracy after model selection (doi, pdf)
    • Standard errors and CIs for predicted means using bagging (bootstrap smoothing)
    • Textbook summary in Efron & Hastie, 2016, Sec 20.2: Accuracy after model selection

Model selection

  • Arlot & Celisse, 2010: A survey of cross-validation procedures for model selection (doi)