Ridge Regression: Structure, Cross-Validation, and Sketching
Abstract
We study the following three fundamental problems about ridge regression: (1) what is the structure of the estimator? (2) how to correctly use cross-validation to choose the regularization parameter? and (3) how to accelerate computation without losing too much accuracy? We consider the three problems in a unified large-data linear model. We give a precise representation of ridge regression as a covariance matrix-dependent linear combination of the true parameter and the noise. We study the bias of $K$-fold cross-validation for choosing the regularization parameter, and propose a simple bias-correction. We analyze the accuracy of primal and dual sketching for ridge regression, showing they are surprisingly accurate. Our results are illustrated by simulations and by analyzing empirical data.
Cite
Text
Liu and Dobriban. "Ridge Regression: Structure, Cross-Validation, and Sketching." International Conference on Learning Representations, 2020.Markdown
[Liu and Dobriban. "Ridge Regression: Structure, Cross-Validation, and Sketching." International Conference on Learning Representations, 2020.](https://mlanthology.org/iclr/2020/liu2020iclr-ridge/)BibTeX
@inproceedings{liu2020iclr-ridge,
title = {{Ridge Regression: Structure, Cross-Validation, and Sketching}},
author = {Liu, Sifan and Dobriban, Edgar},
booktitle = {International Conference on Learning Representations},
year = {2020},
url = {https://mlanthology.org/iclr/2020/liu2020iclr-ridge/}
}