Huber-Norm Regularization for Linear Prediction Models
Abstract
In order to avoid overfitting, it is common practice to regularize linear prediction models using squared or absolute-value norms of the model parameters. In our article we consider a new method of regularization: Huber-norm regularization imposes a combination of $\ell _{1}$ and $\ell _{2}$ -norm regularization on the model parameters. We derive the dual optimization problem, prove an upper bound on the statistical risk of the model class by means of the Rademacher complexity and establish a simple type of oracle inequality on the optimality of the decision rule. Empirically, we observe that logistic regression with Huber-norm regularizer outperforms $\ell _{1}$ -norm, $\ell _{2}$ -norm, and elastic-net regularization for a wide range of benchmark data sets.
Cite
Text
Zadorozhnyi et al. "Huber-Norm Regularization for Linear Prediction Models." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2016. doi:10.1007/978-3-319-46128-1_45Markdown
[Zadorozhnyi et al. "Huber-Norm Regularization for Linear Prediction Models." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2016.](https://mlanthology.org/ecmlpkdd/2016/zadorozhnyi2016ecmlpkdd-hubernorm/) doi:10.1007/978-3-319-46128-1_45BibTeX
@inproceedings{zadorozhnyi2016ecmlpkdd-hubernorm,
title = {{Huber-Norm Regularization for Linear Prediction Models}},
author = {Zadorozhnyi, Oleksandr and Benecke, Gunthard and Mandt, Stephan and Scheffer, Tobias and Kloft, Marius},
booktitle = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases},
year = {2016},
pages = {714-730},
doi = {10.1007/978-3-319-46128-1_45},
url = {https://mlanthology.org/ecmlpkdd/2016/zadorozhnyi2016ecmlpkdd-hubernorm/}
}