Dynamic ReLU

Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, Zicheng Liu

ECCV 2020

doi:10.1007/978-3-030-58529-7_21 /eccv/2020/chen2020eccv-dynamic/

Abstract

Rectified linear units (ReLU) are commonly used in deep neural networks. So far ReLU and its generalizations (non-parametric or parametric) are static, performing identically for all input samples. In this paper, we propose dynamic ReLU (DY-ReLU), a dynamic rectifier of which parameters are generated by a hyper function over all in-put elements. The key insight is that DY-ReLU encodes the global context into the hyper function, and adapts the piecewise linear activation function accordingly. Compared to its static counterpart, DY-ReLU has negligible extra computational cost, but significantly more representation capability, especially for light-weight neural networks. By simply using DY-ReLU for MobileNetV2, the top-1 accuracy on ImageNet classification is boosted from 72.0% to 76.2% with only 5% additional FLOPs.

PDF ECCV Semantic Scholar

Cite

Text

Chen et al. "Dynamic ReLU." Proceedings of the European Conference on Computer Vision (ECCV), 2020. doi:10.1007/978-3-030-58529-7_21

Markdown

[Chen et al. "Dynamic ReLU." Proceedings of the European Conference on Computer Vision (ECCV), 2020.](https://mlanthology.org/eccv/2020/chen2020eccv-dynamic/) doi:10.1007/978-3-030-58529-7_21

BibTeX

@inproceedings{chen2020eccv-dynamic,
  title     = {{Dynamic ReLU}},
  author    = {Chen, Yinpeng and Dai, Xiyang and Liu, Mengchen and Chen, Dongdong and Yuan, Lu and Liu, Zicheng},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year      = {2020},
  doi       = {10.1007/978-3-030-58529-7_21},
  url       = {https://mlanthology.org/eccv/2020/chen2020eccv-dynamic/}
}