SAU: Smooth Activation Function Using Convolution with Approximate Identities
Abstract
Well-known activation functions like ReLU or Leaky ReLU are non-differentiable at the origin. Over the years, many smooth approximations of ReLU have been proposed using various smoothing techniques. We propose new smooth approximations of a non-differentiable activation function by convolving it with approximate identities. In particular, we present smooth approximations of Leaky ReLU and show that they outperform several well-known activation functions in various datasets and models. We call this function Smooth Activation Unit (SAU). Replacing ReLU by SAU, we get 5.63%, 2.95%, and 2.50% improvement with ShuffleNet V2 (2.0x), PreActResNet 50 and ResNet 50 models respectively on the CIFAR100 dataset and 2.31% improvement with ShuffleNet V2 (1.0x) model on ImageNet-1k dataset.
Cite
Text
Biswas et al. "SAU: Smooth Activation Function Using Convolution with Approximate Identities." Proceedings of the European Conference on Computer Vision (ECCV), 2022. doi:10.1007/978-3-031-19803-8_19Markdown
[Biswas et al. "SAU: Smooth Activation Function Using Convolution with Approximate Identities." Proceedings of the European Conference on Computer Vision (ECCV), 2022.](https://mlanthology.org/eccv/2022/biswas2022eccv-sau/) doi:10.1007/978-3-031-19803-8_19BibTeX
@inproceedings{biswas2022eccv-sau,
title = {{SAU: Smooth Activation Function Using Convolution with Approximate Identities}},
author = {Biswas, Koushik and Kumar, Sandeep and Banerjee, Shilpak and Pandey, Ashish Kumar},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
year = {2022},
doi = {10.1007/978-3-031-19803-8_19},
url = {https://mlanthology.org/eccv/2022/biswas2022eccv-sau/}
}