On-Sensor Binarized Fully Convolutional Neural Network for Localisation and Coarse Segmentation

Abstract

Current neural networks are compatible with high-performance GPU/CPUs. However, implementing neural networks on emerging embedded sensor for inference is challenging due to sensor’s unique hardware architecture and stringent computing resources. With this in mind, this work presents new methods to implement fully convolutional neural networks (FCNs) on Pixel Processor Array (PPA) sensors with many techniques to fully use the limited resources on sensor. Specifically, we, for the first time, design and train binarized FCN for both binary weights and activations using batchnorm, group convolution, and learnable threshold for binarization, producing networks small enough to be embedded on the focal plane of the PPA, with limited local memory resources, and using parallel elementary add/subtract, shifting, and bit operations only. We demonstrate the first implementation of an FCN on a PPA device, performing three convolution layers entirely in the pixel-level processors. We use this architecture to demonstrate inference generating heat maps for object segmentation and localisation at over 280 FPS using the SCAMP-5 PPA vision chip.