How Far Can a 1-Pixel Camera Go? Solving Vision Tasks Using Photoreceptors and Computationally Designed Visual Morphology

Abstract

A de facto standard approach in solving computer vision tasks is to use a common high-resolution camera and choose its placement on an agent based on human intuition. On the other hand, extremely simple and well-designed visual sensors found throughout nature allow many organisms to exhibit diverse, complex behaviors. In this work, motivated by these examples, we raise the following questions: 1. How effective simple visual sensors are in solving vision tasks? 2. What role does their design play in their effectiveness? We explore simple sensors with resolutions as low as one-by-one pixel, representing a single photoreceptor. First, we demonstrate that just a few photoreceptors can be enough to solve different vision tasks, such as visual navigation and continuous control, reasonably well, i.e., with the performance significantly better than that of a blind agent and comparable to a high-resolution camera. Second, we show that the design of these simple visual sensors plays a crucial role in their ability to provide useful information and successfully solve these tasks. To find a well-performing design, we present a computational design optimization algorithm and evaluate its effectiveness across different tasks and domains, showing promising results. Finally, we conduct a human survey to evaluate the effectiveness of intuitive designs engineered manually by humans, showing that the computationally found design is among the best designs in most cases.