User Interface Softbots

Abstract

Human-computer interaction (HCI) and artificial intelligence (AI) share a long history of research. Concepts such as problem spaces, goals and operators, rationality, and computational models of cognition have significantly influenced research directions in both fields. Recently the concept of agents has sparked a common interest among AI and HCI researchers. Our demonstration focuses on interface agents, those that assist the user in the rich, often complex environment of the graphical user interface (Maes 1994; Lieberman 1995). Our general interest lies in the interaction between agents and their environments. Conventional interface agents interact with other applications through an application programming interface (API) or access to source code. We have developed a novel class of agents we call interface softbots, or ibots, that control interactive applications through the graphical user interface, as human users do (Zettlemoyer & St. Amant 1999; Zettlemoyer, St. Amant, & Dulberg 1999). Our ibots are based on a programmable substrate that provides sensors and effectors for this purpose. Sensor modules take pixel-level input from the display, run the data through image processing algorithms, and build a structured representation of visible interface objects. Effector modules generate mouse and keyboard gestures to manipulate these objects. These sensors and effectors act as the eyes and hands of an artificial user, controlled by an external cognitive system. Together the sensors, effectors, and controller provide a general-purpose means of managing interactive applications, through the same medium as a real user.