Position: Governments Need to Increase and Interconnect Post-Deployment Monitoring of AI

Abstract

Language-based AI systems are diffusing into society, bringing positive and negative impacts. Mitigating negative impacts depends on accurate impact assessments, drawn from an empirical evidence base that makes causal connections between AI usage and impacts. Interconnected post-deployment monitoring combines information about model integration and use, applications, and real-world incidents and impacts. For example, chain-of-thought and inference data can be combined with monitoring social media for AI generated text, or monitoring societal indicators of disinformation. Drawing on information sharing mechanisms in other industries, we highlight example data sources and specific data points that governments and their AI Safety Institutes could collect to inform AI risk management.