The AI Off-Switch Problem as a Signalling Game: Bounded Rationality and Incomparability
Abstract
The off-switch problem is a critical challenge in AI control: if an AI system resists being switched off, it poses a significant risk. In this paper, we model the off-switch problem as a signalling game, where a human decision-maker communicates its preferences about some underlying decision problem to an AI agent, which then selects actions to maximise the human’s utility. We assume that the human is a bounded rational agent and explore various bounded rationality mechanisms. Using real machine learning models, we reprove prior results and demonstrate that a necessary condition for an AI system to refrain from disabling its off-switch is its uncertainty about the human’s utility. We also analyse how message costs influence optimal strategies and extend the analysis to scenarios involving incomparability.
Cite
Text
Benavoli et al. "The AI Off-Switch Problem as a Signalling Game: Bounded Rationality and Incomparability." Proceedings of the Fourteenth International Symposium on Imprecise Probabilities: Theories and Applications, 2025.Markdown
[Benavoli et al. "The AI Off-Switch Problem as a Signalling Game: Bounded Rationality and Incomparability." Proceedings of the Fourteenth International Symposium on Imprecise Probabilities: Theories and Applications, 2025.](https://mlanthology.org/isipta/2025/benavoli2025isipta-ai/)BibTeX
@inproceedings{benavoli2025isipta-ai,
title = {{The AI Off-Switch Problem as a Signalling Game: Bounded Rationality and Incomparability}},
author = {Benavoli, Alessio and Facchini, Alessandro and Zaffalon, Marco},
booktitle = {Proceedings of the Fourteenth International Symposium on Imprecise Probabilities: Theories and Applications},
year = {2025},
pages = {1-11},
volume = {290},
url = {https://mlanthology.org/isipta/2025/benavoli2025isipta-ai/}
}