Hi-ToM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models
Abstract
Theory of Mind (ToM) is the ability to understand and reason about one's own and others' mental states, which plays a critical role in the development of intelligence, language understanding, and cognitive processes. While existing work has primarily focused on first and second-order ToM, we explore higher-order ToM, which involves recursive reasoning on others' beliefs. We introduce Hi-ToM, a Higher Order Theory of Mind benchmark. Our experimental evaluation using GPT-4 reveals a decline in performance on higher-order ToM tasks, indicating the limitations of current models. This highlights the challenges of reasoning in complex ToM scenarios and emphasizes the need for further advancements in large language models' higher-order ToM capabilities.
Cite
Text
He et al. "Hi-ToM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models." ICML 2023 Workshops: ToM, 2023.Markdown
[He et al. "Hi-ToM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models." ICML 2023 Workshops: ToM, 2023.](https://mlanthology.org/icmlw/2023/he2023icmlw-hitom/)BibTeX
@inproceedings{he2023icmlw-hitom,
title = {{Hi-ToM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models}},
author = {He, Yinghui and Wu, Yufan and Chen, Yulong and Deng, Naihao},
booktitle = {ICML 2023 Workshops: ToM},
year = {2023},
url = {https://mlanthology.org/icmlw/2023/he2023icmlw-hitom/}
}