Thinking Out-of-the-Box: A Comparative Investigation of Human and LLMs in Creative Problem-Solving
Abstract
We explore the creative problem-solving capabilities of modern LLMs in a constrained setting. To this end, we create MacGyver, an automatically generated dataset consisting of 1,600 real-world problems deliberately designed to trigger innovative usage of objects and necessitate out-of-the-box thinking. We then present our collection to both LLMs and humans to compare and contrast their problem-solving abilities. Our task is challenging for both groups, but in unique and complementary ways. For instance, humans excel in tasks they are familiar with but struggle with domain-specific knowledge, leading to a higher variance. In contrast, LLMs, exposed to a variety of specialized knowledge, attempt broader problems but fail by proposing physically-infeasible actions.
Cite
Text
Tian et al. "Thinking Out-of-the-Box: A Comparative Investigation of Human and LLMs in Creative Problem-Solving." ICML 2024 Workshops: LLMs_and_Cognition, 2024.Markdown
[Tian et al. "Thinking Out-of-the-Box: A Comparative Investigation of Human and LLMs in Creative Problem-Solving." ICML 2024 Workshops: LLMs_and_Cognition, 2024.](https://mlanthology.org/icmlw/2024/tian2024icmlw-thinking/)BibTeX
@inproceedings{tian2024icmlw-thinking,
title = {{Thinking Out-of-the-Box: A Comparative Investigation of Human and LLMs in Creative Problem-Solving}},
author = {Tian, Yufei and Ravichander, Abhilasha and Qin, Lianhui and Le Bras, Ronan and Marjieh, Raja and Peng, Nanyun and Choi, Yejin and Griffiths, Thomas L. and Brahman, Faeze},
booktitle = {ICML 2024 Workshops: LLMs_and_Cognition},
year = {2024},
url = {https://mlanthology.org/icmlw/2024/tian2024icmlw-thinking/}
}