Principles of Animal Cognition for LLM Evaluations: A Case Study on Transitive Inference
Abstract
It has become increasingly challenging to understand and evaluate LLM capabilities as these models exhibit a broader range of behaviors. We respond to this challenge by taking inspiration from another field which has developed ideas, practices, and paradigms for probing the behavior of complex intelligent systems: animal cognition. We present five core principles from animal cognition, and explain how they provide invaluable guidance for understanding LLM abilities and behavior. We ground these principles in an empirical case study, and show how they can provide a richer contextual picture of one particular reasoning capability: transitive inference.
Cite
Text
Rane et al. "Principles of Animal Cognition for LLM Evaluations: A Case Study on Transitive Inference." NeurIPS 2024 Workshops: Behavioral_ML, 2024.Markdown
[Rane et al. "Principles of Animal Cognition for LLM Evaluations: A Case Study on Transitive Inference." NeurIPS 2024 Workshops: Behavioral_ML, 2024.](https://mlanthology.org/neuripsw/2024/rane2024neuripsw-principles/)BibTeX
@inproceedings{rane2024neuripsw-principles,
title = {{Principles of Animal Cognition for LLM Evaluations: A Case Study on Transitive Inference}},
author = {Rane, Sunayana and Kirkman, Cyrus and Royka, Amanda and Todd, Graham and Law, Ryan and Foster, Jacob Gates and Cartmill, Erica},
booktitle = {NeurIPS 2024 Workshops: Behavioral_ML},
year = {2024},
url = {https://mlanthology.org/neuripsw/2024/rane2024neuripsw-principles/}
}