How Deeply Do LLMs Internalize Human Citation Practices? a Graph-Structural and Embedding-Based Evaluation
Abstract
As Large Language Models (LLMs) integrate into scientific workflows, understanding how they conceptualize the literature becomes critical. We compare LLM-generated citation suggestions with real references from top AI conferences (AAAI, NeurIPS, ICML, ICLR), analyzing key citation graph properties—centralities, clustering coefficients, and structural differences. Using OpenAI embeddings for paper titles, we quantify the alignment of LLM-generated citations with ground truth references. Our findings reveal that LLM-generated citations closely resemble human references in these distributional properties, deviating significantly from random baselines.
Cite
Text
Mobini et al. "How Deeply Do LLMs Internalize Human Citation Practices? a Graph-Structural and Embedding-Based Evaluation." ICLR 2025 Workshops: HAIC, 2025.Markdown
[Mobini et al. "How Deeply Do LLMs Internalize Human Citation Practices? a Graph-Structural and Embedding-Based Evaluation." ICLR 2025 Workshops: HAIC, 2025.](https://mlanthology.org/iclrw/2025/mobini2025iclrw-deeply/)BibTeX
@inproceedings{mobini2025iclrw-deeply,
title = {{How Deeply Do LLMs Internalize Human Citation Practices? a Graph-Structural and Embedding-Based Evaluation}},
author = {Mobini, Melika and Holst, Vincent and Tori, Floriano and Algaba, Andres and Ginis, Vincent},
booktitle = {ICLR 2025 Workshops: HAIC},
year = {2025},
url = {https://mlanthology.org/iclrw/2025/mobini2025iclrw-deeply/}
}