TreeTop: Topology-Aware Fine-Tuning for LLM Conversation Tree Understanding

Abstract

While Large Language Models (LLMs) have dominated a wide diversity of natural language tasks, improving their capabilities on \emph{structured} inputs such as graphs remains an open challenge. We introduce $\texttt{TreeTop}$, a pre-training framework for LLMs that significantly improves their ability to understand and reason over structural relationships in multi-party, threaded discussions, such as those found on social media platforms. $\texttt{TreeTop}$ is a novel set of 17 QA-style tasks specifically designed to allow LLMs to selectively focus on both the structure of and content in discussion graphs. We find that LLMs fine-tuned with $\texttt{TreeTop}$ outperform their counterparts in every setting: zero-shot/few-shot performance on unseen pretraining tasks as well as downstream social media inference tasks (e.g.rumor detection), as well as fine-tuned performance on the downstream tasks, including their challenging "early-detection" variants. In particular, $\texttt{Gemini Pro}$ fine-tuned with $\texttt{TreeTop}$ and further fine-tuned on downstream tasks surpasses both vanilla $\texttt{Gemini Pro}$ and state-of-the-art GNN baselines. Our framework paves the way for LLMs with enhanced capabilities on heavily-structured inputs.