FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
Abstract
Supervised fine-tuning (SFT) is the most common way of adapting large language models (LLMs) to a new domain. In this paper, we improve the efficiency of SFT by selecting an informative subset of training examples. Specifically, for a fixed budget of training examples, which determines the computational cost of fine-tuning, we select those that maximize information gain, as measured by the Fisher information matrix of the SFT objective. We approximate it efficiently by linearization at the last layer of the LLM. Our approach is computationally efficient, analyzable, and performs well empirically. We demonstrate this on several problems, with both quantitative results and LLM-as-a-judge evaluations.
Cite
Text
Deb et al. "FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain." Proceedings of the 42nd International Conference on Machine Learning, 2025.Markdown
[Deb et al. "FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain." Proceedings of the 42nd International Conference on Machine Learning, 2025.](https://mlanthology.org/icml/2025/deb2025icml-fishersft/)BibTeX
@inproceedings{deb2025icml-fishersft,
title = {{FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain}},
author = {Deb, Rohan and Thekumparampil, Kiran Koshy and Kalantari, Kousha and Hiranandani, Gaurush and Sabach, Shoham and Kveton, Branislav},
booktitle = {Proceedings of the 42nd International Conference on Machine Learning},
year = {2025},
pages = {12924-12943},
volume = {267},
url = {https://mlanthology.org/icml/2025/deb2025icml-fishersft/}
}