LLM Routing with Benchmark Datasets
Abstract
There is a rapidly growing number of open-source Large Language Models (LLMs) and benchmark datasets to compare them. While some models dominate these benchmarks, no single model typically achieves the best accuracy in all tasks and use cases. In this work, we address the challenge of selecting the best LLM out of a collection of models for new tasks. We propose a new formulation for the problem, in which benchmark datasets are repurposed to learn a ``router'' model for this LLM selection, and we show that this problem can be reduced to a collection of binary classification tasks. We demonstrate the utility and limitations of learning model routers from various benchmark datasets. The extended version of the paper is available here: https://arxiv.org/pdf/2309.15789.pdf.
Cite
Text
Shnitzer et al. "LLM Routing with Benchmark Datasets." NeurIPS 2023 Workshops: DistShift, 2023.Markdown
[Shnitzer et al. "LLM Routing with Benchmark Datasets." NeurIPS 2023 Workshops: DistShift, 2023.](https://mlanthology.org/neuripsw/2023/shnitzer2023neuripsw-llm/)BibTeX
@inproceedings{shnitzer2023neuripsw-llm,
title = {{LLM Routing with Benchmark Datasets}},
author = {Shnitzer, Tal and Ou, Anthony and Silva, Mírian and Soule, Kate and Sun, Yuekai and Solomon, Justin and Thompson, Neil and Yurochkin, Mikhail},
booktitle = {NeurIPS 2023 Workshops: DistShift},
year = {2023},
url = {https://mlanthology.org/neuripsw/2023/shnitzer2023neuripsw-llm/}
}