Language-Models-as-a-Service: Overview of a New Paradigm and Its Challenges

Emanuele La Malfa, Aleksandar Petrov, Simon Frieder, Christoph Weinhuber, Ryan Burnell, Raza Nazar, Anthony G. Cohn, Nigel Shadbolt, Michael J. Wooldridge

JAIR 2024 pp. 1497-1523

doi:10.1613/JAIR.1.15865 /jair/2024/malfa2024jair-languagemodelsasaservice/

Abstract

Some of the most powerful language models currently are proprietary systems, accessible only via (typically restrictive) web or software programming interfaces. This is the LanguageModels-as-a-Service (LMaaS) paradigm. In contrast with scenarios where full model access is available, as in the case of open-source models, such closed-off language models present specific challenges for evaluating, benchmarking, and testing them. This paper has two goals: on the one hand, we delineate how the aforementioned challenges act as impediments to the accessibility, reproducibility, reliability, and trustworthiness of LMaaS. We systematically examine the issues that arise from a lack of information about language models for each of these four aspects. We conduct a detailed analysis of existing solutions, put forth a number of recommendations, and highlight directions for future advancements. On the other hand, it serves as a synthesized overview of the licences and capabilities of the most popular LMaaS.

PDF JAIR Semantic Scholar

Cite

Text

La Malfa et al. "Language-Models-as-a-Service: Overview of a New Paradigm and Its Challenges." Journal of Artificial Intelligence Research, 2024. doi:10.1613/JAIR.1.15865

Markdown

[La Malfa et al. "Language-Models-as-a-Service: Overview of a New Paradigm and Its Challenges." Journal of Artificial Intelligence Research, 2024.](https://mlanthology.org/jair/2024/malfa2024jair-languagemodelsasaservice/) doi:10.1613/JAIR.1.15865

BibTeX

@article{malfa2024jair-languagemodelsasaservice,
  title     = {{Language-Models-as-a-Service: Overview of a New Paradigm and Its Challenges}},
  author    = {La Malfa, Emanuele and Petrov, Aleksandar and Frieder, Simon and Weinhuber, Christoph and Burnell, Ryan and Nazar, Raza and Cohn, Anthony G. and Shadbolt, Nigel and Wooldridge, Michael J.},
  journal   = {Journal of Artificial Intelligence Research},
  year      = {2024},
  pages     = {1497-1523},
  doi       = {10.1613/JAIR.1.15865},
  volume    = {80},
  url       = {https://mlanthology.org/jair/2024/malfa2024jair-languagemodelsasaservice/}
}