Can You Answer This? - Exploring Zero-Shot QA Generalization Capabilities in Large Language Models (Student Abstract)
Abstract
The buzz around Transformer-based language models (TLM) such as BERT, RoBERTa, etc. is well-founded owing to their impressive results on an array of tasks. However, when applied to areas needing specialized knowledge (closed-domain), such as medical, finance, etc. their performance takes drastic hits, sometimes more than their older recurrent/convolutional counterparts. In this paper, we explore zero-shot capabilities of large LMs for extractive QA. Our objective is to examine performance change in the face of domain drift i.e. when the target domain data is vastly different in semantic and statistical properties from the source domain and attempt to explain the subsequent behavior. To this end, we present two studies in this paper while planning further experiments later down the road. Our findings indicate flaws in the current generation of TLM limiting their performance on closed-domain tasks.
Cite
Text
Sengupta et al. "Can You Answer This? - Exploring Zero-Shot QA Generalization Capabilities in Large Language Models (Student Abstract)." AAAI Conference on Artificial Intelligence, 2023. doi:10.1609/AAAI.V37I13.27019Markdown
[Sengupta et al. "Can You Answer This? - Exploring Zero-Shot QA Generalization Capabilities in Large Language Models (Student Abstract)." AAAI Conference on Artificial Intelligence, 2023.](https://mlanthology.org/aaai/2023/sengupta2023aaai-you/) doi:10.1609/AAAI.V37I13.27019BibTeX
@inproceedings{sengupta2023aaai-you,
title = {{Can You Answer This? - Exploring Zero-Shot QA Generalization Capabilities in Large Language Models (Student Abstract)}},
author = {Sengupta, Saptarshi and Ghosh, Shreya and Nakov, Preslav and Mitra, Prasenjit},
booktitle = {AAAI Conference on Artificial Intelligence},
year = {2023},
pages = {16318-16319},
doi = {10.1609/AAAI.V37I13.27019},
url = {https://mlanthology.org/aaai/2023/sengupta2023aaai-you/}
}