SPONGE: Competing Sparse Language Representations for Effective Knowledge Transfer

Abstract

In domains with privacy constraints, most knowledge resides in siloed datasets, hindering the development of a model with all relevant knowledge for a task. Clinical NLP is a prime example of these constraints in practice. Research in this area typically falls back to the canonical setting of sequential transfer learning, where a model pre-trained on large corpora is finetuned on a smaller annotated dataset. An avenue for knowledge transfer among diverse clinics is multi-step sequential transfer learning since models are more likely to be shared than private clinical data. This setting poses challenges of cross-linguality, domain diversity, and varying label distributions which undermine generalisation. We propose SPONGE, an efficient prototypical architecture that leverages competing sparse language representations. These encompass distributed knowledge and create the necessary level of redundancy for effective transfer learning across multiple datasets. We identify that prototypical classifiers are critically sensitive to label-recency bias which we mitigate with a novel strategy at inference time. SPONGE in combination with this strategy significantly boosts generalisation performance to unseen data. With the help of medical professionals, we show that the explainability of our models is clinically relevant. We make all source code available.