Predicting the Year of Total Knee Replacement: A Transformer-Based Multimodal Approach

Abstract

Accurate prediction of the year of total knee replacement (TKR) is challenging due to the complex interplay of factors influencing the surgical decision. Current deep learning models often rely on single-modality data, limiting their predictive power. Multimodal approaches integrating imaging and patient data offer the potential to improve predictions and support clinical decisions. This study presents an end-to-end trained, transformer- based multimodal model that integrates MR imaging with tabular data, including clinical variables and image readings, to predict the year of TKR for each subject. Our model lever- ages cross-modal attention to fuse features from an image encoder with a self-supervised pretrained tabular encoder, achieving the highest accuracy of 63.4% among tested mod- els. We evaluated its performance against three unimodal models and four multimodal fusion strategies, including simple concatenation, DAFT, and multimodal interaction. The results demonstrate that our model’s cross-modal interaction approach with pretrained TabNet not only outperformed all unimodal models but also showed improvements over other multimodal fusion techniques, highlighting the effectiveness of cross-modal attention fusion for integrating complex data modalities in TKR year prediction tasks. Source code is available at https://github.com/denizlab/2025_MIDL_time2TKR.