MatFusion: A Multi-Modal Framework Bridging LLMs and Structural Embeddings for Experimental Materials Property Prediction

Abstract

The scarcity of experimental data in materials science often necessitates property predictions based on large-scale simulations, which may suffer from accuracy and reliability limitations. Uni-modal representations derived from simulated structures inherently incorporate approximations—such as the choice of exchange-correlation functional in Density Functional Theory (DFT)—which constrain machine learning models in capturing complex experimental characterizations. In this work, we propose a novel multi-modal framework, MatFusion} that integrates embeddings from domain-specific large language models (LLMs) and structural models to enhance the prediction of experimental material properties. Our approach combines LLM-derived embeddings of material compositions with graph-based structural representations, achieving a 9.15\% reduction in mean absolute error (MAE) for experimental bandgap prediction. By leveraging both experiential knowledge from materials science literature and first-principles structural information, our framework transcends traditional representation constraints, offering a powerful paradigm for improving experimental materials property predictions.