Multi-Modal and Multi-Task Transformer for Small Molecule Drug Discovery
Abstract
We introduce a 1B-parameter transformer model pre-trained from scratch on 2.25T tokens from a massive mixture of datasets centered around drug discovery. These datasets are heterogeneous, coming from dozens of sources and covering 15 data modalities. We demonstrate the model’s capability on various molecular assay prediction tasks, including public benchmarks and internally generated holdouts from real-world drug discovery programs. Following parameter-efficient fine-tuning, the multi-modal transformer excels at multi-task predictions compared to strong molecular property prediction baselines including XGBoost and Chemprop.
Cite
Text
Sirumalla et al. "Multi-Modal and Multi-Task Transformer for Small Molecule Drug Discovery." ICML 2024 Workshops: ML4LMS, 2024.Markdown
[Sirumalla et al. "Multi-Modal and Multi-Task Transformer for Small Molecule Drug Discovery." ICML 2024 Workshops: ML4LMS, 2024.](https://mlanthology.org/icmlw/2024/sirumalla2024icmlw-multimodal/)BibTeX
@inproceedings{sirumalla2024icmlw-multimodal,
title = {{Multi-Modal and Multi-Task Transformer for Small Molecule Drug Discovery}},
author = {Sirumalla, Sai Krishna and Jr, David Stephen Farina and Qiao, Zhuoran and Di Cesare, Daniele Alessandro and Farias, Felipe Costas and O’Connor, Michael Bernard and Bygrave, Peter John and Ding, Feizhi and Dresselhaus, Thomas and de Lacerda, Marcelo Gomes Pereira and Swails, Jason Matthew and Miles, Daniel and Welborn, Matthew and Manby, Fred and Miller, Thomas},
booktitle = {ICML 2024 Workshops: ML4LMS},
year = {2024},
url = {https://mlanthology.org/icmlw/2024/sirumalla2024icmlw-multimodal/}
}