Skip to content

Multi-modal Models

Note

  • Foundation Models:
    • 2024 Nature - A pathology foundation model for cancer diagnosis and prognosis prediction
    • [RNA] 2024 Nature Machine Intelligence - Multi-purpose RNA language modelling with motif-aware pretraining and type-guided fine-tuning
    • [scRNA-seq] 2024 Nature Methods - Large-scale foundation model on single-cell transcriptomics

I. LLMs for Multi-modal Data

  • ✅ 2024 Nature machine intelligence - A transformer-based weakly supervised computational pathology method for clinical-grade diagnosis and molecular marker discovery of gliomas
  • 2024 Nature Methods - scGPT: toward building a foundation model for single-cell multi-omics using generative AI
  • ✅ 2023 Nature - Foundation models for generalist medical artificial intelligence

II. Deep Learning Models for Multi-modal Data

  • 2022 Nature Reviews - Obtaining genetics insights from deep learning via explainable artificial intelligence
  • 2022 Genome Biology - A benchmark study of deep learning‑based multi‑omics data fusion methods for cancer
  • ✅ 2022 Cancer Cell - Pan-cancer integrative histology-genomic analysis via multimodal deep learning
  • 2022 Nature Biotech. - Multi-omics single-cell data integration and regulatory inference with graph-linked embedding
  • ✨ 2021 Nature Communications - MOGONET integrates multi-omics data using graph convolutional networks allowing patient classification and biomarker identification
  • ✨ 2021 Bioinformatics - PathCNN: interpretable convolutional neural networks for survival prediction and pathway analysis applied to glioblastoma
  • 2021 Nature Machine Intelligence - Integration of multiomics data with graph convolutional networks to identify new cancer genes and their associated molecular mechanisms
  • 2021 Bioinformatics - Subtype-GAN: a deep learning approach for integrative cancer subtyping of multi-omics data
  • 2018 Clinical Cancer Research - Deep Learning–Based Multi-Omics Integration Robustly Predicts Survival in Liver Cancer

III. Other Models for Multi-modal Data

  • 2024 Nature Communications - Enhancing NSCLC recurrence prediction with PET/CT habitat imaging, ctDNA, and integrative radiogenomics-blood insights
  • 2023 Nature - Transfer learning enables predictions in network biology
  • 2022 BIB - Blood-based transcriptomic signature panel identification for cancer diagnosis: benchmarking of feature extraction methods
  • ✅ 2022 Bioinformatics - MOMA: a multi-task attention learning algorithm for multi-omics data interpretation and classification
  • ✨ 2022 BIB - A computational framework to unify orthogonal information in DNA methylation and copy number aberrations in cell-free DNA for early cancer detection
  • 2019 Bioinformatics - DIABLO: an integrative approach for identifying key molecular drivers from multi-omics assays
  • 2019 Bioinformatics - Multiomics modeling of the immunome, transcriptome, microbiome, proteome and metabolome adaptations during human pregnancy

IV. More Deep Learning Models

  • Deep Learning on RNA
    • 3D Structure prediction of RNA: 2021 Science - Geometric deep learning of RNA structure
    • 2D structure (Transfer learning): 2019 Nature Commn. - SPOT-RNA: RNA secondary structure prediction using an ensemble of two-dimensional deep neural networks and transfer learning
    • RNA/DNA-Protein Binding (DeepBind): 2015 Nature Biotech. - DeepBind: Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning
    • AS of RNA: 2019 Cell - Predicting Splicing from Primary Sequence with Deep Learning
    • AS of RNA (DARTS): 2019 Nature Methods - Deep-learning augmented RNA-seq analysis of transcript splicing
    • APA of RNA: 2019 Cell - A Deep Neural Network for Predicting and Engineering Alternative Polyadenylation
  • Deep Learning on DNA
    • 2022 Nature - The evolution, evolvability and engineering of gene regulatory DNA
  • Deep Learning on Protein
    • 2022 Nature Biotech. - Using deep learning to annotate the protein universe
    • AlphaFold2 - 2021 Nature - Highly accurate protein structure prediction with AlphaFold
    • Baker et al. - 2021 Science - Accurate prediction of protein structures and interactions using a three-track neural network