Multimodal biomedical AI

JN Acosta, GJ Falcone, P Rajpurkar, EJ Topol - Nature Medicine, 2022 - nature.com
The increasing availability of biomedical data from large biobanks, electronic health records,
medical imaging, wearable and ambient biosensors, and the lower cost of genome and …

On the challenges and perspectives of foundation models for medical image analysis

S Zhang, D Metaxas - Medical image analysis, 2024 - Elsevier
This article discusses the opportunities, applications and future directions of large-scale
pretrained models, ie, foundation models, which promise to significantly improve the …

nnformer: Volumetric medical image segmentation via a 3d transformer

HY Zhou, J Guo, Y Zhang, X Han, L Yu… - … on Image Processing, 2023 - ieeexplore.ieee.org
Transformer, the model of choice for natural language processing, has drawn scant attention
from the medical imaging community. Given the ability to exploit long-term dependencies …

A transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics

HY Zhou, Y Yu, C Wang, S Zhang, Y Gao… - Nature biomedical …, 2023 - nature.com
During the diagnostic process, clinicians leverage multimodal information, such as the chief
complaint, medical images and laboratory test results. Deep-learning models for aiding …

Knowledge-enhanced visual-language pre-training on chest radiology images

X Zhang, C Wu, Y Zhang, W **e, Y Wang - Nature Communications, 2023 - nature.com
While multi-modal foundation models pre-trained on large-scale data have been successful
in natural language understanding and vision recognition, their use in medical domains is …

Improved distribution matching for dataset condensation

G Zhao, G Li, Y Qin, Y Yu - … of the IEEE/CVF Conference on …, 2023 - openaccess.thecvf.com
Dataset Condensation aims to condense a large dataset into a smaller one while
maintaining its ability to train a well-performing model, thus reducing the storage cost and …

A medical multimodal large language model for future pandemics

F Liu, T Zhu, X Wu, B Yang, C You, C Wang, L Lu… - NPJ Digital …, 2023 - nature.com
Deep neural networks have been integrated into the whole clinical decision procedure
which can improve the efficiency of diagnosis and alleviate the heavy workload of …

Clip in medical imaging: A comprehensive survey

Z Zhao, Y Liu, H Wu, M Wang, Y Li, S Wang… - arxiv preprint arxiv …, 2023 - arxiv.org
Contrastive Language-Image Pre-training (CLIP), a simple yet effective pre-training
paradigm, successfully introduces text supervision to vision models. It has shown promising …

A survey on graph neural networks and graph transformers in computer vision: A task-oriented perspective

C Chen, Y Wu, Q Dai, HY Zhou, M Xu… - … on Pattern Analysis …, 2024 - ieeexplore.ieee.org
Graph Neural Networks (GNNs) have gained momentum in graph representation learning
and boosted the state of the art in a variety of areas, such as data mining (eg, social network …

Evaluating progress in automatic chest x-ray radiology report generation

F Yu, M Endo, R Krishnan, I Pan, A Tsai, EP Reis… - Patterns, 2023 - cell.com
Artificial intelligence (AI) models for automatic generation of narrative radiology reports from
images have the potential to enhance efficiency and reduce the workload of radiologists …