Has multimodal learning delivered universal intelligence in healthcare? A comprehensive survey

Q Lin, Y Zhu, X Mei, L Huang, J Ma, K He, Z Peng… - Information …, 2024 - Elsevier
The rapid development of artificial intelligence has constantly reshaped the field of
intelligent healthcare and medicine. As a vital technology, multimodal learning has …

A generalist vision–language foundation model for diverse biomedical tasks

K Zhang, R Zhou, E Adhikarla, Z Yan, Y Liu, J Yu… - Nature Medicine, 2024 - nature.com
Traditional biomedical artificial intelligence (AI) models, designed for specific tasks or
modalities, often exhibit limited flexibility in real-world deployment and struggle to utilize …

A multimodal generative AI copilot for human pathology

MY Lu, B Chen, DFK Williamson, RJ Chen, M Zhao… - Nature, 2024 - nature.com
Computational pathology, has witnessed considerable progress in the development of both
task-specific predictive models and task-agnostic self-supervised vision encoders …

A visual-language foundation model for computational pathology

MY Lu, B Chen, DFK Williamson, RJ Chen, I Liang… - Nature Medicine, 2024 - nature.com
The accelerated adoption of digital pathology and advances in deep learning have enabled
the development of robust models for various pathology tasks across a diverse array of …

From text to multimodality: Exploring the evolution and impact of large language models in medical practice

Q Niu, K Chen, M Li, P Feng, Z Bi, LKQ Yan… - arxiv preprint arxiv …, 2024 - arxiv.org
Large Language Models (LLMs) have rapidly evolved from text-based systems to
multimodal platforms, significantly impacting various sectors including healthcare. This …

Biomedgpt: A unified and generalist biomedical generative pre-trained transformer for vision, language, and multimodal tasks

K Zhang, J Yu, E Adhikarla, R Zhou, Z Yan… - arxiv e …, 2023 - ui.adsabs.harvard.edu
Conventional task-and modality-specific artificial intelligence (AI) models are inflexible in
real-world deployment and maintenance for biomedicine. At the same time, the growing …

A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities

T Zhao, Y Gu, J Yang, N Usuyama, HH Lee, S Kiblawi… - Nature …, 2024 - nature.com
Biomedical image analysis is fundamental for biomedical discovery. Holistic image analysis
comprises interdependent subtasks such as segmentation, detection and recognition, which …

Rocov2: Radiology objects in context version 2, an updated multimodal image dataset

J Rückert, L Bloch, R Brüngel, A Idrissi-Yaghir… - Scientific Data, 2024 - nature.com
Automated medical image analysis systems often require large amounts of training data with
high quality labels, which are difficult and time consuming to generate. This paper …

Medtrinity-25m: A large-scale multimodal dataset with multigranular annotations for medicine

Y **e, C Zhou, L Gao, J Wu, X Li, HY Zhou… - arxiv preprint arxiv …, 2024 - arxiv.org
This paper introduces MedTrinity-25M, a comprehensive, large-scale multimodal dataset for
medicine, covering over 25 million images across 10 modalities, with multigranular …

Mmedagent: Learning to use medical tools with multi-modal agent

B Li, T Yan, Y Pan, J Luo, R Ji, J Ding, Z Xu… - arxiv preprint arxiv …, 2024 - arxiv.org
Multi-Modal Large Language Models (MLLMs), despite being successful, exhibit limited
generality and often fall short when compared to specialized models. Recently, LLM-based …