ALLPCB
Multimodal framework
We visualize an interactive framework in which foundation models trained on multimodal data from various sources across the healthcare ecosystem can perform tasks spanning healthcare and biomedicine. The first column lists several data sources, including care providers, payers, institutions (universities, non-profits, and government), pharmaceutical companies, wearable devices, and medical publications/forums. The second column shows data modalities produced by these sources, including images (for example, chest x-rays), video (for example, ultrasound), molecular graphs, electronic health record (EHR) tables, clinical notes and other text, time series such as electrocardiograms, and genetic data. The third column visualizes foundation models trained on those data and applied to healthcare, and the fourth column lists downstream biomedical tasks. This process can generate new data that further improves the foundation models, creating a bidirectional relationship between models and tasks.
Representative healthcare tasks
Below are examples of important healthcare tasks that can benefit from foundation models.
Provider-facing interfaces
Foundation models can improve the efficiency and accuracy of care delivery. Clinicians spend large amounts of time editing electronic health records, and preventable medical errors (for example, readmissions and surgical errors) contribute to wasted resources. Foundation models can be adapted as efficient, accurate interfaces to EHRs (clinical notes, laboratory value histories, and imaging files), assisting clinicians by generating patient visit summaries, retrieving relevant cases and literature, and suggesting laboratory tests, diagnoses, treatments, and discharge plans. Foundation models can also be employed to support surgical robots in monitoring and achieving surgical precision.
Patient-facing interfaces
Foundation models can serve as patient-facing interfaces that provide appointment-related information, answer preventive care questions, offer medical explanations (for example, text and graphical explanations of conditions), and support robotic systems that assist in patient care. They can also act as public-facing interfaces that answer questions related to public health and epidemic prevention.
Biomedical research applications
Foundation models can accelerate biomedical research, such as drug discovery and disease understanding, and ultimately translate into improved healthcare solutions. The following are examples of biomedical tasks that can benefit from foundation models.
Drug discovery
To discover therapies that treat disease, researchers must first identify targets (for example, proteins, genes, or RNAs causally related to a disease), then search for molecules (for example, small molecules or antibodies) that bind to those targets and have therapeutic effects. Identifying suitable targets and generating candidate molecules typically requires many years of costly experiments.
The generative capabilities of foundation models can expand the search space and improve efficiency, reducing experimental burden and aiding the discovery of novel, improved therapeutics. Additionally, a single foundation model that addresses related drug discovery problems concurrently (for example, target identification, efficacy prediction, and side effect prediction) may improve solutions across these tasks. One promising area is modeling proteins with language-model techniques, with successful applications ranging from predicting viral mutations that can escape vaccine-induced immunity to predicting protein docking potential for designing therapeutic antibodies.
Personalized medicine
Personalized medicine aims to select the best treatment for an individual patient based on their medical history, genetics, imaging, and other personal measurements. For example, given a set of drugs and a patient genome, a foundation model may help predict which drug is most likely to treat the patient while causing the fewest side effects. Foundation models have unique strength in integrating multimodal patient data—from EHRs to drug and molecular measurements—to make improved predictions.
Clinical trials
Clinical trials evaluate the safety and efficacy of treatments or candidate drugs. Traditional clinical trials are often inefficient and costly: many trials fail due to lack of efficacy, safety issues, or patient-matching problems. Foundation models can help by predicting likely trial failures based on existing studies and by designing promising trial protocols (for example, eligibility criteria). They can also automatically match eligible patients by evaluating multimodal patient profiles that include EHRs, genomic sequences, and other data.
