Retrieval-Augmented Multimodal Language Model with Heterogeneous Evidence Integration for Clinical Prognosis
We introduce PRISM (Population-Retrieved Inference for Structured Multimodal Prognosis), a multimodal language model that combines per-feature numerical tokenization, recency-weighted aggregation of longitudinal ECGs, and a dual-modality population memory bank.
ProofPaper ↗
Key points
- Clinical prognosis requires integrating structured measurements, physiological waveforms, longitudinal history, and population context.
- Retrieval operates in representation space: the model receives mean-pooled embeddings of similar training-cohort encounters, while no individual retrieved record enters the prompt.
- On the 1,443-task MDS-ED benchmark, PRISM achieves AUROCs of 86.2% for diagnosis, 93.1% for deterioration, 91.8% for ICU admission, and 92.8% for mortality.
- These results support structure-aware multimodal integration and supervised external adaptation.
Sources (1)
- [1]Retrieval-Augmented Multimodal Language Model with Heterogeneous Evidence Integration for Clinical PrognosisarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 12:35 PM
We introduce PRISM (Population-Retrieved Inference for Structured Multimodal Prognosis), a multimodal language model that combines per-feature numerical tokenization, recency-weighted aggregation of longitudinal ECGs, and a dual-modality population memory bank.
Clinical prognosis requires integrating structured measurements, physiological waveforms, longitudinal history, and population context.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 9, 2026Continually Post-training Small Models via Synthetic Self-distillation
- Oct 8, 2026Understanding Jev, the new model everyone is talking about
- Oct 8, 2026Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review
- Oct 8, 2026SuperNav: An Agentic Navigation System for Any Task in Any Scene
- Oct 8, 2026VibeEdit: Image Editing with Canvas Instructions
- Oct 8, 2026SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models