## Understanding Health System Learning and the NeuroVFM Foundation Model
Modern medical artificial intelligence faces a fundamental challenge: how to learn effectively from clinical data without compromising patient privacy or requiring massive manual annotation. A new approach, described through the development of **NeuroVFM**, demonstrates how “health system learning” can enable generalist medical foundation models directly from routine clinical data.
### What is Health System Learning?
Health system learning represents a paradigm shift from traditional medical AI development. Unlike conventional approaches that train specialized models for specific diseases (such as detecting brain tumors or diagnosing Alzheimer’s), or internet-scale pretraining that lacks access to private clinical data, health system learning leverages the vast troves of data already generated during routine clinical operations.
This approach:
– Trains models without specific diagnosis targets
– Uses unlabeled data from clinical workflows
– Learns directly from rich, real-world clinical observations
– Enables generalist models that understand the full spectrum of medicine rather than narrow specialties
The key insight is that health system learning doesn’t require learning from *all* data within a health system, but rather creates an environment where foundation models can emerge naturally from clinical practice patterns.
### The UM-NeuroImages Dataset
The foundation of this approach is the UM-NeuroImages dataset, representing two decades of clinical neuroimaging at a large academic medical center. This comprehensive resource includes:
– **566,915 neuroimaging studies** (275,981 MRI and 290,934 CT scans)
– **5.2 million imaging series** with corresponding radiology reports
– Data spanning from 2004 to 2024, providing longitudinal clinical insights
– Temporal split for training (before June 2023) and testing (June 2023-May 2024)
The dataset’s power lies in its authenticity—it represents actual clinical practice, not curated research cohorts. Researchers used automated pipelines to standardize imaging protocols, with intelligent windowing for CT scans and systematic preprocessing that reduced storage needs by over an order of magnitude while preserving diagnostic quality.
### NeuroVFM: The Imaging-First Foundation Model
**NeuroVFM** represents the first implementation of health system learning through an imaging-first approach to clinical neuroimaging. Key innovations include:
**Training Methodology:**
– **Vol-JEPA architecture**: A joint embedding predictive architecture that learns through self-supervision
– **Masked modeling**: The model predicts masked portions of 3D medical volumes from visible context
– **No labels required**: Pretraining occurs without any diagnostic annotations or radiology reports
– **Multi-modal adaptation**: Different strategies for MRI versus CT imaging
**Technical Innovation:**
The model treats entire medical volumes as sequences of 3D patches, learning anatomical relationships across space. A clever masking strategy focuses on clinically relevant regions (the head), while the architecture handles variable-length input sequences—essential for medical imaging where study sizes vary dramatically.
### Grounded Learning Without Object Detection
One of the most significant challenges in medical AI is verifying that models actually “see” the right things. NeuroVFM addresses this through:
**Attention-Based Multiple Instance Learning (AB-MIL):**
– Identifies which regions of medical images influence diagnostic decisions
– Provides interpretable attention maps showing “where” the model is looking
– Uses a novel “classify-then-aggregate” approach that resolves the fundamental ambiguity of traditional attention methods
– Enables ground-truth evaluation without requiring object-level annotations
This approach is particularly important for neuroimaging, where pathologies may be tiny relative to the full scan and traditional object detection methods are impractical at scale.
### Performance and Validation
The research team conducted comprehensive validation:
**Diagnostic Performance:**
– Evaluated across **156 diagnostic conditions** spanning neurology, psychiatry, and neuroradiology
– Demonstrated strong performance on the **University of Michigan neuroimaging test set**
– Maintained accuracy even when patients appeared in both training and test sets (reflecting real-world deployment)
**External Validation:**
– Tested on **eight public neuroimaging benchmarks**
– Evaluated diverse conditions including Alzheimer’s disease, autism spectrum disorder, Parkinson’s disease, and intracranial hemorrhage
– Showed strong out-of-distribution generalization capabilities
**Comparison to Alternatives:**
– Outperformed internet-scale pretrained models (DINOv3, CLIP)
– Competed with report-supervised models (HLIP, PRIMA) despite not using radiology reports during pretraining
– Demonstrated that the self-supervised learning objective itself was the key differentiator
### Beyond Imaging: Multimodal Applications
The researchers extended NeuroVFM into multimodal applications through:
**NeuroVFM-LLaVA Integration:**
– Connected the visual encoder to the Qwen3-14B language model
– Developed a custom Perceiver-based connector to handle the high dimensionality of medical images
– Enabled radiology report generation through instruction tuning
– Demonstrated competitive performance against proprietary frontier models (GPT-5, Claude Sonnet 4.5)
**Clinical Triage Assessment:**
– Conducted a prospective study evaluating whether generated reports could support clinical triage
– Demonstrated that AI-generated reports could accurately identify urgent cases
– Showed potential for reducing clinician workload through intelligent pre-screening
### Frequently Asked Questions
**Q: How does health system learning differ from traditional medical AI approaches?**
A: Traditional approaches either train specialized models for narrow tasks, use internet-scale data without clinical context, or require expensive manual annotation. Health system learning leverages existing clinical data streams to train generalist models that understand medicine broadly, without specific diagnostic targets or manual labeling.
**Q: Does NeuroVFM replace radiologists?**
A: No. The model is designed as a clinical decision support tool that can assist with diagnosis and triage. The prospective feasibility study demonstrated it could help prioritize urgent cases, but it remains a research tool that requires human oversight.
**Q: How does NeuroVFM handle different imaging protocols and scanner variations?**
A: By learning directly from diverse clinical data spanning multiple decades and equipment, NeuroVFM develops robust representations that generalize across different imaging protocols. The model’s validation on multiple external datasets confirms this cross-scanner generalization capability.
**Q: What about patient privacy?**
A: All data remained within HIPAA-compliant infrastructure, and model weights are shared under a license that prevents the leakage of protected health information. The approach actually enhances privacy by learning patterns without requiring access to identifiable data during inference.
**Q: Can this approach work outside of neuroimaging?**
A: The health system learning framework is potentially applicable to any clinical imaging modality. The researchers note that the approach could extend to other anatomical regions and imaging types, pending appropriate validation.
### Conclusion
NeuroVFM represents a significant step toward practical medical foundation models by demonstrating that health system learning can produce generalist medical AI from routine clinical data. The model achieves competitive performance across diverse diagnostic tasks while maintaining computational efficiency and clinical interpretability.
The approach addresses key limitations in current medical AI development—particularly the dependence on labeled data and narrow task specialization—while respecting privacy constraints inherent in clinical practice. As healthcare systems worldwide struggle with workforce shortages and increasing imaging volumes, such foundation models offer a promising path toward augmenting rather than replacing clinical expertise.
The research establishes a blueprint for future medical AI development: models that learn directly from clinical practice, understand medicine as a holistic discipline, and integrate seamlessly into existing healthcare workflows without disrupting clinical operations or compromising patient privacy.
