# A Modular Generative Framework for Longitudinal Multimodal Clinical Data: Design, Validation, and Applications
## Introduction
Modern healthcare generates vast quantities of multimodal data—laboratory results, imaging, clinical notes, and molecular profiles—that accumulate over time for each patient. Harnessing this heterogeneity to predict disease, impute missing measurements, and model physiological trajectories remains a central challenge in biomedical informatics. Existing approaches typically handle one modality at a time or rely on simplistic late-fusion strategies that fail to capture the deep interdependencies between different clinical signals across visits.
To address these limitations, a new variational generative framework has been developed that models each patient as an evolving latent physiological process. The framework integrates heterogeneous data types—including tabular laboratory results, imaging, text-based clinical narratives, and high-dimensional molecular profiles—into a unified latent representation that evolves over time. By explicitly modeling temporal dependencies and cross-modal interactions, this approach enables more accurate predictions, robust imputation of missing data, and biologically interpretable representations of patient trajectories.
This article provides a comprehensive overview of the framework’s architecture, its application across diverse clinical cohorts, benchmarks against established methods, and its potential implications for precision medicine.
—
## Datasets and Clinical Cohorts
The framework was developed and validated using several large-scale, longitudinal clinical datasets spanning multiple countries and clinical contexts.
### UK Biobank Cohort
The UK Biobank provided a deeply phenotyped resource comprising approximately 500,000 participants aged 40–69 years recruited between 2006 and 2010. For the modeling tasks described here, data from roughly 10,000 participants were utilized, including multiomics profiles from blood samples (laboratory tests, proteomics, and metabolomics), medical histories, retinal fundus images, and longitudinal diagnostic records tracking disease outcomes over time.
### Systemic Diseases Prediction Cohort
A large longitudinal adult cohort was assembled from two hospitals in Wenzhou, China, contributing over 127,000 participants and more than 650,000 clinical visits. Each visit included either retinal fundus photography, laboratory test panels, or both, with records linked to International Classification of Diseases (ICD)-coded diagnostic follow-up data. This cohort was used to evaluate the framework’s ability to predict systemic diseases—including ischemic stroke, myocardial infarction, heart failure, and dementia—over five-year horizons.
### Myopia Prediction Cohort
A pediatric cohort of over 24,000 children contributed approximately 50,000 visits from the same Wenzhou institutions. Longitudinal records tracked the development and progression of myopia using axial length (AL) and spherical equivalent (SE) measurements, with laboratory tests and fundus images available at various visits. This cohort tested the framework’s capacity to forecast refractive trajectories over one- to five-year periods.
### Wenzhou ICU Cohort for Readmission Prediction
A critically ill adult cohort of over 13,000 ICU survivors from a single large hospital in Wenzhou was used to assess 30-day post-discharge readmission risk. Each ICU stay included structured laboratory tests and unstructured clinical notes recorded during the first 48 hours, with follow-up data linked to readmission outcomes. The study was registered on ClinicalTrials.gov (NCT06791486) and conducted in compliance with Chinese health regulations and the Declaration of Helsinki.
### External Validation Cohort
To evaluate generalizability, the framework was also tested on the MIMIC-III database, a widely used US-based ICU repository containing records from over 44,000 adult patients, alongside the Wenzhou ICU cohort for a multicenter comparison.
—
## Framework Architecture
At its core, the framework is built on a variational generative model that represents each patient visit as a point in a shared latent physiological space. The architecture consists of several interconnected components designed to handle missing modalities, temporal evolution, and cross-modal alignment.
### Modality-Specific Encoders and Historical State Integration
For each clinical modality—such as laboratory tests, proteomics, metabolomics, medical history, or fundus images—a dedicated variational encoder maps the observed data into a shared latent space. These encoders project heterogeneous inputs into a common 16-dimensional isotropic Gaussian distribution. Medical history records are segmented into five-year windows, embedded using word-level representations, aggregated within each window, and processed by a recurrent network to produce a temporal summary. Additionally, longitudinal diagnosis records can be converted into natural-language descriptions and encoded using large language models as an alternative patient-context representation.
A separate Past-State encoder summarizes all historical visits into a compact physiological context that constrains biologically plausible transitions between visits. This historical state acts as individualized physiological conditioning rather than a rigid template, enabling the model to preserve meaningful visit-specific variation while maintaining temporal coherence.
### Joint Latent Space and Distributional Alignment
To integrate current-visit modality representations with the historical context, the framework employs a distributional alignment strategy. Rather than forcing exact geometric identity across latent branches, it uses pairwise Jeffrey divergence—a symmetrized form of Kullback-Leibler divergence—to encourage biological semantic consistency across all available modality branches and the Past-State. This alignment penalty ensures that different clinical signals converge on a coherent physiological representation without suppressing visit-specific features.
The unified Visit-State is then computed by equal-weight fusion of the posterior means from all currently available modalities and the Past-State branch. Importantly, the historical state is incorporated exactly once, preventing double-counting, and if no current modality is available, the Visit-State defaults entirely to the Past-State representation.
### Temporal Propagation
After estimating the Visit-State at each time point, the framework propagates it forward to construct the historical context for subsequent visits. Two temporal propagation mechanisms are available: a gated recurrent transition that updates the historical state using the current Visit-State, and a transformer-like causal attention module that computes the historical state from the entire causally masked sequence of previous Visit-States. Both approaches serve the same purpose—summarizing prior patient-specific latent states for consumption at the next visit.
### Modality-Specific Decoders and Reconstruction
The Visit-State is passed through modality-specific decoders to reconstruct all target modalities. This design allows the framework to place visits with complete and incomplete measurements in a common latent space and to impute missing modalities from the fused physiological representation.
### Training Objectives and Strategies
The model is trained by minimizing a composite variational objective comprising four terms: a reconstruction loss, a KL regularization term that drives latent distributions toward a standard normal prior, an alignment penalty for cross-modal distributional consistency, and an adversarial regularizer.
An active masking strategy is employed during training to prevent the model from relying on trivial within-visit shortcuts. When multiple modalities are observed at a visit, one modality is randomly selected and masked with a specified probability, removing it from the encoder input while retaining its ground-truth measurement as a reconstruction target. This forces the model to leverage remaining modalities together with the historical state.
A modality-wise adversarial regularizer further ensures that the learned reconstructions reflect genuine physiological structure rather than artifacts induced by missing-data patterns. A gradient reversal layer is inserted before each adversary so that while the adversary is trained to detect missingness, the reconstruction pathway is simultaneously encouraged to remove missingness-specific cues from generated outputs.
Optimization uses the AdamW algorithm with a learning rate of 1 × 10⁻⁴, weight decay of 1 × 10⁻³, and a minibatch size of 64. A threefold cross-validation strategy ensures robust performance assessment, training proceeds for up to 500 epochs with early stopping triggered after ten epochs of validation loss stagnation, and KL annealing down-weights the prior regularization during the first 50 epochs to mitigate posterior collapse.
—
## Adaptation of Single-Cell Frameworks for Clinical Data
To contextualize the framework’s performance, three representative single-cell multimodal approaches were adapted to handle continuous clinical data for cross-modality imputation. The clinical dataset comprising laboratory tests and metabolomics biomarkers from 10,000 UK Biobank participants was log-transformed and Z-score normalized, then split at the patient level into training (70%) and test (30%) sets. Each visit was treated as an independent paired sample, converting longitudinal data into a static dataset of 14,000 paired observations.
A product-of-experts architecture (MIDAS) was adapted by replacing its negative binomial likelihood with a Gaussian likelihood and using modality-specific encoders to map laboratory and metabolomic features into a shared latent space. A variational autoencoder with a missing mask mechanism (scVAEIT) was adapted to concatenate both modality types and learn the conditional distribution of metabolomics given laboratory tests through randomized feature masking. A nongenerative topology-based approach (StabMap) established a reference coordinate system using paired training data and projected test samples via shared laboratory features to retrieve corresponding metabolomics values.
Unlike the longitudinal framework, these static baselines relied solely on current-visit laboratory data at test time and had no access to temporal or historical state variables.
—
## Benchmarking Against Longitudinal Clinical Modeling Paradigms
The framework was benchmarked against three representative longitudinal approaches on a plasma proteomics generation task using the UK Biobank-PPP cohort of 50,000 participants. Each sample comprised all medical history before the current visit, current-visit routine laboratory measurements, and the corresponding plasma proteomics profile. All methods were trained and evaluated under identical patient-level data splits against the same prediction target.
A text-embedding baseline converted all precurrent-visit diagnosis records into chronological natural-language descriptions, appended current laboratory values, and encoded the summary using a frozen large language model before passing the embeddings through a regression head. This tested whether general-purpose language representations of longitudinal clinical context alone could suffice for proteomics generation.
A standard Transformer baseline represented longitudinal diagnosis history and current laboratory data as a structured token sequence with age-aware event tokens and discretized laboratory tokens, processed by a Transformer encoder whose classification token representation was passed to a regression head. This approach directly used all available structured inputs without explicit cross-modal alignment.
A Delphi-style diagnosis-trajectory baseline used longitudinal disease-event history to derive a patient trajectory representation for proteomics prediction. Since this class of model is designed primarily for event sequences rather than multimodal completion, it could not incorporate current routine laboratory measurements and served as a diagnosis-history-only longitudinal baseline.
The key distinction is that the framework explicitly models current laboratory data and historical context as complementary branches within a shared latent physiological space, whereas comparator methods relied on event-sequence modeling, text embedding, or direct token-sequence encoding without explicit cross-modal alignment.
—
## Biological Aging Trajectories
The latent representations generated by the framework were used to computationally infer aging progression trajectories. A k-nearest neighbor graph was constructed from the unified latent space, and diffusion pseudotime was computed to derive transition probability vector fields. Streamlines representing long-range directional flows were obtained via numerical integration and projected onto a two-dimensional visualization space for inspection.
To isolate trajectory effects from chronological age, participants were binned into 0.2-year age intervals, ranked by their spatial coordinates within each bin, and partitioned into three equal-sized tertiles. Assignments were pooled across all age bins to form three trajectory groups spanning the full age range. Clinical validation employed sex-stratified Kaplan-Meier survival analysis, assessing cumulative incidence of major age-related diseases across trajectory groups.
—
## Multicenter ICU Readmission Prediction
The framework’s clinical robustness was evaluated on the task of predicting 30-day hospital readmission following ICU discharge, using both the Wenzhou ICU cohort and the MIMIC-III database. The prediction start point was defined as the patient’s first ICU entry timestamp, and the initial 48 hours were divided into 12 nonoverlapping four-hour bins.
Within each time bin, structured measurements—including vital signs and laboratory variables—were summarized using six statistics (mean, median, maximum, minimum, count, and variance) to create a fixed-length longitudinal representation. Time-stamped clinical notes within each bin were concatenated and encoded using a frozen large language model to produce step-level embeddings. Positive readmission labels were assigned when a subsequent hospital admission occurred within 30 days of discharge, with in-hospital deaths, newborn stays, and admissions shorter than 24 hours excluded.
The framework was compared against three baselines: a structured-only Transformer operating on summary statistics across time steps, a text-only model encoding all clinical notes through the same language model and passing them to a multilayer perceptron, and a late-fusion approach that encoded structured and text modalities separately and combined them only at the final prediction stage. Unlike the late-fusion baseline, the framework preserved multimodal alignment at each four-hour time step while conditioning on a Past-State propagated from previous steps, enabling it to capture both within-step multimodal correspondence and across-step physiological progression.
Both the Wenzhou and MIMIC-III cohorts were split at the patient level into training, validation, and test sets in a 7:1:2 ratio, and binary classifiers were optimized using class-weighted loss to account for the low incidence of readmission.
—
## Plasma Proteomic Disease Prediction
For each targeted disease—ischemic stroke, myocardial infarction, heart failure, and dementia—a prediction model was developed using gradient-boosted trees. The task was to classify whether a participant healthy at baseline would develop the specified disease or remain healthy within a 10-year follow-up period, excluding individuals with prevalent diagnoses at baseline.
Models were trained on generated plasma proteomic features combined with age and sex, then evaluated on held-out test sets by comparing predictions from generated proteomic features against those from true proteomic features. A separate model was also trained on the combined inputs of routine laboratory tests, an encoded medical history context vector, age, and sex to directly assess the predictive power of non-proteomic inputs.
For systemic disease prediction, after cross-modal alignment pretraining, latent representations from visits containing fundus images were fine-tuned for binary disease classification. Both single-visit and multivisit prediction modes were evaluated, with multivisit mode aggregating longitudinal trajectories across all available fundus images before classification. Myopia progression prediction similarly used fine-tuned laboratory test encoders to forecast axial length and spherical equivalent trajectories over one- to five-year horizons, with performance assessed using coefficient of determination, mean absolute error, and Pearson correlation.
—
## Statistical Rigor and Reproducibility
All analyses adhered to rigorous statistical standards. Correlation analyses employed Pearson’s coefficient for linear relationships and Spearman’s rank correlation for monotonic relationships. The significance of group differences was assessed using two-sided t-tests or Wilcoxon signed-rank tests as appropriate for data distributions. Bootstrap resampling with 200 iterations was used to estimate 95% confidence intervals for model performance metrics and regression coefficients. Exact P values were reported where P ≥ 0.0001; smaller values were denoted as P < 0.0001.All source code for model implementation, training, and evaluation is publicly available. The UK Biobank data are accessible through the UK Biobank access procedure, MIMIC-III data are available via PhysioNet, and access requests for the restricted Wenzhou cohorts are processed through data access committees. All experiments were conducted using fixed software environments with documented package versions, and model checkpoints corresponding to optimal validation performance were retained.---## Computational EfficiencyThe framework is designed for practical deployment. Training for the metabolomics imputation task on the UK Biobank can be completed in under one hour on a single modern GPU with 24 GB of memory. The proteomics imputation task requires less than three hours on the same hardware. Critically, inference does not require a GPU and takes less than one second per patient on a standard laptop, making the framework suitable for real-time clinical applications.---## FAQ**What types of clinical data can the framework handle?** The framework supports a wide range of modalities, including structured laboratory test results, high-dimensional molecular profiles (proteomics and metabolomics), retinal fundus images, free-text clinical notes, and coded medical histories. Its modular encoder design allows new modalities to be incorporated by adding a modality-specific encoder and decoder pair.**How does the framework deal with missing data?** Missing modalities are handled naturally by the latent fusion mechanism. The Visit-State is computed using only the modalities currently available at a given visit, with the Past-State providing physiological context from prior encounters. The active masking strategy during training further ensures robust imputation by forcing the model to reconstruct masked modalities using remaining inputs and historical information.**What distinguishes this approach from simple late fusion methods?** Late fusion combines modality-specific representations only at the final prediction stage, without enforcing alignment during intermediate processing. The framework, by contrast, aligns all modalities within a shared latent space at every time step and conditions this alignment on a propagated physiological state, capturing both cross-modal relationships within a visit and their temporal evolution across visits.**Can the framework be used for tasks other than disease prediction?** Yes. The framework is general-purpose and has been demonstrated for cross-modal imputation, biological aging trajectory inference, ICU readmission risk prediction, and proteomic generation. Its modular design makes it adaptable to any clinical task involving longitudinal multimodal data.**How computationally expensive is training?** For most tasks, training completes within three hours on a single modern GPU, and inference is fast enough for real-time use on standard hardware. The framework uses efficient variational inference and early stopping to balance model capacity with computational cost.**Was the framework validated on data from more than one country?** Yes. The framework was validated on cohorts from both the United Kingdom (UK Biobank, MIMIC-III) and China (Wenzhou ICU, systemic diseases, and myopia cohorts), demonstrating cross-population generalizability.---## ConclusionThe variational generative framework presented here represents a significant advance in the modeling of longitudinal multimodal clinical data. By explicitly representing each patient as an evolving physiological process and integrating heterogeneous data streams within a shared latent space, it addresses fundamental limitations of existing approaches—namely, the inability to handle missing modalities gracefully, the failure to capture temporal dependencies across visits, and the lack of principled cross-modal alignment.Across diverse clinical applications—including metabolomic and proteomic imputation, systemic disease prediction, myopia progression forecasting, ICU readmission risk assessment, and biological aging trajectory inference—the framework consistently demonstrates superior or competitive performance against established baselines while maintaining computational efficiency suitable for real-world deployment. Its validation across multiple international cohorts further underscores its robustness and generalizability.The availability of source code and the framework's modular architecture open the door to broad adoption and continued development by the clinical informatics community. As healthcare systems increasingly generate rich longitudinal multimodal data, frameworks of this kind will be essential for translating raw clinical information into actionable insights that improve patient outcomes.Thank you for reading
