**Article: Scalable Multimodal Learning for Multiplexed Imaging with VirTues**
Multiplexed imaging technologies such as immunofluorescence multiplexed imaging (IMC), CODEX, Orion, and MIBI enable the simultaneous visualization of dozens to hundreds of molecular markers on tissue samples. These advances have dramatically improved our ability to characterize complex tissue microenvironments. Yet, despite their power, standard deep learning architectures struggle to leverage these rich datasets due to modality-specific design constraints. VirTues is introduced as a scalable solution — a vision transformer (ViT)-based encoder–decoder architecture purpose-built for highly multiplexed biological imaging.
—
### **VirTues Architecture**
VirTues is designed to overcome key limitations of conventional convolutional neural networks (CNNs) and standard vision transformers when applied to multiplexed imaging. Unlike RGB-based models that assume three fixed channels with predefined semantic meanings, multiplexed data involve variable numbers of channels, each carrying distinct biological information.
The architecture adopts an **encoder–decoder framework** built on modified ViT blocks. Input images are divided into small patches (128×128 pixels), and both **image tokens** and **biologically informed marker tokens** are fused at the token level. This fusion ensures that each channel’s biological meaning is preserved and explicitly represented throughout the model.
To handle variable channel counts and capture complex inter-channel relationships, VirTues introduces:
– **Multi-channel tokenization** that preserves spatial and channel-specific information.
– **Protein embeddings** derived from ESM-2 (a protein language model), providing a biologically informed prior for each marker.
– **Sparse attention mechanisms**, including:
– **Marker attention**, which captures correlations between different markers at the same spatial location.
– **Spatial attention**, which models relationships among patches within the same marker.
The encoder consists of 16 alternating transformer blocks, each using eight attention heads. A shallow decoder with four blocks reconstructs the original image or generates new channels, enabling both representation learning and channel synthesis.
—
### **Tokenization and Masking**
Each image channel is split into 8×8 pixel patches, and every patch is flattened and projected into a shared embedding space. A learned protein embedding corresponding to the marker is added to all patches from that channel, ensuring that the model can distinguish the origin of each token.
During training, **channel-wise masking** is applied by randomly masking between 60% and 100% of image tokens. Masked tokens retain their marker information, allowing VirTues to learn robust representations even when large portions of the input are missing.
—
### **Pretraining and Representation Learning**
VirTues is trained using a **masked autoencoding objective**, minimizing the mean squared error between reconstructed and original pixel intensities. The model is pretrained end-to-end on large-scale, curated collections of multiplexed imaging data.
Two main datasets support pretraining:
– An **IMC-only corpus** comprising 14 datasets with over 8,800 images and 146 markers.
– An **extended multi-technology corpus** that integrates CODEX, Orion, MIBI, and additional IMC datasets, spanning over 5,100 patients and 239 markers.
Data augmentation includes random rotations, flips, hierarchical subsampling, and controlled marker dropout to improve robustness and generalization.
—
### **Downstream Applications**
VirTues supports multiple downstream tasks through frozen or fine-tuned encoders:
#### **1. Cell-, Niche-, and Tissue-Level Representations**
Patch-level embeddings are aggregated using attention-weighted averaging to produce:
– **Cell-level representations**, based on overlapping image crops and pixel-area weighting.
– **Niche- and tissue-level representations**, derived from non-overlapping crops.
Attention-based aggregation allows the model to emphasize biologically relevant regions in multi-omic contexts.
#### **2. Panoptic Segmentation**
VirTues includes a U-Net-style segmentation head for:
– **Cell instance segmentation**, identifying individual cell boundaries.
– **Cell type prediction**, assigning each cell a phenotype label.
The segmentation module operates on encoded patch tokens and has been validated against specialized tools such as Cellpose, InstanSeg, and StarDist.
—
### **Evaluation and Benchmarking**
VirTues is evaluated across a wide range of tasks:
– **Masked reconstruction** under different masking strategies.
– **Zero-shot generalization** to unseen markers and datasets.
– **Cell classification** across multiple cohorts and annotation schemes.
– **Cross-cohort segmentation and annotation**, demonstrating transferability across technologies.
– **Clinical risk stratification** based on tissue architecture.
– **Therapeutic response prediction** in breast cancer patients.
Performance is benchmarked against ResNet, CA-MAE, and KRONOS. VirTues consistently achieves superior accuracy with far fewer parameters and greater flexibility in marker configuration.
—
### **Key Findings**
– VirTues effectively handles highly multiplexed data with hundreds of markers.
– Marker embeddings from protein language models are critical for generalization.
– The architecture supports zero-shot learning for new markers and unseen datasets.
– Sparse attention mechanisms enable efficient training and inference.
– VirTues-derived representations capture biologically meaningful tissue structure and predict patient outcomes.
—
### **FAQ**
**Q1: What makes VirTues different from standard vision transformers?**
VirTues modifies the ViT architecture to handle variable numbers of channels, each with distinct biological meaning. It fuses image tokens with protein-derived marker embeddings and uses specialized marker and spatial attention mechanisms.
**Q2: Can VirTues work with new markers not seen during pretraining?**
Yes. Thanks to protein embeddings and token-based fusion, VirTues can generalize to unseen markers in a zero-shot setting.
**Q3: How does VirTues handle different imaging technologies?**
The model is pretrained on a multi-technology corpus, enabling robust cross-technology generalization without retraining.
**Q4: What downstream tasks does VirTues support?**
VirTues supports cell classification, instance segmentation, tissue representation, survival analysis, and therapeutic response prediction.
**Q5: Is VirTues computationally efficient?**
Yes. Despite its flexibility and accuracy, VirTues uses significantly fewer parameters than comparable models such as CA-MAE, thanks to its tokenization strategy and sparse attention design.
—
### **Conclusion**
VirTues represents a major step forward in scalable, multimodal learning for spatial omics. By unifying information across diverse imaging technologies and marker panels, it provides a flexible and generalizable framework for analyzing complex tissue biology. Its ability to support zero-shot learning, segmentation, and clinically relevant risk modeling highlights its potential to become a foundational model for precision pathology and systems-level tissue analysis.
