Radiomics Pathomics Fusion Agent

AI-powered multimodal fusion of radiology (CT/MRI/PET) and pathology (H&E/IHC) imaging with clinical and genomic data for comprehensive cancer diagnostics and treatment prediction.

bg-szy Updated Sep 17, 2026

Install

Press C to copy

$ npx skillmds@latest add bg-szy/radiomics-pathomics-fusion-agent

Installs into ~/.claude/skills (or .claude/skills inside a project). Restart Claude Code to pick it up.

Download SKILL.md

＋ Add to collection

README badge

[![SkillMD](https://skillmd.com/badge/bg-szy/radiomics-pathomics-fusion-agent.svg)](https://skillmd.com/skills/bg-szy/radiomics-pathomics-fusion-agent)

Trust & safety

Audits pending

Not yet audited · commit dad26cc

Works with: Claude Code Claude.ai OpenAI Codex +

File contents

Radiomics Pathomics Fusion Agent

The Radiomics Pathomics Fusion Agent integrates multimodal medical imaging data from radiology (CT, MRI, PET) and digital pathology (H&E, IHC whole slide images) with clinical and genomic data using deep learning fusion architectures. It enables comprehensive cancer phenotyping, treatment response prediction, and prognostic modeling.

When to Use This Skill

When predicting treatment response using multimodal imaging.
For comprehensive tumor phenotyping combining macro and micro views.
To identify imaging biomarkers correlated with genomic features.
When building prognostic models from combined radiology-pathology.
For AI-powered second opinion integrating all imaging modalities.

Core Capabilities

Cross-Modal Fusion: Integrate radiology and pathology features using attention.
Radiomics Extraction: Compute 3D texture, shape, intensity features from CT/MRI.
Pathomics Extraction: Extract histopathological features from WSI.
Clinical Integration: Combine imaging with clinical variables and genomics.
Treatment Response Prediction: Predict chemotherapy, immunotherapy response.
Survival Prediction: Multi-modal prognostic modeling.

Supported Imaging Modalities

Modality	Features Extracted	Resolution
CT	Texture, shape, density	Volumetric 3D
MRI	Multi-sequence, perfusion	Volumetric 3D
PET	SUV, metabolic features	Volumetric 3D
H&E WSI	Nuclear, tissue architecture	40x magnification
IHC WSI	Marker quantification	20-40x
Multiplexed IF	Spatial protein patterns	Subcellular

Fusion Architectures

Architecture	Method	Strengths
Early Fusion	Concatenate features	Simple, baseline
Late Fusion	Combine predictions	Modular
Attention Fusion	Cross-modal attention	Interpretable
Multimodal Transformer	Self-attention across modalities	State-of-art
Graph Fusion	GNN for relationships	Spatial awareness

Workflow

Input: CT/MRI DICOM, pathology WSI, clinical data, optional genomics.
Segmentation: Tumor ROI extraction from radiology.
Radiomics: Extract 3D radiomic features.
Pathomics: Extract histopathology features via foundation models.
Fusion: Multimodal feature integration.
Prediction: Treatment response, survival, biomarker prediction.
Output: Integrated predictions, attention maps, explanations.

Example Usage

User: "Predict immunotherapy response for this lung cancer patient using their CT scan and biopsy pathology."

Agent Action:

python3 Skills/Oncology/Radiomics_Pathomics_Fusion_Agent/fusion_predict.py \
    --ct_dicom ct_scan/ \
    --wsi_path biopsy.svs \
    --clinical_data patient_clinical.json \
    --genomic_data tumor_wes.vcf \
    --task immunotherapy_response \
    --cancer_type nsclc \
    --fusion_method attention \
    --output fusion_prediction/

Radiomic Feature Categories

Category	Features	Count
Shape	Volume, surface area, sphericity	14
First-Order	Mean, variance, skewness, entropy	18
GLCM	Contrast, correlation, homogeneity	24
GLRLM	Run length, gray level emphasis	16
GLSZM	Zone size, gray level variance	16
GLDM	Dependence features	14
NGTDM	Texture features	5
Total		~107

Pathomics Feature Categories

Category	Source	Features
Nuclear	Segmentation	Size, shape, texture
Cellular	Detection	Density, clustering
Tissue	Architecture	Glandular, stromal ratios
Foundation Model	CONCH, TITAN, UNI	Deep embeddings
Spatial	Graph analysis	Neighborhood patterns

Output Components

Output	Description	Format
Prediction	Response/outcome probability	.json
Confidence	Prediction uncertainty	.json
Attention Maps	Cross-modal importance	.npy, .png
Feature Importance	Shapley values	.csv
ROI Highlights	Predictive regions	DICOM-SEG, GeoJSON
Report	Clinical summary	.pdf

Clinical Applications

Application	Modalities Used	Performance
NSCLC Immunotherapy	CT + H&E	AUC 0.82-0.88
HCC Survival	MRI + H&E	C-index 0.78
Breast Neoadjuvant	MRI + H&E	AUC 0.85
HNSCC HPV/Response	CT + H&E	AUC 0.89
CRC MSI Prediction	CT + H&E	AUC 0.86

AI/ML Components

Radiomics Pipeline:

PyRadiomics for feature extraction
3D-CNN for learned features
Transformer for volumetric analysis

Pathomics Pipeline:

Foundation models (CONCH, UNI, TITAN)
MIL (Multiple Instance Learning) for WSI
Graph networks for spatial patterns

Fusion Models:

Cross-attention transformers
Multimodal variational autoencoders
Contrastive learning for alignment

Prerequisites

Python 3.10+
PyRadiomics, SimpleITK
OpenSlide, HistoEncoder
PyTorch, transformers
CONCH/TITAN model weights
GPU with 16GB+ VRAM

Related Skills

Pathology_AI/CONCH_Agent - Pathology foundation model
Radiology_AI agents - Modality-specific analysis
Pan_Cancer_MultiOmics_Agent - Genomic integration
TMB_Estimation_Agent - Tumor mutational burden

Multimodal Integration Strategies

Strategy	Description	Use Case
Feature-Level	Combine extracted features	Limited data
Embedding-Level	Fuse latent representations	Moderate data
Decision-Level	Ensemble predictions	Interpretability
End-to-End	Joint training	Large data

Special Considerations

Data Alignment: Ensure imaging from same timepoint
Missing Modalities: Handle incomplete multimodal data
Class Imbalance: Balance training across outcomes
Interpretability: Attention maps for clinical trust
Validation: External multi-site validation essential

Quality Control

QC Check	Threshold	Action
CT coverage	>90% tumor	Rescan if needed
WSI quality	Blur score <X	Re-scan slide
Segmentation	Dice >0.85	Manual review
Feature stability	ICC >0.8	Robust features only

Regulatory Considerations

Aspect	Status
FDA Clearance	Individual modality tools cleared
Multimodal Fusion	Research use only (RUO)
Clinical Integration	PACS/LIS integration pathways
Explainability	Required for clinical adoption

Author

AI Group - Biomedical AI Platform

---name: radiomics-pathomics-fusion-agentdescription: AI-powered multimodal fusion of radiology (CT/MRI/PET) and pathology (H&E/IHC) imaging with clinical and genomic data for comprehensive cancer diagnostics and treatment prediction.---# Radiomics Pathomics Fusion AgentThe **Radiomics Pathomics Fusion Agent** integrates multimodal medical imaging data from radiology (CT, MRI, PET) and digital pathology (H&E, IHC whole slide images) with clinical and genomic data using deep learning fusion architectures. It enables comprehensive cancer phenotyping, treatment response prediction, and prognostic modeling.## When to Use This Skill* When predicting treatment response using multimodal imaging.* For comprehensive tumor phenotyping combining macro and micro views.* To identify imaging biomarkers correlated with genomic features.* When building prognostic models from combined radiology-pathology.* For AI-powered second opinion integrating all imaging modalities.## Core Capabilities1. **Cross-Modal Fusion**: Integrate radiology and pathology features using attention.2. **Radiomics Extraction**: Compute 3D texture, shape, intensity features from CT/MRI.3. **Pathomics Extraction**: Extract histopathological features from WSI.4. **Clinical Integration**: Combine imaging with clinical variables and genomics.5. **Treatment Response Prediction**: Predict chemotherapy, immunotherapy response.6. **Survival Prediction**: Multi-modal prognostic modeling.## Supported Imaging Modalities| Modality | Features Extracted | Resolution ||----------|-------------------|------------|| CT | Texture, shape, density | Volumetric 3D || MRI | Multi-sequence, perfusion | Volumetric 3D || PET | SUV, metabolic features | Volumetric 3D || H&E WSI | Nuclear, tissue architecture | 40x magnification || IHC WSI | Marker quantification | 20-40x || Multiplexed IF | Spatial protein patterns | Subcellular |## Fusion Architectures| Architecture | Method | Strengths ||--------------|--------|-----------|| Early Fusion | Concatenate features | Simple, baseline || Late Fusion | Combine predictions | Modular || Attention Fusion | Cross-modal attention | Interpretable || Multimodal Transformer | Self-attention across modalities | State-of-art || Graph Fusion | GNN for relationships | Spatial awareness |## Workflow1. **Input**: CT/MRI DICOM, pathology WSI, clinical data, optional genomics.2. **Segmentation**: Tumor ROI extraction from radiology.3. **Radiomics**: Extract 3D radiomic features.4. **Pathomics**: Extract histopathology features via foundation models.5. **Fusion**: Multimodal feature integration.6. **Prediction**: Treatment response, survival, biomarker prediction.7. **Output**: Integrated predictions, attention maps, explanations.## Example Usage**User**: "Predict immunotherapy response for this lung cancer patient using their CT scan and biopsy pathology."**Agent Action**:```bashpython3 Skills/Oncology/Radiomics_Pathomics_Fusion_Agent/fusion_predict.py \ --ct_dicom ct_scan/ \ --wsi_path biopsy.svs \ --clinical_data patient_clinical.json \ --genomic_data tumor_wes.vcf \ --task immunotherapy_response \ --cancer_type nsclc \ --fusion_method attention \ --output fusion_prediction/```## Radiomic Feature Categories| Category | Features | Count ||----------|----------|-------|| Shape | Volume, surface area, sphericity | 14 || First-Order | Mean, variance, skewness, entropy | 18 || GLCM | Contrast, correlation, homogeneity | 24 || GLRLM | Run length, gray level emphasis | 16 || GLSZM | Zone size, gray level variance | 16 || GLDM | Dependence features | 14 || NGTDM | Texture features | 5 || **Total** | | **~107** |## Pathomics Feature Categories| Category | Source | Features ||----------|--------|----------|| Nuclear | Segmentation | Size, shape, texture || Cellular | Detection | Density, clustering || Tissue | Architecture | Glandular, stromal ratios || Foundation Model | CONCH, TITAN, UNI | Deep embeddings || Spatial | Graph analysis | Neighborhood patterns |## Output Components| Output | Description | Format ||--------|-------------|--------|| Prediction | Response/outcome probability | .json || Confidence | Prediction uncertainty | .json || Attention Maps | Cross-modal importance | .npy, .png || Feature Importance | Shapley values | .csv || ROI Highlights | Predictive regions | DICOM-SEG, GeoJSON || Report | Clinical summary | .pdf |## Clinical Applications| Application | Modalities Used | Performance ||-------------|-----------------|-------------|| NSCLC Immunotherapy | CT + H&E | AUC 0.82-0.88 || HCC Survival | MRI + H&E | C-index 0.78 || Breast Neoadjuvant | MRI + H&E | AUC 0.85 || HNSCC HPV/Response | CT + H&E | AUC 0.89 || CRC MSI Prediction | CT + H&E | AUC 0.86 |## AI/ML Components**Radiomics Pipeline**:- PyRadiomics for feature extraction- 3D-CNN for learned features- Transformer for volumetric analysis**Pathomics Pipeline**:- Foundation models (CONCH, UNI, TITAN)- MIL (Multiple Instance Learning) for WSI- Graph networks for spatial patterns**Fusion Models**:- Cross-attention transformers- Multimodal variational autoencoders- Contrastive learning for alignment## Prerequisites* Python 3.10+* PyRadiomics, SimpleITK* OpenSlide, HistoEncoder* PyTorch, transformers* CONCH/TITAN model weights* GPU with 16GB+ VRAM## Related Skills* Pathology_AI/CONCH_Agent - Pathology foundation model* Radiology_AI agents - Modality-specific analysis* Pan_Cancer_MultiOmics_Agent - Genomic integration* TMB_Estimation_Agent - Tumor mutational burden## Multimodal Integration Strategies| Strategy | Description | Use Case ||----------|-------------|----------|| Feature-Level | Combine extracted features | Limited data || Embedding-Level | Fuse latent representations | Moderate data || Decision-Level | Ensemble predictions | Interpretability || End-to-End | Joint training | Large data |## Special Considerations1. **Data Alignment**: Ensure imaging from same timepoint2. **Missing Modalities**: Handle incomplete multimodal data3. **Class Imbalance**: Balance training across outcomes4. **Interpretability**: Attention maps for clinical trust5. **Validation**: External multi-site validation essential## Quality Control| QC Check | Threshold | Action ||----------|-----------|--------|| CT coverage | >90% tumor | Rescan if needed || WSI quality | Blur score <X | Re-scan slide || Segmentation | Dice >0.85 | Manual review || Feature stability | ICC >0.8 | Robust features only |## Regulatory Considerations| Aspect | Status ||--------|--------|| FDA Clearance | Individual modality tools cleared || Multimodal Fusion | Research use only (RUO) || Clinical Integration | PACS/LIS integration pathways || Explainability | Required for clinical adoption |## AuthorAI Group - Biomedical AI Platform

bg-szy/TOP-SKILLS/tree/main/skills/awesome-skills/radiomics-pathomics-fusion-agent commit dad26ccb34

Frequently asked questions

npx skillmds@latest add bg-szy/radiomics-pathomics-fusion-agent