Clinical Survival & Outcome Analysis

Clinical Survival & Outcome Analysis

Kaplan-Meier survival estimation, Cox proportional hazards regression, and risk stratification for clinical and real-world evidence (RWE) datasets.

When to Use This Skill

Use this skill when you need to:

• Estimate survival curves (Kaplan-Meier) with confidence intervals and risk tables
• Identify prognostic factors via Cox proportional hazards regression
• Stratify patients by risk using Cox model linear predictor
• Test proportional hazards assumption with Schoenfeld residuals
• Compare survival between groups (molecular subtypes, treatment arms, biomarker levels)
• Generate forest plots of hazard ratios for multi-covariate models

Don't use this skill for:

• ❌ Biomarker panel selection from omics → use lasso-biomarker-panel
• ❌ Differential expression analysis → use bulk-rnaseq-counts-to-de-deseq2
• ❌ Disease trajectory / longitudinal modeling → use disease-progression-longitudinal
• ❌ Genetic association / Mendelian randomization → use mendelian-randomization-twosamplemr

Installation

options(repos = c(CRAN = "https://cloud.r-project.org"))
if (!require('BiocManager', quietly = TRUE)) install.packages('BiocManager')

# Core (required)
install.packages(c('survival', 'ggplot2', 'ggprism', 'scales'))

# Enhanced KM curves with risk tables (recommended)
install.packages('survminer')

# Example data: TCGA BRCA (optional, needed for tcga_brca demo)
BiocManager::install('RTCGA.clinical')

Inputs

Required:

• Clinical data with columns for:
  • Time-to-event (numeric: days, months, or years)
  • Event indicator (binary: 0 = censored, 1 = event)
• Minimum 50 patients recommended (20+ events for reliable Cox estimates)

Optional:

• Stratification variable (e.g., molecular subtype, treatment arm, biomarker group)
• Covariates for Cox model (age, stage, receptor status, etc.)
• Pre-computed risk scores from upstream skills (e.g., lasso-biomarker-panel)

Formats: CSV/TSV with headers, or R data frame

Outputs

Primary results:

• cox_coefficients.csv — Hazard ratios with 95% CI and p-values for all covariates
• risk_scores.csv — Patient-level risk scores and risk group assignments
• clinical_annotated.csv — Full clinical data with added risk group column
• survival_summary.csv — Summary statistics per risk group (N, events, event rate, median survival)
• ph_assumption_test.csv — Schoenfeld residual test results (chi-sq, p-value per covariate)

Analysis objects (RDS):

• survival_model.rds — Complete analysis object for downstream use
  • Load with: model <- readRDS('results/survival_model.rds')
  • Contains: KM fits, Cox model, PH test, risk groups, clinical data, metadata
  • Access risk scores: model$cox$risk_scores
  • Access Cox model: model$cox$model
  • Required for: lasso-biomarker-panel (risk scores as features), downstream integration

Plots (PNG + SVG at 300 DPI):

• km_overall.png/.svg — Overall Kaplan-Meier curve with confidence interval
• km_stratified.png/.svg — Stratified survival curves with log-rank p-value
• forest_plot.png/.svg — Forest plot of hazard ratios with significance markers
• km_risk_groups.png/.svg — Risk group survival curves with log-rank test
• schoenfeld_diagnostics.png/.svg — PH assumption diagnostic plots
• cumulative_hazard.png/.svg — Cumulative hazard function

Reports:

• survival_report.md — Comprehensive markdown report
• survival_report.pdf — Agent-generated PDF report with Introduction, Methods, Results, Conclusions, and embedded figures

⚠️ PDF style rules:

• US Letter page size (8.5 × 11 in) — always set page dimensions explicitly; do not rely on library defaults
• No Unicode superscripts — use 3.36e-06 or 3.36 × 10^(-6), not Unicode superscript chars (they render as ■ in PDF fonts)
• No half-empty pages — group headings with their content; only page-break before major sections (Results, Conclusions)
• Figures ≥80% page width — multi-panel figures must be large enough to read; never embed below 50% width

Clarification Questions

🚨 ALWAYS ask Question 1 FIRST.

1. Example or Own Data? (ASK THIS FIRST):

• a) TCGA Breast Cancer (recommended for demo)
  • 1,100+ patients with overall survival, molecular subtypes (HR+/HER2-, HR+/HER2+, HER2+, Triple Negative), stage, age, ER/PR/HER2 status
  • Requires download (~50MB via RTCGA.clinical, cached after first run)
• b) NCCTG Lung Cancer (quick demo, no download)
  • 228 advanced lung cancer patients, sex stratification, ECOG performance status
  • Built-in R dataset — runs instantly
• c) I have my own clinical data to analyze
  • Continue to Questions 2-3 below

IF EXAMPLE SELECTED (option a or b): Proceed to Question 2 for analysis options. Skip Question 3.

2. Analysis Options (structured — for all datasets):

• Stratification variable?
  • a) Default for dataset (mol_subtype for TCGA BRCA, sex for Lung)
  • b) Stage
  • c) Age group
• Risk stratification method?
  • a) Median split — 2 groups (recommended)
  • b) Tertiles — 3 groups
  • c) Quartiles — 4 groups

3. Data Details (own data only — free-text OK):

• What is the time column name? Units (days/months/years)?
• What is the event column name? What does 1 represent (death/relapse/progression)?
• What stratification variable? What covariates for the Cox model?

Standard Workflow

Note: Run from the OmicsClaw root directory and add the workflow scripts to sys.path:

import sys; import os; sys.path.insert(0, os.path.abspath('knowledge_base/scripts/survival-analysis-clinical'))

🚨 MANDATORY: USE SCRIPTS EXACTLY AS SHOWN - DO NOT WRITE INLINE CODE 🚨

Step 1 - Load data:

source("scripts/load_example_data.R")
data <- load_example_data(dataset = "tcga_brca")
# OR: data <- load_example_data(dataset = "lung")
# OR: data <- load_user_data("path/to/clinical.csv", time_col = "time", event_col = "status")

DO NOT write custom data loading code. Use the loader functions.

✅ VERIFICATION: You MUST see: "✓ TCGA BRCA data loaded successfully!" (or similar)

Step 2 - Run survival analysis:

source("scripts/basic_workflow.R")
result <- run_survival_analysis(data)
# Optional: result <- run_survival_analysis(data, risk_strata_method = "tertiles")
# Optional: result <- run_survival_analysis(data, covariates = c("age", "stage"))

DO NOT write inline Cox/KM code (coxph, survfit, etc.). Just source and call.

✅ VERIFICATION: You MUST see: "✓ Survival analysis completed successfully!"

❌ IF YOU DON'T SEE THIS: You wrote inline code. Stop and use source().

Step 3 - Generate visualizations:

source("scripts/survival_plots.R")
generate_all_plots(result, output_dir = "results")

🚨 DO NOT write inline plotting code (ggsave, ggplot, ggsurvplot, etc.). Just use generate_all_plots(). 🚨

The script handles PNG + SVG export with graceful fallback for SVG dependencies.

✅ VERIFICATION: You MUST see: "✓ All survival plots generated successfully!"

Step 4 - Export results:

source("scripts/export_results.R")
export_all(result, output_dir = "results")

DO NOT write custom export code. Use export_all() to save all outputs including RDS.

✅ VERIFICATION: You MUST see:

• "=== Export Complete ==="

⚠️ CRITICAL - DO NOT:

• ❌ Write inline Cox/KM code (coxph, survfit) → STOP: Use source("scripts/basic_workflow.R")
• ❌ Write inline plotting code (ggsave, ggplot, ggsurvplot) → STOP: Use generate_all_plots()
• ❌ Write custom export code → STOP: Use export_all()
• ❌ Try to install svglite → script handles SVG fallback automatically

⚠️ IF SCRIPTS FAIL - Script Failure Hierarchy:

1. Fix and Retry (90%) - Install missing package, re-run script
2. Modify Script (5%) - Edit the script file itself, document changes
3. Use as Reference (4%) - Read script, adapt approach, cite source
4. Write from Scratch (1%) - Only if genuinely impossible, explain why

NEVER skip directly to writing inline code without trying the script first.

Common Issues

Agent Summary Guidelines

When presenting final results to the user, the agent MUST:

1. Report the C-index (concordance) from the Cox model — but see EPV rule below
2. Check result$median_reliable — if FALSE, report "Median survival: Not reached" and use landmark survival rates (from result$landmark_survival) instead
3. Report landmark survival rates (1-year, 3-year, 5-year OS with 95% CI) — these are always more robust than median, especially for low-event datasets
4. State PH assumption result (satisfied or violated, with global p-value)
5. List significant covariates with HR, 95% CI, and p-value
6. Report EPV (events per variable) — if result$epv < 10, warn that model may be overfitted
7. Report excluded patients — if result$n_excluded > 0, note how many were excluded from Cox model
8. Report risk group separation (log-rank chi-sq and p-value)
9. Report PDF status — if PDF generation failed, say so and note markdown report is available
10. Never fabricate survival curve descriptions — reference the actual generated plots
11. Never report unreliable medians as if they are reliable — when upper CI = NA, the KM curve did not cross 50% and the median is an unreliable extrapolation
12. Methods section MUST match actual model — list only covariates from names(coef(result$cox$model)). Check result$dropped_covariates and report what was excluded and why. NEVER list covariates from memory; always verify against the fitted model.
13. Report dropped covariates — if result$dropped_covariates is non-empty, list each dropped variable and reason (rare levels, collinearity) in the Methods section
14. Report reference groups — for each categorical covariate, state the reference level and its N (from result$reference_levels). If N < 50, flag the HR as "unstable due to small reference group (N=X)"
15. Report informative missingness — if any entry in result$diagnostics$missing_assessment has informative = TRUE, report the event rate comparison prominently and note selection bias risk
16. Report follow-up anomalies — if result$diagnostics$followup_anomaly is TRUE, investigate and explain prominently. Do NOT dismiss as "expected" without evidence.

⚠️ CRITICAL REPORTING RULES:

• EPV < 10 + C-index: If result$epv < 10, you MUST describe the C-index as "potentially overfitted" or "unreliable". NEVER use "good" or "moderate discrimination" without this caveat. The C-index is optimistically biased when EPV is low.
• PH violation + forest plot/Cox table: If global PH test p < 0.05, you MUST include a prominent warning on the forest plot caption AND any Cox results table: "PH assumption violated (p=X) — HRs represent time-averaged effects and may be misleading." Do NOT present HRs as primary findings without this warning.
• Small reference groups: If a key finding involves a categorical covariate whose reference group has N < 50, flag the estimate as unstable. State the reference group N explicitly.
• Never fabricate group sizes or statistics. All Ns, HRs, CIs, and p-values in the report text MUST be copied from the script console output or exported CSV files. Do NOT estimate, round from memory, or recalculate group sizes. If a number is not in the output, re-run the relevant step or read the exported file.

Interpretation Guidelines

• C-index > 0.7: Good model discrimination — ONLY if EPV >= 10. If EPV < 10, say "potentially overfitted (EPV = X)"
• C-index 0.6-0.7: Moderate — useful combined with clinical factors
• C-index ~ 0.5: No better than chance
• HR > 1: Higher hazard (worse prognosis) per unit increase
• HR < 1: Lower hazard (protective effect)
• HR 95% CI includes 1.0: Not statistically significant
• PH global p < 0.05: Proportional hazards assumption violated — HRs are time-averaged and may be misleading. Must be stated prominently on forest plots and Cox tables, not buried in a later section.
• EPV < 10: Model underpowered — C-index likely optimistically biased; consider fewer covariates. NEVER call the C-index "good" when EPV < 10.
• Median survival "Not reached": KM curve never crosses 50% — use landmark survival rates instead
• Low event rate (<15%): KM curves may drop steeply in the tail due to small at-risk set (heavy censoring), not because most patients die. Always check N at risk at each timepoint.
• Median follow-up < 2 yr with max obs > 5 yr: Likely a data quality artifact — investigate completeness of follow-up times for censored patients before interpreting results.

Suggested Next Steps

1. Biomarker panel discovery — Use risk scores as features → lasso-biomarker-panel
2. Pathway enrichment — If molecular subtypes differ → functional-enrichment-from-degs
3. Multi-omics integration — Combine clinical + omics → multi-omics-integration-mofa
4. Disease trajectory — Map temporal progression → disease-progression-longitudinal
5. Clinical trial landscape — Search related interventional trials → clinicaltrials-landscape

Related Skills

References

• Cox DR. Regression Models and Life-Tables. J R Stat Soc B. 1972;34(2):187-220.
• Kaplan EL, Meier P. Nonparametric Estimation from Incomplete Observations. JASA. 1958;53(282):457-481.
• Cancer Genome Atlas Network. Comprehensive molecular portraits of human breast tumours. Nature. 2012;490:61-70.
• Loprinzi CL, et al. Prospective evaluation of prognostic variables from patient-completed questionnaires. J Clin Oncol. 1994;12:601-607.
• Therneau TM. A Package for Survival Analysis in R. R package survival.
• See references/cox-regression-guide.md for detailed Cox PH interpretation
• See references/risk-stratification-guide.md for risk group methodology