Model Calibration Curve

Model Calibration Curve

When to Use

Use this skill when you need to:

• validate a survival model with bootstrap calibration curves;
• compare predicted and observed survival probabilities at multiple horizons;
• export calibration statistics together with a PDF visualization.

Typical user requests:

• "Generate 1-, 2-, and 3-year calibration curves for this prognosis model."
• "Check whether the Cox model built from age, gender, and risk is well calibrated."
• "Export calibration statistics and a calibration PDF from this clinical cohort."

When Not to Use

Do not use this skill for:

• nomogram construction;
• univariate or multivariable Cox feature screening;
• ROC, calibration-free discrimination, or decision-curve analysis;
• non-survival endpoints or multiclass classification tasks.

When to Read External Files

Input Validation

This skill accepts:

• A clinical CSV file with sample IDs as row names, survival time, event indicator, and pre-selected prognostic features
• Requests to assess calibration of a survival (Cox) model via bootstrap resampling at one or more prediction horizons

If the user's request does not involve survival model calibration from a clinical CSV file — for example, asking to construct a nomogram, screen Cox features, generate an ROC curve, analyze a decision curve, or work with non-survival outcomes — do not proceed with this workflow. Instead respond:

"model-calibration-curve is designed to validate survival model calibration by generating bootstrap calibration curves from a clinical CSV file. Your request appears to be outside this scope. Please use a nomogram-construction skill for nomogram building, a roc-diagnostic-performance skill for ROC analysis, or a decision-curve-analysis skill for DCA."

Prerequisites

R packages required: rms, qs, openxlsx, optparse.

Install with:

install.packages(c("rms", "qs", "openxlsx", "optparse"), repos = "https://cloud.r-project.org")

Or run the bootstrap installer:

Rscript scripts/install_dependencies.R

Note: --help requires optparse to be loaded. If the package check fires before option parsing, install optparse first, then run --help. The root fix (deferring heavy package checks until after argument parsing) must be applied in scripts/main.R.

Usage

Rscript scripts/main.R \
  --data_file ./clinical_data.csv \
  --features age,stage,risk \
  --years 1,2,3 \
  --output_dir ./output/

Arguments

Input Format

Clinical Data (--data_file)

CSV file with row names as sample IDs and columns for model features, survival time, and event indicator.

"",age,gender,stage,futime,fustat,risk
"SAMPLE_001",">65","Female","StageI&II",1.12,0,"high"
"SAMPLE_002","<=65","Male","StageIII&IV",1.92,1,"high"
"SAMPLE_003",">65","Male","StageI&II",4.47,1,"low"

Requirements

• File extension must be .csv.
• Row names must be unique sample IDs.
• All requested features plus time_col and event_col must exist.
• Survival time values must be finite numbers greater than 0.
• Event values must use 0/1 encoding.
• Complete-case filtering must leave at least 30 samples and at least 10 events.

Feature Selection (--features)

• Comma-separated without line breaks: age,gender,risk
• Character predictors are converted to factors before Cox fitting.
• If every requested feature is absent after validation, the run stops.

Output Files

calibration_statistics.xlsx

Workbook sheets:

• Time_Point_Stats: predicted mean, observed mean, and bias-corrected mean for each calibration horizon.
• Model_Summary: overall C-index, sample count, event count, selected features, and fitted formula.

Workflow

Step 1: Validate Input

• Confirm the clinical CSV exists and is readable.
• Check that requested features, survival time, and event columns are present.
• Remove incomplete rows across all required columns.

Step 2: Prepare Survival Modeling Data

• Convert survival time and event columns to numeric.
• Convert character predictors to factors.
• Reject data with non-positive follow-up times, invalid event coding, too few samples, or too few events.

Step 3: Build Calibration Models

• Fit a Cox proportional hazards model with the requested features.
• Run rms::calibrate() for each prediction horizon using bootstrap resampling.
• Compute the concordance index for the fitted model.

Step 4: Save Outputs

• Serialize the calibration result bundle as .qs.
• Export statistics to Excel.
• Render a combined calibration PDF.
• Record session metadata for reproducibility.

Examples

Basic Calibration Analysis

Rscript scripts/main.R \
  --data_file clinical_data.csv \
  --features age,stage,risk \
  --output_dir ./output/

Custom Horizons And Bootstrap Count

Rscript scripts/main.R \
  --data_file clinical_data.csv \
  --features age,gender,risk \
  --years 1,3,5 \
  --bootstrap_reps 1500 \
  --output_dir ./custom_output/

Custom Plot Styling

Rscript scripts/main.R \
  --data_file clinical_data.csv \
  --features age,stage,risk \
  --plot_width 7 \
  --plot_height 6 \
  --line_width 2 \
  --colors "#1B9E77,#D95F02,#7570B3" \
  --plot_title "Three-Horizon Calibration" \
  --output_dir ./styled_output/

With Bundled Test Data

Rscript scripts/main.R \
  --data_file tests/data/sample_clinical_survival_data.csv \
  --features age,gender,risk \
  --bootstrap_reps 20 \
  --output_dir tests/output/ \
  --overwrite

Error Handling

IF error persists, READ: references/troubleshooting.md

Testing

Smoke Test With Included Data

Rscript scripts/main.R --help

Rscript scripts/main.R \
  --data_file tests/data/sample_clinical_survival_data.csv \
  --features age,gender,risk \
  --bootstrap_reps 20 \
  --output_dir tests/output/ \
  --overwrite

Automated Smoke Test Script

Rscript tests/run_smoke_test.R

Optional shell wrapper:

bash tests/run_smoke_test.sh

Expected Output

tests/output/
|-- data/calibration_data.qs
|-- plot/calibration_curve.pdf
|-- session_info.txt
`-- table/calibration_statistics.xlsx

References

1. Harrell FE. Regression Modeling Strategies.
2. Steyerberg EW et al. Assessing the performance of prediction models.
3. Austin PC, Steyerberg EW. Graphical assessment of calibration for survival models.

For detailed algorithm, READ: references/algorithm.md

Implementation Checklist

• CLI parsing with optparse
• set.seed() for reproducibility
• Top-level CRAN dependency checks
• Session info recording
• Timeout parameter exposed as CLI option
• Relative-path source() usage via get_script_dir()
• Modular script structure in scripts/
• Test data provided in tests/data/
• Error handling with SKILL_* codes
• Reference docs provided in references/

Last updated: 2026-04-27 | Version: 2.1.0