Comparator

Comparator Skill

Creates and maintains weighted XLSX comparison matrices. Evidence can come from CIRs, working-folder documents, LLM knowledge, or user description — a pre-existing CIR is not required. Python scripts (skills/comparator/matrix_ops.py, skills/comparator/matrix_builder.py) handle ALL deterministic XLSX work; the LLM handles capability discovery, evidence reading, and tick judgment.

Phase 0 — Determine Operation & Collect Inputs

Identify which operation the user wants:

Also collect:

• Domain label — e.g., devops-platforms (used to find domains/{domain}.md and to name output files). Can be inferred from solution names if not provided.
• Output path — defaults to reports/<domain>/<domain>-matrix.xlsx or a timestamped copy of the source matrix.

Phase 1 — Load Context

Step 1: Domain knowledge (optional)

ls domains/ 2>/dev/null || echo "no domains dir"
cat domains/{domain}.md 2>/dev/null || echo "no domain file — will bootstrap if needed"

If the domain file exists, use it for: archetype validation, inference patterns, functional equivalence rules, CIR evidence rules, category cross-reference.

If no domain file exists: proceed without it. One will be created in Phase 8 after the first run.

Step 2: Find available evidence

Search for evidence about the solutions being compared, in priority order:

# 1. CIRs from solution-researcher
ls reports/*/*.md 2>/dev/null | grep -i "<solution-name>"

# 2. Any markdown/text files in the working folder
ls *.md *.txt 2>/dev/null

# 3. Matrix (if operating on an existing one)
python3 skills/comparator/matrix_ops.py info --src [matrix.xlsx]
python3 skills/comparator/matrix_ops.py extract-features --src [matrix.xlsx]
python3 skills/comparator/matrix_ops.py scores --src [matrix.xlsx]

Triage the evidence found:

• CIR present → primary evidence source (high confidence)
• Other document present → secondary evidence (medium confidence)
• No documents → use LLM knowledge (label confidence as "inferred from LLM knowledge")

Announce to the user what evidence was found and its confidence level before proceeding.

Phase 2 — Capability Framework

Run this phase only for compare and build operations. Skip for add-platform, reorder, combo, verify.

Step 1: Derive capability categories and features

Using all available evidence (CIRs, documents, LLM knowledge), derive a capability framework for the domain:

1. Identify 5–12 capability categories relevant to the solutions being compared
2. For each category, identify 3–10 specific features that differentiate solutions
3. Number categories: 1. Category Name, 2. Category Name, …

Good capability frameworks are:

• User-facing — observable behaviours, not implementation details
• Differentiating — features where solutions actually differ
• Comprehensive — cover the full product surface, not just the most obvious area

Write the proposed framework as a structured list. For example:

1. Core Architecture
   - Multi-tenancy support
   - Kubernetes-native deployment
   - Serverless option

2. Developer Experience
   - CLI tool
   - GitOps integration
   - Preview environments
   ...

Step 2: Confirm with user

Present the framework and ask:

"Here's the capability framework I'll use for this comparison. Does this cover what matters to you? Add, remove, or rename any categories or features before I score."

Wait for confirmation or edits. Incorporate changes, then proceed.

Phase 3 — Priority Assignment (Optional)

Run this phase for compare and build operations. Optional for add-platform (can use existing priorities from the matrix).

Default priorities

If the user wants to proceed with defaults, use:

• Critical (weight 5): Must-have capabilities — showstoppers if absent
• High (weight 3): Important differentiators
• Medium (weight 2): Nice-to-have
• Low (weight 1): Minor or edge-case features

Assign default priorities based on the capability framework and domain context.

Interactive priority review (optional)

Ask the user:

"Should I assign priorities automatically, or would you like to review them? Priorities drive the weighted score — Critical features count 5×, High 3×, Medium 2×, Low 1×. (Say 'auto' to skip this step.)"

If user says 'auto': assign priorities using LLM judgment and document the rationale. Proceed.

If user wants to review: present each category with proposed priorities and allow adjustment:

1. Core Architecture
   - Multi-tenancy support            → Critical  (showstopper for enterprise)
   - Kubernetes-native deployment     → High      (important but workaroundable)
   - Serverless option                → Low       (niche use case)

Accept changes feature by feature or category by category. Produce the final priority mapping before continuing.

Phase 4 — Process Evidence (Tick Decisions)

This is where the LLM adds value. For each solution, read all available evidence and decide tick / no-tick for every feature in the capability framework.

Step 1: Identify evidence variant (if a CIR is present)

• Variant A — Has status markers (✅ Core, 🔧 Optional, ⚠️ Unverified, ❌ Not Documented). Both narrative and checklist are evidence.
• Variant B — Has product-agnostic checklist (no status markers, unchecked items). Only narrative sections are evidence. Checklist MUST NOT be used for tick decisions.
• Non-CIR document — Use judgment; treat all sections as narrative evidence.
• LLM knowledge only — Apply with care; flag confidence as "inferred".

Step 2: Map features — work matrix-first

Go category by category. For each feature:

1. Search all available evidence for that capability
2. Decide tick (✔) or no-tick
3. Apply inference patterns from domain knowledge if available
4. Record confidence: CIR-confirmed | doc-confirmed | inferred | user-confirmed

Write features mapping to a temp file:

cat > /tmp/comparator-features-<solution>.json << 'EOF'
{
  "Single Sign-On (SSO)": true,
  "Multi-tenancy support": true,
  "Custom ingress controller": false
}
EOF

Rules:

• Every feature in the framework MUST appear — either true or false. No omissions.
• When no evidence exists for a feature, default to false and note "no evidence found".
• Check archetype alignment if domain knowledge defines archetypes.

Step 3: New rows (if any)

If evidence reveals capabilities not in the framework:

1. Apply the new row criteria (genuine user-facing capability, not implementation detail)
2. For mature matrices (400+ features), zero new rows is expected
3. Write to a temp file:

cat > /tmp/comparator-new-rows.json << 'EOF'
[{"category": "3. CI/CD", "feature": "Pipeline analytics dashboard", "priority": "Medium", "ticked": true}]
EOF

Phase 5 — Execute (Python)

For compare (new matrix from scratch):

First, build the config JSON:

cat > /tmp/comparator-build-config.json << 'EOF'
{
  "title": "<Domain> — Capability Comparison Matrix",
  "categories": [
    {
      "name": "1. Category Name",
      "features": [
        {"name": "Feature A", "priority": "Critical"},
        {"name": "Feature B", "priority": "High"}
      ]
    }
  ],
  "platforms": [
    {"name": "Solution A", "features": ["Feature A"]},
    {"name": "Solution B", "features": ["Feature A", "Feature B"]}
  ]
}
EOF

Then build the matrix:

python3 skills/comparator/matrix_builder.py \
    --config /tmp/comparator-build-config.json \
    --out [output.xlsx]

Then reorder columns by score:

python3 skills/comparator/matrix_ops.py reorder-columns \
    --src [output.xlsx] --out [output.xlsx]

For add-platform (existing matrix):

python3 skills/comparator/matrix_ops.py add-platform \
    --src [matrix.xlsx] \
    --out [output.xlsx] \
    --platform "[Platform Name]" \
    --features /tmp/comparator-features-<platform>.json \
    --new-rows /tmp/comparator-new-rows.json

python3 skills/comparator/matrix_ops.py reorder-columns \
    --src [output.xlsx] --out [output.xlsx]

For other operations:

# Combo column
python3 skills/comparator/matrix_ops.py combo \
    --src [matrix] --out [output] --name "A+B" --platform-a "A" --platform-b "B"

# Verify ticks
python3 skills/comparator/matrix_ops.py verify --src [matrix]

# Reorder rows within category
python3 skills/comparator/matrix_ops.py reorder-rows \
    --src [matrix] --out [output] --category "Category Name" --order /tmp/order.json

# Reorder categories
python3 skills/comparator/matrix_ops.py reorder-categories \
    --src [matrix] --out [output] --order /tmp/cat-order.json

Phase 6 — Verify

python3 skills/comparator/matrix_ops.py scores --src [output.xlsx]

Cross-check:

• Does the tick count align with the expected archetype range (if domain knowledge defines archetypes)?
• Are the strong/weak category patterns consistent with the evidence?
• Orphans list from add-platform: if > 0, investigate — wrong feature name or new row candidate.

Phase 7 — Markdown Summary

Always produce this output, regardless of operation type. Write to reports/<domain>/<task-name>-comparison-summary.md.

Structure:

# <Solution A> vs <Solution B> — Comparison Summary
*Generated: YYYY-MM-DD | Matrix: <filename> | Total capabilities: N*

## Overall Scores (Priority-Weighted)

| Rank | Solution | Weighted Score | Raw Tick Count |
|---|---|---|---|
| 1 | Solution A | 142 | 67 |
| 2 | Solution B | 118 | 61 |

**Winner: Solution A** (+24 weighted points, +6 raw capabilities)

## Category-by-Category Breakdown

| Category | Solution A | Solution B | Winner |
|---|---|---|---|
| 1. Core Architecture | 28 | 22 | Solution A |
| 2. Developer Experience | 35 | 40 | Solution B |
| ... | | | |

## Key Differentiators

**Solution A leads on:**
- [feature] — present in A, absent in B (Priority: Critical)
- ...

**Solution B leads on:**
- [feature] — present in B, absent in A (Priority: High)
- ...

## Shared Capabilities
Both solutions support: [feature list]

## Gaps in Both
Neither solution supports: [feature list] — potential shortlist criterion.

## Evidence Quality
| Solution | Primary Source | Confidence |
|---|---|---|
| Solution A | CIR: reports/... | CIR-confirmed |
| Solution B | LLM knowledge | Inferred |

For operations other than compare (reorder, combo, verify), produce a shorter summary covering: what changed, new rankings, output file location.

Phase 8 — Domain Knowledge Enrichment

After completing the operation, propose timestamped append-only additions to domains/{domain}.md.

If no domain file exists: create it with this structure:

# <Domain> Domain Knowledge

## Evaluation Categories
[list from the capability framework]

## Key Terminology
[domain-specific terms identified during this run]

## Platform Archetypes
[archetypes identified, with expected tick ranges]

## Inference Patterns
[patterns used during tick decisions]

## CIR Evidence Rules
[rules applied]

If domain file exists: propose additions only:

• New archetype discovered
• New inference pattern
• Tick-count baseline update
• New feature name equivalence
• New evaluation category

Always ask for approval before writing. Additions are append-only and timestamped.

Phase 9 — Present Results

Show the user:

1. Ranked scores table (from scores command)
2. Markdown summary (path to the generated file)
3. Matrix file location
4. What changed: platforms added, new rows, orphans, columns reordered
5. Domain knowledge additions (pending approval)
6. Evidence used and confidence level for each solution

Phase 10 — Self-Improve

After each successful operation, append a run log entry.

### {YYYY-MM-DD} — {Operation}: {Solution(s)}
- Matrix: {file}
- Evidence: {CIR / document / LLM knowledge}
- Tick count: {count per solution} (archetype: {expected range or N/A})
- New rows added: {count}
- Orphan features: {count}
- Priority adjustments: {any user changes}
- Observations: {accuracy notes, patterns}
- Changes made: {any script updates}

Scope boundary: Only update files inside skills/comparator/. Never modify other skills' files.

Run Log

2026-03-18 — Northflank (IT-SC-05) — add-platform

• Matrix: /tmp/it-sc05-matrix.xlsx (test fixture, 4 existing platforms, 7 feature rows)
• Output: /tmp/it-sc05-matrix-northflank.xlsx
• Evidence: CIR (solution-researcher output)
• Tick count: 6/7 (archetype: PaaS/IDP K8s-native — expected range 5–8 ✓)
• Features ticked: GitOps ✓, Blue/Green Deployments ✓, CLI Tool ✓, Preview Environments ✓, Kubernetes Support ✓, Multi-cloud ✓
• Features not ticked: Canary Deployments ✗ (no mention in any of 4 responding AI sources — inferred absent)
• New rows added: 0
• Orphans: 0
• Observations: matrix_ops.py add-platform --out flag required (not --dst); inference from CIR evidence for Canary was reliable
• Changes made: none