Regression Bisect

regression-bisect

Automatically identify the commit that introduced a regression using git bisect.

Triggers

Alternate expressions and non-obvious activations (primary phrases are matched automatically from the skill description):

• "git bisect" → binary search for regression commit
• "when did this break" → regression commit discovery

Purpose

This skill automates regression root cause analysis by:

• Using git bisect to binary search commit history
• Running automated tests at each bisect point
• Identifying the exact commit that introduced a failure
• Providing blame information and diff context
• Suggesting potential fixes based on the breaking change

Behavior

When triggered, this skill:

1. Validates prerequisites:

   • Confirms reproducible failure exists
   • Verifies test automation is available
   • Ensures clean working directory
   • Identifies good and bad commits

2. Executes git bisect:

   • Starts bisect with known good/bad commits
   • Runs test suite at each bisect point
   • Marks commits as good/bad based on test results
   • Continues until breaking commit is found

3. Analyzes breaking commit:

   • Shows commit message and metadata
   • Displays full diff of the breaking change
   • Identifies changed files and functions
   • Maps to requirements/issues if available

4. Provides context:

   • Shows blame information for affected code
   • Lists related commits in the area
   • Identifies potential reviewers (commit authors)
   • Suggests rollback or fix strategies

5. Documents findings:

   • Creates regression analysis report
   • Links to issue tracker if available
   • Updates regression baseline with findings
   • Generates timeline of when regression was introduced

Bisect Configuration

Manual Bisect

bisect_manual:
  mode: interactive
  good_commit: v1.2.0  # Known good state
  bad_commit: HEAD     # Known bad state
  test_command: "npm test -- auth.test.ts"

  workflow:
    - git bisect start
    - git bisect bad HEAD
    - git bisect good v1.2.0
    - git bisect run npm test -- auth.test.ts

Automated Bisect

bisect_automated:
  mode: automated
  detect_good: auto  # Use last passing CI build
  detect_bad: auto   # Use current failing state
  test_command: "npm test"
  timeout: 600       # 10 minutes per commit test

  auto_detection:
    - check_ci_history
    - find_last_green_build
    - set_as_good_commit
    - use_current_as_bad

Bisect with Custom Test

bisect_custom:
  mode: custom
  test_script: scripts/regression-test.sh
  success_exit_code: 0
  failure_exit_code: 1

  test_script_template: |
    #!/bin/bash
    # Custom regression test
    npm run build || exit 1
    npm test -- specific-test.ts || exit 1
    ./validate-output.sh || exit 1
    exit 0

Bisect Report Format

# Regression Bisect Report

**Date**: 2026-01-28
**Issue**: Authentication fails on login
**Analyzer**: regression-bisect skill

## Executive Summary

**Breaking Commit**: abc123def456
**Author**: John Developer <john@example.com>
**Date**: 2026-01-15 14:32:00
**Bisect Duration**: 8 commits tested, 12 minutes

**Root Cause**: Token validation logic changed to require `iss` claim

## Bisect Results

### Commit Range

- **Good**: v1.2.0 (2026-01-10) - 50 commits ago
- **Bad**: HEAD (2026-01-28) - Current state
- **Breaking**: abc123def456 (2026-01-15) - 25 commits into range

### Bisect Log

| Commit | Status | Test Result | Duration |
|--------|--------|-------------|----------|
| abc123 | ❌ Bad | auth.test.ts failed | 45s |
| def789 | ❌ Bad | auth.test.ts failed | 42s |
| ghi012 | ✅ Good | all tests passed | 48s |
| jkl345 | ✅ Good | all tests passed | 46s |
| mno678 | ❌ Bad | auth.test.ts failed | 44s |
| **pqr901** | **❌ Breaking** | **First failure** | **43s** |
| stu234 | ✅ Good | all tests passed | 47s |
| vwx567 | ✅ Good | all tests passed | 45s |

## Breaking Commit Analysis

### Commit Details

commit pqr901xyz Author: John Developer john@example.com Date: Mon Jan 15 14:32:00 2026 +0000

feat(auth): add JWT issuer validation

- Require `iss` claim in JWT tokens
- Validate issuer against whitelist
- Update token generation to include issuer

Closes #456

### Files Changed

| File | Changes | Impact |
|------|---------|--------|
| src/auth/validate-token.ts | +15/-5 | High - Core validation logic |
| src/auth/generate-token.ts | +8/-2 | Medium - Token generation |
| test/auth/token.test.ts | +25/-0 | Low - Test updates |

### Diff Analysis

```diff
diff --git a/src/auth/validate-token.ts b/src/auth/validate-token.ts
index abc123..def456 100644
--- a/src/auth/validate-token.ts
+++ b/src/auth/validate-token.ts
@@ -15,6 +15,11 @@ export function validateToken(token: string): TokenPayload {
     throw new Error('Invalid token signature');
   }

+  // NEW: Require issuer claim
+  if (!payload.iss) {
+    throw new Error('Token missing issuer claim');
+  }
+
   return payload;
 }

Breaking Change: The new validation requires iss claim, but existing tokens don't include it.

Root Cause

The commit added mandatory iss (issuer) claim validation without:

1. Updating existing token generation (added in same commit but missed migration)
2. Providing backward compatibility for existing tokens
3. Coordinating with deployment to invalidate old sessions

Impact: All existing user sessions became invalid immediately.

Affected Components

• Authentication: Token validation fails for all existing sessions
• API Gateway: 401 errors on all authenticated endpoints
• User Experience: Force logout of all active users

Related Information

Traceability

• Issue: #456 - "Add JWT issuer validation for security"
• Requirement: @.aiwg/requirements/use-cases/UC-AUTH-003-jwt-security.md
• Review: PR #789 - Merged without integration testing

Similar Issues

Git history shows related changes:

• commit xyz789 (2026-01-10): Initial JWT implementation
• commit uvw456 (2026-01-12): Add JWT expiration validation
• commit pqr901 (2026-01-15): Add JWT issuer validation ← BREAKING

Code Ownership

src/auth/validate-token.ts:
  85% John Developer <john@example.com>
  10% Jane Reviewer <jane@example.com>
   5% Bob Maintainer <bob@example.com>

Recommendations

Immediate (Hotfix)

• Option A: Revert commit pqr901 to restore service
• Option B: Make iss validation optional with feature flag
• Option C: Deploy token refresh that includes iss claim

Recommended: Option B - Feature flag allows gradual rollout

Short-term (This Sprint)

• Add integration tests for auth changes
• Add backward compatibility tests for token validation
• Update deployment runbook for auth changes
• Add session migration logic for breaking changes

Ongoing

• Require integration tests for auth PRs
• Add "breaking change" checklist to PR template
• Schedule auth regression suite in CI
• Document token format in API docs

Fix Strategy

Proposed Fix

// src/auth/validate-token.ts
export function validateToken(token: string): TokenPayload {
  const payload = jwt.verify(token, SECRET_KEY);

  if (!payload.sub) {
    throw new Error('Invalid token signature');
  }

  // FIXED: Make issuer validation optional during migration
  if (featureFlags.requireIssuer && !payload.iss) {
    throw new Error('Token missing issuer claim');
  }

  return payload;
}

Migration Path

1. Deploy fix with requireIssuer: false
2. Update token generation to include iss
3. Wait for all old tokens to expire (24 hours)
4. Enable requireIssuer: true
5. Remove feature flag after validation

Timeline

2026-01-10: v1.2.0 released (working)
2026-01-15: Commit pqr901 merged (breaking change introduced)
2026-01-15: No CI failure (tests didn't catch integration issue)
2026-01-16: Deployed to production
2026-01-16: User reports started (authentication failures)
2026-01-28: Bisect identified root cause

Time to Detection: 13 days Time to Root Cause: Immediate (via bisect)

Prevention

This regression could have been prevented by:

1. Integration test covering token validation end-to-end
2. Staged rollout with feature flag
3. Backward compatibility review in PR checklist
4. Session invalidation notice to users

References

• Bisect documentation: docs/debugging/git-bisect-guide.md
• Breaking change policy: docs/development/breaking-changes.md
• Auth architecture: @.aiwg/architecture/auth-architecture.md

## Usage Examples

### Basic Regression Bisect

User: "Find which commit broke authentication"

Skill executes:

1. Identify current failure: auth tests failing
2. Find last known good: v1.2.0 (CI history)
3. Run git bisect with test: npm test -- auth.test.ts
4. Test 8 commits in 12 minutes
5. Identify breaking commit: pqr901

Output: "Regression Bisect Complete

Breaking Commit: pqr901xyz Author: John Developer Date: 2026-01-15 14:32:00 Message: feat(auth): add JWT issuer validation

Root Cause: Required iss claim without backward compatibility

Files Changed:

• src/auth/validate-token.ts (+15/-5)
• src/auth/generate-token.ts (+8/-2)

Recommendation: Make validation optional with feature flag

Full report: .aiwg/testing/regression-bisect-auth.md"

### Bisect with Custom Test

User: "Bisect the payment processing regression"

Skill asks: "What is a good commit (when it worked)?" User: "v2.1.0"

Skill executes:

1. git bisect start
2. git bisect bad HEAD
3. git bisect good v2.1.0
4. git bisect run scripts/payment-test.sh

Output: "Breaking Commit: def456 Broke payment refund logic Changed: src/payments/refund.ts

Bisected 45 commits in 32 minutes See .aiwg/testing/regression-bisect-payment.md"

### Automatic Good/Bad Detection

User: "When did the API start returning 500 errors?"

Skill analyzes:

• Current state: 500 errors on /api/users
• CI history: Last green build was 2 days ago
• Auto-detect good: commit xyz789 (last green)
• Auto-detect bad: HEAD (current failing)

Runs bisect automatically.

Output: "Regression introduced in commit abc123 2 days ago by Jane Developer Changed database query logic

Breaking change: Added JOIN without NULL check See full analysis in .aiwg/testing/regression-api-500.md"

## Integration

This skill uses:
- `regression-baseline`: Load known good states
- `project-awareness`: Detect test commands and CI configuration
- `test-coverage`: Identify relevant test suites
- Git tools: For bisect operations and history analysis

## Agent Orchestration

```yaml
agents:
  analysis:
    agent: debugger
    focus: Root cause analysis and blame

  testing:
    agent: test-engineer
    focus: Test execution and validation

  documentation:
    agent: technical-writer
    focus: Report generation

Configuration

Bisect Presets

bisect_presets:
  auth_regression:
    test_command: npm test -- auth.test.ts
    timeout: 300
    auto_detect_good: true

  api_regression:
    test_command: npm run test:api
    timeout: 600
    auto_detect_good: true

  performance_regression:
    test_command: npm run benchmark
    threshold: "response_time < 1000ms"
    timeout: 900

Timeout Strategy

timeout_config:
  per_commit_test: 600     # 10 minutes
  total_bisect: 3600       # 1 hour max
  on_timeout: skip_commit  # or fail_bisect

Error Handling

Unstable Tests

If test results are inconsistent:

1. Run test 3 times per commit
2. Require 2/3 passes to mark as good
3. Flag flaky tests in report
4. Recommend test stabilization

Build Failures

If commit doesn't build:

1. Skip commit and mark as bad
2. Continue bisect with next commit
3. Flag build issues in report
4. Note potential false positive

Timeout Exceeded

If bisect takes too long:

1. Stop bisect at timeout
2. Report progress so far
3. Suggest narrowing commit range
4. Offer to resume later

Output Locations

• Bisect reports: .aiwg/testing/regression-bisect-{issue}.md
• Bisect logs: .aiwg/testing/bisect-logs/
• Fix recommendations: .aiwg/working/regression-fixes/

References

• @$AIWG_ROOT/agentic/code/frameworks/sdlc-complete/schemas/test/regression-schema.yaml
• @$AIWG_ROOT/agentic/code/frameworks/sdlc-complete/commands/regression-test.md
• @$AIWG_ROOT/agentic/code/frameworks/sdlc-complete/agents/debugger.md