Plans Yankeeinlondon Adaptive Shell

Testing Framework and Planning Skill Enhancement

Date: 2025-10-21

Status: Planning

Overview

This plan addresses two interconnected needs for the ~/.config/sh repository:

1. Adapt the Planning Skill - The .claude/skills/planning/SKILL.md was ported from a TypeScript project and needs updates to work with this bash scripting repository.

2. Establish a Testing Framework - Current tests in tests/ rely heavily on manual visual inspection rather than boolean assertions. We need a robust, automated testing framework suitable for bash scripts.

Current State Analysis

Planning Skill Issues

The current planning skill has these considerations for our bash repo:

• References pnpm test and pnpm test:types (appropriate since we'll publish to npm)
• Discusses "type tests" which are primarily relevant to TypeScript
• References Vitest testing patterns (actually appropriate for our approach)
• Uses tests/unit/WIP/ directory structure
• File naming conventions like *.test.ts
• TypeScript-specific test examples

Current Testing Approach

We have three test files with varying quality:

1. tests/color.sh - Purely demonstrative, requires manual visual inspection
2. tests/lists.sh - Has boolean assertions with pass/fail tracking (good pattern)
3. tests/file-deps.sh - Most sophisticated, uses boolean checks and structured output

Problems:

• No unified test runner
• No standardized assertion library
• No aggregate pass/fail reporting
• No CI/CD integration capability
• Tests require manual execution and review

Proposed Testing Framework

Design Principles

1. Leverage TypeScript Expertise - Ken is a TypeScript expert, use Vitest
2. Boolean Assertions - All automated tests must have pass/fail checks
3. Hybrid Approach - TypeScript tests for automation, bash demos for visual testing
4. Structured Output - Support both human-readable and machine-parseable formats
5. Exit Codes - Proper exit codes for CI/CD integration
6. npm Publishing Ready - Aligned with eventual npm package distribution

Framework Choice: Vitest

Rationale:

• NPM Publishing - Repo will likely be published to npm, so pnpm is already required
• Expert Alignment - Ken is TypeScript expert, leverage existing expertise
• Superior Tooling - Watch mode, UI, coverage reports, parallel execution
• Bash Testing - Easy to test bash scripts via execSync() from TypeScript
• Best of Both Worlds - Keep visual bash demos, add robust automated tests

Hybrid Architecture

tests/
├── demos/              # Bash visual demos (manual testing)
│   ├── color-demo.sh   # Interactive color showcase
│   └── showcase.sh     # Feature demonstrations
├── helpers/            # TypeScript test utilities
│   └── bash.ts         # Helpers for testing bash from TS
├── lists.test.ts       # Automated Vitest tests
├── color.test.ts
├── file-deps.test.ts
└── WIP/                # In-progress tests (TDD workflow)

Plan Phases

Phase 0: Planning Skill Adaptation

Goals:

• Update planning skill for bash/TypeScript hybrid context
• Keep Vitest references (they're appropriate!)
• Add bash-specific testing patterns
• Preserve TDD methodology and phase-based approach

Tasks:

1. Update .claude/skills/planning/SKILL.md:

   • Keep TypeScript/Vitest examples (they're correct for our approach)
   • Add bash-specific testing patterns using bash helper
   • Update to reflect hybrid approach (TypeScript tests + bash source)
   • Clarify when to use demos vs automated tests
   • Keep start-position.ts reference (it works well)

2. Update test file naming conventions:

   • Automated tests: *.test.ts
   • Demo scripts: *-demo.sh or showcase-*.sh

3. Update directory structure recommendations:

   • tests/ for automated tests
   • tests/demos/ for visual demonstrations
   • tests/WIP/ for in-progress work
   • tests/helpers/ for TypeScript utilities

Deliverables:

• Updated .claude/skills/planning/SKILL.md
• Keep start-position.ts as-is (it's already TypeScript)

No TDD Required - This is documentation work

Duration: 1-2 hours

Phase 1: Vitest Setup & Bash Test Helpers

Goals:

• Set up Vitest testing infrastructure
• Create bash helper utilities for testing bash from TypeScript
• Establish test patterns and conventions

Tasks:

1. Package Setup:

   • Create package.json with vitest dependency
   • Create vitest.config.ts configuration
   • Create tsconfig.json for test files
   • Add test scripts to package.json

2. Bash Test Helper (tests/helpers/bash.ts):

   import { execSync } from 'child_process'
   
   export interface BashOptions {
     env?: Record<string, string>
     cwd?: string
   }
   
   /**
    * Execute bash script and return stdout
    */
   export function bash(script: string, options?: BashOptions): string {
     return execSync(script, {
       shell: '/bin/bash',
       encoding: 'utf-8',
       cwd: options?.cwd || process.cwd(),
       env: {
         ...process.env,
         ROOT: process.cwd(),
         ...options?.env
       }
     }).trim()
   }
   
   /**
    * Source a bash file and execute script
    */
   export function sourcedBash(file: string, script: string, options?: BashOptions): string {
     return bash(`source ${file} && ${script}`, options)
   }
   
   /**
    * Execute bash and return exit code
    */
   export function bashExitCode(script: string, options?: BashOptions): number {
     try {
       bash(script, options)
       return 0
     } catch (error: any) {
       return error.status || 1
     }
   }
   

3. Example Test Pattern Documentation:

   Create example in tests/helpers/example.test.ts:

   import { describe, it, expect } from 'vitest'
   import { sourcedBash, bashExitCode } from './bash'
   
   describe('example: testing bash functions', () => {
     it('should test bash function output', () => {
       const result = sourcedBash('./utils/lists.sh', `
         items=("apple" "banana" "cherry")
         retain_prefixes_ref items "a" "b"
       `)
   
       expect(result).toContain('apple')
       expect(result).toContain('banana')
       expect(result).not.toContain('cherry')
     })
   
     it('should test bash function exit codes', () => {
       const exitCode = bashExitCode(`
         source ./utils/lists.sh
         items=("apple" "banana")
         list_contains_ref items "banana"
       `)
   
       expect(exitCode).toBe(0)
     })
   })
   

TDD Approach:

1. Create tests/WIP/bash-helper.test.ts to test the bash helper itself
2. Write tests for basic bash execution
3. Implement bash helper to pass tests
4. Write tests for sourced bash execution
5. Implement sourcedBash to pass tests
6. Move from WIP to tests/helpers/bash.test.ts

Deliverables:

• package.json with vitest and scripts
• vitest.config.ts
• tsconfig.json
• tests/helpers/bash.ts
• tests/helpers/bash.test.ts (tests for the helper)
• tests/helpers/example.test.ts (documentation example)

Duration: 2-3 hours

Phase 2: Migrate Existing Tests to Vitest

Goals:

• Convert existing test files to Vitest
• Separate visual demos from automated tests
• Establish patterns for the rest of the codebase

Tasks:

1. Migrate tests/lists.sh → tests/lists.test.ts:

   • Convert all 8 tests to Vitest format
   • Use sourcedBash() helper
   • Keep all existing test logic and assertions
   • Remove manual pass/fail tracking (Vitest handles this)

2. Migrate tests/file-deps.sh → tests/file-deps.test.ts:

   • Keep JSON validation approach
   • Use Vitest's expect().toMatchObject() for JSON
   • Test both console and JSON output formats
   • Add tests for edge cases (empty dependencies, etc.)

3. Convert tests/color.sh:

   • Move to tests/demos/color-demo.sh (keep as visual demo)
   • Create tests/color.test.ts for automated tests:
     • Test colorize() function output structure
     • Test rgb_text() with known RGB values
     • Test color shortcut functions return non-empty strings
     • Test color variables are set correctly by setup_colors()

TDD Approach:

For each test file:

1. Create new test in tests/WIP/MODULE.test.ts
2. Write tests using Vitest and bash helper
3. Verify tests pass against existing functionality
4. If tests fail, determine if bug in source or test
5. Remove old bash test file (or move to demos/)
6. Move new test from WIP to tests/

Deliverables:

• tests/lists.test.ts
• tests/file-deps.test.ts
• tests/color.test.ts
• tests/demos/color-demo.sh (visual demo)

Duration: 2-3 hours

Phase 3: Core Utilities Test Coverage

Goals:

• Add comprehensive test coverage for critical utility functions
• Focus on most-used functions first
• Aim for 80%+ coverage of exported functions
• Fix any bugs discovered during testing

Priority Functions to Test:

From utils/ directory:

1. utils/text.sh - String manipulation functions (trim, uppercase, lowercase, etc.)
2. utils/typeof.sh - Type checking utilities (is_function, is_number, etc.)
3. utils/filesystem.sh - File operations (file_exists, dir_exists, etc.)
4. utils/empty.sh - Empty/null checking (is_empty, not_empty, etc.)
5. utils/detection.sh - System detection (get_os, get_shell, etc.)
6. utils/errors.sh - Error handling (panic, error, warn, etc.)

TDD Approach:

For each utility module:

1. Create tests/WIP/MODULE.test.ts
2. List all exported functions in the module
3. Write tests for each function:
   • Happy path (normal usage)
   • Edge cases (empty strings, null, undefined, boundaries)
   • Error cases (invalid input)
4. Run tests - many will fail initially (discovering bugs or behavior issues)
5. Fix bugs in source code OR clarify test expectations
6. Ensure all tests pass
7. Migrate test from WIP to tests/MODULE.test.ts

Example Test Structure:

import { describe, it, expect } from 'vitest'
import { sourcedBash, bashExitCode } from './helpers/bash'

describe('text utilities', () => {
  describe('trim()', () => {
    it('should remove leading and trailing whitespace', () => {
      const result = sourcedBash('./utils/text.sh', `
        trim "  hello world  "
      `)
      expect(result).toBe('hello world')
    })

    it('should handle empty string', () => {
      const result = sourcedBash('./utils/text.sh', `trim ""`)
      expect(result).toBe('')
    })

    it('should handle string with only whitespace', () => {
      const result = sourcedBash('./utils/text.sh', `trim "   "`)
      expect(result).toBe('')
    })
  })

  describe('uppercase()', () => {
    it('should convert string to uppercase', () => {
      const result = sourcedBash('./utils/text.sh', `uppercase "hello"`)
      expect(result).toBe('HELLO')
    })

    it('should handle mixed case', () => {
      const result = sourcedBash('./utils/text.sh', `uppercase "HeLLo"`)
      expect(result).toBe('HELLO')
    })
  })
})

Deliverables:

• tests/text.test.ts
• tests/typeof.test.ts
• tests/filesystem.test.ts
• tests/empty.test.ts
• tests/detection.test.ts
• tests/errors.test.ts
• Bug fixes discovered during testing
• Documentation updates if function behavior is clarified

Duration: 4-6 hours

Phase 4: Documentation and CI Integration

Goals:

• Document testing conventions and best practices
• Add CI/CD integration with GitHub Actions
• Create testing guidelines for contributors
• Set up useful npm scripts

Tasks:

1. Create tests/README.md:

   • How to run tests (pnpm test)
   • How to write tests using bash helper
   • Vitest commands reference:
     • pnpm test - run all tests once
     • pnpm test:watch - run in watch mode
     • pnpm test:ui - open Vitest UI
     • pnpm test:coverage - generate coverage report
   • Test writing patterns and examples
   • When to use demos vs automated tests
   • Best practices for testing bash from TypeScript

2. Update CLAUDE.md:

   • Add comprehensive testing section
   • Link to tests/README.md
   • Document TDD expectations
   • Include example of testing bash functions
   • Explain hybrid approach (demos + automated tests)

3. Package.json scripts:

   {
     "scripts": {
       "test": "vitest run",
       "test:watch": "vitest",
       "test:ui": "vitest --ui",
       "test:coverage": "vitest run --coverage"
     },
     "devDependencies": {
       "vitest": "^latest",
       "@vitest/ui": "^latest",
       "@vitest/coverage-v8": "^latest",
       "typescript": "^latest"
     }
   }
   

4. Create .github/workflows/test.yml:

   name: Tests
   
   on:
     push:
       branches: [ main ]
     pull_request:
       branches: [ main ]
   
   jobs:
     test:
       runs-on: ubuntu-latest
   
       steps:
         - uses: actions/checkout@v3
   
         - uses: pnpm/action-setup@v2
           with:
             version: 8
   
         - uses: actions/setup-node@v3
           with:
             node-version: '20'
             cache: 'pnpm'
   
         - run: pnpm install
   
         - run: pnpm test
   

5. Add pre-commit hook example (.git/hooks/pre-commit.example):

   #!/bin/bash
   # Run tests before committing
   pnpm test || exit 1
   

Deliverables:

• tests/README.md
• Updated CLAUDE.md
• Updated package.json with scripts
• .github/workflows/test.yml
• .git/hooks/pre-commit.example

Duration: 1-2 hours

Testing Standards

Test File Conventions

• Automated tests: *.test.ts in tests/ directory
• Demo scripts: *-demo.sh or showcase-*.sh in tests/demos/
• Helper utilities: *.ts in tests/helpers/
• Work in progress: Use tests/WIP/ during development

Test Organization

tests/
├── demos/                 # Visual demonstration scripts
│   ├── color-demo.sh     # Interactive color showcase
│   └── showcase.sh       # General feature demonstrations
├── helpers/              # TypeScript test utilities
│   ├── bash.ts          # Bash execution helpers
│   ├── bash.test.ts     # Tests for bash helper
│   └── example.test.ts  # Example/documentation
├── WIP/                  # Work in progress (during TDD)
├── *.test.ts            # Automated test files
└── README.md            # Testing documentation

Writing Tests

Principles:

1. One behavior per test - Each it() block tests a single thing
2. Descriptive names - Test names should explain what's being verified
3. Arrange-Act-Assert - Clear setup, execution, and verification
4. Independence - Tests should not depend on each other
5. Cleanup - Clean up any temporary files/state (use beforeEach/afterEach)

Example Pattern:

import { describe, it, expect } from 'vitest'
import { sourcedBash } from './helpers/bash'

describe('function_name', () => {
  it('should handle empty input gracefully', () => {
    // Arrange
    const input = ""

    // Act
    const result = sourcedBash('./utils/module.sh', `
      function_name "${input}"
    `)

    // Assert
    expect(result).toBe('default_value')
  })

  it('should process normal input correctly', () => {
    const result = sourcedBash('./utils/module.sh', `
      function_name "test input"
    `)

    expect(result).toContain('test input')
  })
})

Running Tests

# Install dependencies (first time)
pnpm install

# Run all tests
pnpm test

# Run tests in watch mode (auto-rerun on file changes)
pnpm test:watch

# Run tests with UI
pnpm test:ui

# Run tests with coverage report
pnpm test:coverage

# Run specific test file
pnpm test lists

# Run tests matching pattern
pnpm test --grep "retain_prefixes"

When to Use Demos vs Automated Tests

Use Demo Scripts (tests/demos/) when:

• Visual inspection is required (e.g., color output)
• Interactive demonstration of features
• Showcasing functionality to users
• Manual testing during development

Use Automated Tests (tests/*.test.ts) when:

• Boolean pass/fail verification is possible
• Testing function behavior and logic
• CI/CD integration needed
• Regression testing required

Open Questions

1. Should we keep start-position.ts or rewrite in bash?

   • Decision: Keep TypeScript - it works well and we're using TypeScript for tests anyway

2. Should tests be in tests/ or co-located with source?

   • Decision: Keep centralized in tests/ - cleaner separation for bash source files

3. Do we need a WIP directory?

   • Decision: Yes, essential for TDD workflow without breaking main test suite

4. Coverage threshold?

   • Recommendation: Start with 80% for core utils, can adjust based on what makes sense

Success Criteria

Phase 0 (Planning Skill)

• Planning skill updated for bash/TypeScript hybrid approach
• TypeScript/Vitest references kept and clarified
• Bash-specific testing patterns added
• Demo vs automated test guidance provided

Phase 1 (Vitest Setup)

• package.json created with vitest dependencies
• vitest.config.ts and tsconfig.json configured
• Bash helper implemented and tested
• Example tests created as documentation
• All tests pass

Phase 2 (Existing Tests)

• lists.test.ts migrated with all tests passing
• file-deps.test.ts migrated with all tests passing
• color.test.ts created for automated tests
• color-demo.sh moved to demos/ directory
• All tests passing, zero regressions

Phase 3 (Coverage)

• Tests exist for 6 core utility modules
• Each exported function has at least one test
• 80%+ coverage of critical functions
• Any discovered bugs fixed
• All tests passing

Phase 4 (Documentation)

• tests/README.md comprehensive and clear
• CLAUDE.md includes testing guidance
• GitHub Actions workflow functional
• Pre-commit hook example provided
• All npm scripts working correctly

Timeline Estimate

• Phase 0: 1-2 hours (documentation)
• Phase 1: 2-3 hours (Vitest setup and bash helper)
• Phase 2: 2-3 hours (migrate existing tests)
• Phase 3: 4-6 hours (new test coverage)
• Phase 4: 1-2 hours (documentation and CI)

Total: 10-16 hours of focused work

Next Steps

After approval of this plan:

1. Begin Phase 0 - Update planning skill documentation
2. Execute Phase 1 - Set up Vitest and create bash helper (with TDD)
3. Checkpoint after Phase 1 to review test patterns
4. Execute Phase 2 - Migrate existing tests
5. Execute Phase 3 - Add coverage for core utilities
6. Execute Phase 4 - Documentation and CI integration

Benefits of This Approach

1. Leverages Expertise - Uses TypeScript, which Ken knows well
2. NPM Ready - Aligns with npm publishing plans
3. Superior DX - Watch mode, UI, coverage built-in
4. Best of Both - Visual demos + robust automation
5. CI/CD Ready - Proper exit codes, JSON output, GitHub Actions
6. Maintainable - Industry-standard tooling, familiar to contributors