Run Simulations

Run Simulations Skill

You are helping set up and run trading engine simulations, then performing comprehensive diagnostic analysis.

Current Phase: DUAL-FOCUS (Validation + Optimization)

System confidence: ~85% accurate. The simulation system is largely trustworthy, but discrepancies still exist and are critical to find.

Priority #1: ACCURACY DISCREPANCIES (Most Important)

Any gap between expected vs actual behavior is a potential bug or misunderstanding. These must be surfaced prominently:

• Does behavior match config parameters?
• Are there logical inconsistencies in the data?
• Do results align with what the strategy SHOULD produce?

When you spot a discrepancy, flag it prominently. Even small accuracy issues compound over thousands of trades.

Priority #2: PROFIT OPTIMIZATION (Active Research)

With a trustworthy system, we're now actively researching:

• Which configs perform best?
• What market conditions favor which strategies?
• How do parameters affect PnL?

Surface interesting profit patterns AND accuracy concerns. Both matter now.

IMPORTANT: This skill should be invoked BEFORE running simulations to help with setup decisions (config selection, segment selection via collections, batch sizing).

Recommended Batch Size

Current recommendation: 2,000-3,000 simulations per batch.

• Runs quickly (~30-60 seconds with tick caching)
• Provides statistically meaningful results
• More than 3,000 doesn't add much value at current stage
• Formula: configs × segments = total runs (e.g., 20 configs × 100 segments = 2,000 runs)

Phase 1: Setup & Run

⚠️ USER INPUT TAKES PRIORITY

If the user provides ANY arguments, instructions, or context when invoking this skill, those take absolute priority over all defaults below.

• User says "run config X" → run config X, ignore default config selection
• User says "test these 5 segments" → test those 5 segments, ignore default segment query
• User asks a specific question → answer that question, don't run the full default workflow
• User provides partial instructions → fill in gaps with defaults, but honor what they specified

The defaults below are ONLY for when the skill is invoked with no arguments at all.

Default Behavior (no args only)

Pick a diverse spread of configs and segments using segment collections (legacy filters are deprecated):

1. Query available configs: SELECT name, config FROM strategy_configs ORDER BY name
2. Select 10-20 configs covering different indicator types (EMA, RSI, Bollinger, combined)
3. Choose or create a segment collection (AND-only annotations in v1):

   bun src/systems/trading-engine/bd-segments.ts list
   bun src/systems/trading-engine/bd-segments.ts create --name=high-activity --annotations=high_activity --limit=100
   bun src/systems/trading-engine/bd-segments.ts preview --name=high-activity
   

4. Run batch simulation:

   bun src/systems/trading-engine/batch-runner.ts --config-pattern="<pattern>" --collection=high-activity --save
   

Batch Runner CLI Reference

Location: src/systems/trading-engine/batch-runner.ts

Usage:

bun src/systems/trading-engine/batch-runner.ts [options]

Options:

Examples:

# Run one config on a segment collection
bun src/systems/trading-engine/bd-segments.ts create --name=swings100 --annotations=high_activity
bun src/systems/trading-engine/batch-runner.ts --config=abc123 --collection=swings100 --save

# Run all configs on a collection
bun src/systems/trading-engine/batch-runner.ts --all-configs --collection=swings50 --save

# Run all configs on specific segment IDs (snapshot collection)
bun src/systems/trading-engine/bd-segments.ts create --name=segment-set --segment-ids=949,950,951,952 --mode=snapshot
bun src/systems/trading-engine/batch-runner.ts --all-configs --collection=segment-set --save

# Run configs matching "RSI-*" on high_activity segments
bun src/systems/trading-engine/bd-segments.ts create --name=rsi-activity --annotations=high_activity
bun src/systems/trading-engine/batch-runner.ts --config-pattern="RSI-%" --collection=rsi-activity --save

# Dry run to see what would execute
bun src/systems/trading-engine/batch-runner.ts --all-configs --collection=swings100 --dry-run

Key CLI Flags (Quick Reference)

• --collection=NAME|ID or -C: REQUIRED segment collection selector
• --show-selection: Preview resolved segment list and exit
• --config-pattern="TEST-%" or -p: Filter configs by name pattern
• --strategy-type=grid: Filter configs by strategy type

Phase 2: Comprehensive Analysis (ALWAYS RUN)

After simulations complete, run ALL of the following diagnostic queries. Present results in tables.

2.1 Core Stats

SELECT
  sc.name,
  COUNT(*) as runs,
  ROUND(AVG(trade_count)::numeric, 1) as avg_trades,
  ROUND(STDDEV(trade_count)::numeric, 1) as stddev_trades,
  MIN(trade_count) as min_trades,
  MAX(trade_count) as max_trades
FROM (
  SELECT r.id, sc.name, COUNT(t.id) as trade_count
  FROM sim_runs r
  JOIN strategy_configs sc ON r.config_id = sc.id
  LEFT JOIN sim_trades t ON t.run_id = r.id
  WHERE r.created_at > NOW() - INTERVAL '1 hour'
  GROUP BY r.id, sc.name
) sub
JOIN strategy_configs sc ON sc.name = sub.name
GROUP BY sc.name
ORDER BY avg_trades;

2.2 Entry Signal Frequency (RED FLAG: median < 60s is suspicious)

WITH entries AS (
  SELECT
    t.run_id, sc.name as config, t.entry_at,
    LAG(t.entry_at) OVER (PARTITION BY t.run_id ORDER BY t.entry_at) as prev_entry
  FROM sim_trades t
  JOIN sim_runs r ON t.run_id = r.id
  JOIN strategy_configs sc ON r.config_id = sc.id
  WHERE r.created_at > NOW() - INTERVAL '1 hour'
)
SELECT
  config,
  COUNT(*) as entry_gaps,
  ROUND(AVG((entry_at - prev_entry)/1000.0)::numeric, 1) as avg_gap_sec,
  ROUND(PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY (entry_at - prev_entry)/1000.0)::numeric, 1) as median_gap_sec,
  COUNT(*) FILTER (WHERE entry_at - prev_entry < 1000) as gaps_under_1s,
  COUNT(*) FILTER (WHERE entry_at - prev_entry < 5000) as gaps_under_5s,
  COUNT(*) FILTER (WHERE entry_at - prev_entry < 60000) as gaps_under_1min
FROM entries
WHERE prev_entry IS NOT NULL
GROUP BY config
ORDER BY median_gap_sec;

Interpretation: Median gaps < 60s suggest overtrading. Healthy configs should have gaps in minutes (20-30min typical). Gaps < 1s are a MAJOR RED FLAG - indicates indicator firing constantly.

2.3 Position Overlap

-- How many concurrent positions on average?
WITH trade_events AS (
  SELECT t.run_id, sc.name as config, t.entry_at as ts, 1 as delta
  FROM sim_trades t
  JOIN sim_runs r ON t.run_id = r.id
  JOIN strategy_configs sc ON r.config_id = sc.id
  WHERE r.created_at > NOW() - INTERVAL '1 hour'
  UNION ALL
  SELECT t.run_id, sc.name as config, t.exit_at as ts, -1 as delta
  FROM sim_trades t
  JOIN sim_runs r ON t.run_id = r.id
  JOIN strategy_configs sc ON r.config_id = sc.id
  WHERE r.created_at > NOW() - INTERVAL '1 hour' AND t.exit_at IS NOT NULL
),
running_pos AS (
  SELECT run_id, config, ts,
    SUM(delta) OVER (PARTITION BY run_id ORDER BY ts, delta DESC) as open_positions
  FROM trade_events
)
SELECT
  config,
  MAX(open_positions) as max_concurrent,
  ROUND(AVG(open_positions)::numeric, 2) as avg_concurrent,
  PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY open_positions) as median_concurrent
FROM running_pos
GROUP BY config
ORDER BY avg_concurrent DESC;

2.4 Holding Duration Distribution

SELECT
  sc.name as config,
  CASE
    WHEN t.holding_duration_ms < 30000 THEN '<30s'
    WHEN t.holding_duration_ms < 60000 THEN '30-60s'
    WHEN t.holding_duration_ms < 120000 THEN '1-2min'
    WHEN t.holding_duration_ms < 180000 THEN '2-3min'
    WHEN t.holding_duration_ms < 240000 THEN '3-4min'
    WHEN t.holding_duration_ms < 300000 THEN '4-5min'
    ELSE '5min+'
  END as hold_bucket,
  COUNT(*) as trades,
  ROUND(AVG(t.net_pnl_cents)::numeric, 1) as avg_pnl,
  ROUND(COUNT(*)::numeric / SUM(COUNT(*)) OVER (PARTITION BY sc.name) * 100, 1) as pct
FROM sim_trades t
JOIN sim_runs r ON t.run_id = r.id
JOIN strategy_configs sc ON r.config_id = sc.id
WHERE r.created_at > NOW() - INTERVAL '1 hour' AND t.holding_duration_ms IS NOT NULL
GROUP BY sc.name, hold_bucket
ORDER BY sc.name, MIN(t.holding_duration_ms);

2.5 Exit Reason Analysis

SELECT
  sc.name as config,
  t.exit_reason,
  COUNT(*) as trades,
  ROUND(AVG(t.net_pnl_cents)::numeric, 1) as avg_pnl,
  ROUND(COUNT(*)::numeric / SUM(COUNT(*)) OVER (PARTITION BY sc.name) * 100, 1) as pct
FROM sim_trades t
JOIN sim_runs r ON t.run_id = r.id
JOIN strategy_configs sc ON r.config_id = sc.id
WHERE r.created_at > NOW() - INTERVAL '1 hour'
GROUP BY sc.name, t.exit_reason
ORDER BY sc.name, trades DESC;

2.6 YES vs NO Side Analysis (IMPORTANT - check for asymmetries)

SELECT
  sc.name as config,
  t.side,
  COUNT(*) as trades,
  ROUND(AVG(t.net_pnl_cents)::numeric, 1) as avg_pnl,
  ROUND(AVG(CASE WHEN t.net_pnl_cents > 0 THEN 1 ELSE 0 END)::numeric * 100, 1) as win_pct,
  ROUND(AVG(t.entry_price)::numeric, 1) as avg_entry_price,
  ROUND(AVG(t.exit_price)::numeric, 1) as avg_exit_price,
  ROUND(AVG(t.quantity)::numeric, 1) as avg_qty
FROM sim_trades t
JOIN sim_runs r ON t.run_id = r.id
JOIN strategy_configs sc ON r.config_id = sc.id
WHERE r.created_at > NOW() - INTERVAL '1 hour'
GROUP BY sc.name, t.side
ORDER BY sc.name, t.side;

Interpretation: Significant asymmetry between YES/NO performance suggests bugs in price conversion or side selection logic. Both sides should have similar characteristics unless the strategy explicitly favors one.

2.7 Position Sizing Verification

-- Check if fills match expected sizes and align with order book
SELECT
  sc.name as config,
  t.quantity as fill_qty,
  COUNT(*) as occurrences,
  ROUND(AVG(t.entry_price)::numeric, 1) as avg_entry_price,
  ROUND(AVG(t.net_pnl_cents)::numeric, 1) as avg_pnl
FROM sim_trades t
JOIN sim_runs r ON t.run_id = r.id
JOIN strategy_configs sc ON r.config_id = sc.id
WHERE r.created_at > NOW() - INTERVAL '1 hour'
GROUP BY sc.name, t.quantity
ORDER BY sc.name, t.quantity;

Check: Does quantity match config's maxTradeSize? Are partial fills happening? Compare with maxPositionPerMarket.

2.8 Trade Sequence Performance

WITH numbered_trades AS (
  SELECT t.*, sc.name as config,
    ROW_NUMBER() OVER (PARTITION BY t.run_id ORDER BY t.entry_at) as trade_num
  FROM sim_trades t
  JOIN sim_runs r ON t.run_id = r.id
  JOIN strategy_configs sc ON r.config_id = sc.id
  WHERE r.created_at > NOW() - INTERVAL '1 hour'
)
SELECT
  config,
  CASE
    WHEN trade_num = 1 THEN '1st'
    WHEN trade_num BETWEEN 2 AND 5 THEN '2-5th'
    WHEN trade_num BETWEEN 6 AND 10 THEN '6-10th'
    WHEN trade_num BETWEEN 11 AND 20 THEN '11-20th'
    ELSE '21st+'
  END as position,
  COUNT(*) as trades,
  ROUND(AVG(net_pnl_cents)::numeric, 1) as avg_pnl,
  ROUND(AVG(CASE WHEN net_pnl_cents > 0 THEN 1 ELSE 0 END)::numeric * 100, 1) as win_pct
FROM numbered_trades
GROUP BY config, position
ORDER BY config, MIN(trade_num);

2.9 Market Segment Analysis

SELECT
  ms.definition_name,
  COUNT(DISTINCT r.id) as runs,
  COUNT(t.id) as trades,
  ROUND(AVG(r.total_pnl_cents)::numeric, 1) as avg_run_pnl,
  ROUND(AVG((ms.metrics->>'swing_count')::int)::numeric, 1) as avg_swings,
  ROUND(AVG((ms.metrics->>'reversion_rate')::float)::numeric, 2) as avg_reversion_rate
FROM sim_runs r
JOIN market_segments ms ON ms.source_meta->>'legacy_test_case_id' = r.test_case_id::text
LEFT JOIN sim_trades t ON t.run_id = r.id
JOIN strategy_configs sc ON r.config_id = sc.id
WHERE r.created_at > NOW() - INTERVAL '1 hour'
GROUP BY ms.definition_name
ORDER BY runs DESC;

2.10 Run Health Check

SELECT status, COUNT(*) as count
FROM sim_runs
WHERE created_at > NOW() - INTERVAL '1 hour'
GROUP BY status;

RED FLAG: Any "running" status after batch completes = crashed simulations.

Phase 3: Log Analysis (ALWAYS RUN)

Check recent simulation logs for anomalies:

# Find most recent sim log
ls -lt logs/sim/ | head -5

# Sample entries from most recent log with content
LOG=$(ls -ltS logs/sim/*.log | head -1 | awk '{print $NF}')
echo "Analyzing: $LOG ($(du -h "$LOG" | cut -f1))"

# Check log volume by category (PERFORMANCE RED FLAG: >1M indicator logs)
echo "Log volume by category:"
rg '"category"' "$LOG" | jq -r '.category' 2>/dev/null | sort | uniq -c | sort -rn

# Check run duration from timestamps
echo "Run duration:"
FIRST_TS=$(head -1 "$LOG" | jq -r '.time' 2>/dev/null)
LAST_TS=$(tail -1 "$LOG" | jq -r '.time' 2>/dev/null)
if [ "$FIRST_TS" != "null" ] && [ "$LAST_TS" != "null" ]; then
  echo "Duration: $(( (LAST_TS - FIRST_TS) / 1000 )) seconds"
fi

# Count exit reasons
echo "Exit reason distribution:"
rg '"reason"' "$LOG" | jq -r '.reason' 2>/dev/null | sort | uniq -c | sort -rn | head -20

# Check for errors
echo "Errors in log:"
rg -i '"level":50' "$LOG" | head -10

# Check for warnings
echo "Warnings in log:"
rg -i '"level":40' "$LOG" | head -10

Look for:

• Unusual exit reasons (not profit_target, time_limit, collapse, end_of_data)
• Error level (50) entries
• Patterns suggesting bugs (repeated identical entries, missing fields)
• PERFORMANCE: Log file > 500MB = excessive logging
• PERFORMANCE: >1M indicator category logs = debug logging bottleneck

Phase 3.5: Performance Analysis

Check simulation throughput:

-- Runs per second over recent batch
SELECT
  DATE_TRUNC('minute', created_at) as minute,
  COUNT(*) as runs,
  ROUND(COUNT(*)::numeric / 60, 1) as runs_per_sec
FROM sim_runs
WHERE created_at > NOW() - INTERVAL '1 hour'
GROUP BY minute
ORDER BY minute DESC
LIMIT 10;

Expected throughput: 20-40 runs/sec with tick caching. If < 10 runs/sec, investigate:

• Is tick cache being used? (check for "Pre-loaded" in batch script output)
• Is indicator debug logging enabled? (check log size)
• Are workers I/O bound? (check worker count vs CPU cores)

Phase 4: Analysis Summary

DUAL FOCUS: Surface both accuracy concerns AND profit insights.

First: Accuracy Check (Priority #1)

Flag any discrepancies between expected vs actual behavior:

1. Does config behavior match parameters?
2. Are there unexplained patterns or anomalies?
3. Do metrics make logical sense?

Accuracy issues take precedence. A 1% accuracy bug can invalidate profit analysis.

Then: Profit Insights (Priority #2)

With accuracy validated, analyze:

1. Which configs are most profitable?
2. What patterns emerge across market conditions?
3. Are there optimization opportunities?

Validation Questions to Answer

For each finding, ask: "Does this match what we'd expect? If not, why?"

Key Validation Checks

1. Indicator Firing Frequency

   • Does the gap between entries match what the indicator period implies?
   • EMA(20) with 60s candles = 20 min of data needed. Are we seeing entries faster than that makes sense?

2. Price/Side Consistency

   • YES and NO should be mirror images. Major asymmetry = likely bug in price handling.
   • Check: Are we converting bid/ask correctly for each side?

3. Config Parameters Actually Applied

   • Is stopLossCents actually triggering at that price?
   • Is profitTargetCents actually triggering?
   • Is maxSpreadToEnter being checked?

4. Data Flow Integrity

   • Are ticks reaching the strategy in order?
   • Is depth data being parsed correctly?
   • Are fills being recorded accurately?

Phase 5: Findings & Next Steps

Based on analysis, organize findings into:

Accuracy Concerns (Surface First)

1. Likely Bugs - Discrepancies that suggest code errors
   • "X should be Y but we see Z" → investigate code path
2. Unclear Behavior - Things we don't understand yet
   • "We expected X, got Y, not sure why" → needs deeper investigation
3. Confirmed Working - Things that match expectations
   • "This behaves as expected" → increased confidence

Profit Insights (Surface Second)

1. Top Performers - Which configs show best results?
2. Market Patterns - What conditions favor which strategies?
3. Optimization Opportunities - Parameter tuning suggestions

Next Steps

• Investigation priorities for accuracy concerns
• Experiments to validate profit hypotheses
• Config variations to test

Remember: Accuracy comes first. Surface discrepancies prominently, then provide profit optimization insights. Both matter, but a small accuracy bug can invalidate all profit analysis.