Development Tools

Development Tools

Development commands for the multigres project.

Command Structure

/mt-dev [test-type] [args...]

For Claude Code

When executing commands:

• Always run make build before integration tests
• Port pool: Before integration tests, run scripts/portpool.sh start (idempotent, safe to call every time). Then prefix the go test command with MULTIGRES_PORT_POOL_ADDR=/tmp/multigres-port-pool.sock to coordinate port allocation and avoid flaky collisions. If the socket already exists (e.g. from a previous session), just set the env var — no need to restart.
• Show the actual command being executed before running it
• Summarize test results (passed/failed counts, execution time)
• If tests fail, offer to show detailed output or logs
• For verbose output, provide a summary rather than dumping everything

Unit Tests

Unit tests are fast, isolated tests for individual functions and packages. They don't require external services or build artifacts. Note: plain go test ./go/... will also traverse go/test/endtoend/...; use -short for unit-focused runs.

Command Syntax

Run all unit tests:

/mt-dev unit all

Executes: go test -short ./go/...

Run specific package:

/mt-dev unit <package-path>

Examples:

• /mt-dev unit ./go/services/multipooler/... - All multipooler package tests
• /mt-dev unit ./go/multigateway/... - All multigateway package tests
• /mt-dev unit ./go/pgprotocol/... - All pgprotocol package tests

Run specific test:

/mt-dev unit <package-path> <TestName>

Examples:

• /mt-dev unit ./go/services/multipooler TestConnectionPool
• /mt-dev unit ./go/pgprotocol TestParseQuery

Run with pattern matching:

/mt-dev unit <package-path> <TestPattern>

Examples:

• /mt-dev unit ./go/services/multipooler TestConn.* - All tests starting with TestConn
• /mt-dev unit ./go/multigateway Test.*Route.* - All tests with "Route" in name

Common Flags

• -v - Verbose output (shows all test names as they run)
• -race - Enable race detector (slower, catches concurrency bugs)
• -cover - Show coverage percentage
• -coverprofile=coverage.out - Generate coverage report
• -count=N - Run tests N times (useful for flaky test detection); -count=1 also forces re-run and bypasses test cache
• -timeout=30s - Set timeout (default: 10m)
• -short - Skip long-running tests
• -parallel=N - Run N tests in parallel (default: GOMAXPROCS)

Examples

# Quick test run
/mt-dev unit all

# Verbose with race detection
/mt-dev unit ./go/services/multipooler/... -v -race

# Coverage report
/mt-dev unit ./go/pgprotocol/... -cover

# Test for flakiness
/mt-dev unit ./go/multigateway TestRouting -count=10

# Fast tests only
/mt-dev unit all -short

# Specific test with verbose output
/mt-dev unit ./go/services/multipooler TestConnectionPool -v

Natural Language Support

• "run unit tests" → /mt-dev unit all
• "test the multipooler package" → /mt-dev unit ./go/services/multipooler/...
• "run TestConnectionPool" → /mt-dev unit ./go/services/multipooler TestConnectionPool
• "run all unit tests with coverage" → /mt-dev unit all -cover
• "check for race conditions in multigateway" → /mt-dev unit ./go/multigateway/... -race

Integration Tests

Integration tests are end-to-end tests that start real components (Multigateway, Multipooler, PostgreSQL) and test their interactions. These tests are slower and require building the project first.

IMPORTANT: Integration tests always run make build first.

Available Test Packages

• all - Run all integration tests
• multipooler - Connection pooling, pool lifecycle, connection management
• multiorch - Orchestration, failover, leader election, consensus protocol
• queryserving - Query routing, execution, transaction handling
• localprovisioner - Local cluster provisioning and setup
• shardsetup - Shard configuration and management
• pgregresstest - PostgreSQL regression tests (opt-in, comprehensive)

Integration Test Command Syntax

Run all integration tests:

/mt-dev integration all

Executes: make build && go test ./go/test/endtoend/...

Run specific package:

/mt-dev integration <package-name>

Examples:

• /mt-dev integration multipooler → make build && go test ./go/test/endtoend/multipooler/...
• /mt-dev integration multiorch → make build && go test ./go/test/endtoend/multiorch/...
• /mt-dev integration queryserving → make build && go test ./go/test/endtoend/queryserving/...

Run specific test:

/mt-dev integration <package-name> <TestName>

Examples:

• /mt-dev integration multiorch TestFixReplication → make build && go test -run TestFixReplication ./go/test/endtoend/multiorch/...
• /mt-dev integration multipooler TestConnCache → make build && go test -run TestConnCache ./go/test/endtoend/multipooler/...

Run specific test in all packages:

/mt-dev integration all <TestName>

Example:

• /mt-dev integration all TestBootstrap → make build && go test -run TestBootstrap ./go/test/endtoend/...

Run with pattern matching:

/mt-dev integration <package-name> <TestPattern>

Examples:

• /mt-dev integration queryserving Test.*Transaction.* - All transaction tests
• /mt-dev integration multipooler TestConn.* - All connection tests

Integration Test Flags

Same flags as unit tests, plus:

• -timeout=30m - Integration tests often need longer timeouts (default: 10m)
• -p=1 - Run packages sequentially (useful if tests conflict on resources)
• -count=N - Run tests N times (useful to detect flakes); -count=1 also forces re-run and bypasses test cache

Integration Test Examples

# Run all integration tests
/mt-dev integration all

# Test multipooler with verbose output
/mt-dev integration multipooler -v

# Test specific failure scenario
/mt-dev integration multiorch TestFixReplication

# Check for race conditions in query serving
/mt-dev integration queryserving -race

# Test for flakiness (run 10 times)
/mt-dev integration multipooler TestConnCache -count=10

# Run with extended timeout
/mt-dev integration all -timeout=45m

# Sequential execution to avoid resource conflicts
/mt-dev integration all -p=1

Integration Test Natural Language

• "run integration tests" → /mt-dev integration all
• "run multipooler tests" → /mt-dev integration multipooler
• "test multiorch TestFixReplication" → /mt-dev integration multiorch TestFixReplication
• "run all integration tests with race detector" → /mt-dev integration all -race
• "test query serving" → /mt-dev integration queryserving

Interpreting Test Results

Success Output

PASS
ok      github.com/multigres/multigres/go/services/multipooler    2.456s

• All tests passed
• Shows package path and execution time

Failure Output

--- FAIL: TestConnectionPool (0.15s)
    pool_test.go:45: expected 10 connections, got 8
FAIL
FAIL    github.com/multigres/multigres/go/services/multipooler    2.456s

• Shows which test failed
• Shows file, line number, and failure message
• Claude should summarize: "TestConnectionPool failed in pool_test.go:45"

Build Failure

# github.com/multigres/multigres/go/services/multipooler
./connection.go:123:45: undefined: somethingMissing
FAIL    github.com/multigres/multigres/go/services/multipooler [build failed]

• Compilation error before tests could run
• Claude should highlight the build error and suggest checking the code

Race Condition Detected

==================
WARNING: DATA RACE
Read at 0x00c0001a2080 by goroutine 7:
  ...
==================

• Race detector found a potential concurrency bug
• Claude should flag this as critical and recommend investigation

Timeout

panic: test timed out after 10m0s

• Test exceeded timeout
• Claude should suggest increasing timeout or investigating hanging test

Common Workflows

Before Committing Code

# 1. Run unit tests (fast feedback)
/mt-dev unit all

# 2. If unit tests pass, run integration tests
/mt-dev integration all

# 3. Check for race conditions
/mt-dev integration all -race

Debugging a Failing Test

# 1. Run the specific test with verbose output
/mt-dev integration multipooler TestConnCache -v

# 2. Check if it's flaky (intermittent failure)
/mt-dev integration multipooler TestConnCache -count=10

# 3. Run with race detector
/mt-dev integration multipooler TestConnCache -race -v

Testing a Specific Package After Changes

# Unit tests first (fast)
/mt-dev unit ./go/services/multipooler/... -v

# Integration tests second
/mt-dev integration multipooler -v

Troubleshooting

"build failed" during integration tests

• Run make build manually to see detailed error
• Check for uncommitted generated files (protobuf)
• Verify all dependencies are installed

Flaky tests (pass sometimes, fail others)

• Run with -count=10 to reproduce
• Enable race detector: -race
• Check for timing-dependent code or shared state

Debugging Integration Test Failures

When integration tests fail, logs are automatically preserved in a temp directory. Follow this systematic approach to debug failures:

Step 1: Find the Log Location

After a test failure, look for this message in the test output:

==== TEST LOGS PRESERVED ====
Logs available at: /tmp/shardsetup_test_XXXXXXXXXX
Set TEST_PRINT_LOGS=1 to print log contents
===========================

The temp directory contains all component logs from the failed test.

Step 2: Understand the Directory Structure

Integration test log directories follow this structure:

/tmp/shardsetup_test_XXXXXXXXXX/
├── multigateway.log              # Multigateway service logs
├── temp-multiorch/               # Temporary multiorch used during bootstrap
│   └── multiorch.log
├── pooler-1/                     # First multipooler instance (pooler-1, pooler-2, etc.)
│   ├── multipooler.log           # Multipooler service logs
│   ├── pgctld.log                # PostgreSQL control daemon logs
│   └── data/
│       ├── pg_data/
│       │   └── postgresql.log    # PostgreSQL server logs
│       ├── pg_sockets/           # Unix sockets for PostgreSQL connections
│       └── pgbackrest/
│           └── logs/             # Backup/restore logs
│               ├── multigres-backup.log
│               ├── multigres-restore.log
│               └── all-server.log
├── pooler-2/                     # Second multipooler instance
│   └── (same structure as pooler-1)
├── pooler-3/                     # Third multipooler instance
│   └── (same structure as pooler-1)
├── etcd_data/                    # etcd data directory
├── backup-repo/                  # pgBackRest backup repository
└── certs/                        # TLS certificates for pgBackRest

Step 3: Identify the Failing Component

Read the test failure message to identify which component is involved:

Example failure patterns:

• Connection/query errors → Check multipooler logs and PostgreSQL logs
• Replication issues → Check all pooler's PostgreSQL logs and multipooler logs
• Consensus/failover issues → Check multipooler logs (consensus protocol) and temp-multiorch logs
• Backup/restore errors → Check pgbackrest logs in pooler-*/data/pgbackrest/logs/
• Gateway routing issues → Check multigateway.log
• Bootstrap failures → Check temp-multiorch/multiorch.log and all pooler logs

Step 4: Examine Relevant Logs

Use this decision tree:

1. What component is the test about?

   • Multipooler → Start with pooler-*/multipooler.log
   • PostgreSQL → Start with pooler-*/data/pg_data/postgresql.log
   • Replication → Check PostgreSQL logs on all poolers
   • Multigateway → Check multigateway.log
   • Bootstrap/orchestration → Check temp-multiorch/multiorch.log

2. Look for ERROR or FATAL level messages in the relevant logs

3. Check timestamps - Look at logs around the time of the test failure

4. Follow the chain - If one component reports an error from another, check that component's logs next

Step 5: Common Debugging Commands

# View all log files in the preserved directory
find /tmp/shardsetup_test_XXXXXXXXXX -name "*.log" -type f

# Search for errors across all logs
grep -r "ERROR\|FATAL" /tmp/shardsetup_test_XXXXXXXXXX/

# View logs for a specific pooler
cat /tmp/shardsetup_test_XXXXXXXXXX/pooler-1/multipooler.log
cat /tmp/shardsetup_test_XXXXXXXXXX/pooler-1/data/pg_data/postgresql.log

# View multigateway logs
cat /tmp/shardsetup_test_XXXXXXXXXX/multigateway.log

# View backup/restore logs
ls /tmp/shardsetup_test_XXXXXXXXXX/pooler-1/data/pgbackrest/logs/

Step 6: Print Full Logs in CI

To see full log contents printed directly in test output (useful for CI):

TEST_PRINT_LOGS=1 /mt-dev integration shardsetup TestName

This will print all log files to stdout after the test fails, making them visible in CI logs without needing to download artifacts.

Example Debugging Workflow

Scenario: TestReplicationWorks fails with "replication not configured"

1. Find logs: Test output shows /tmp/shardsetup_test_1342939839

2. Identify components: Replication involves multipooler and PostgreSQL on all nodes

3. Check pooler logs:

   # Check multipooler logs for all poolers
   grep -i "replication\|primary_conninfo" /tmp/shardsetup_test_1342939839/pooler-*/multipooler.log
   
   # Check PostgreSQL logs
   grep -i "replication\|standby" /tmp/shardsetup_test_1342939839/pooler-*/data/pg_data/postgresql.log
   

4. Look for specific issues:

   • Is primary_conninfo being set?
   • Are there connection errors from standby to primary?
   • Is WAL streaming active?
   • Check for permission or authentication errors

5. Check multiorch logs if bootstrap or failover is involved:

   cat /tmp/shardsetup_test_1342939839/temp-multiorch/multiorch.log
   

Important Notes

• Logs are only preserved on test failure - successful tests clean up automatically
• Logs are cleaned on test success - if you need logs from passing tests, set a breakpoint or add t.Fatal() at the end
• Each test run gets a unique temp directory - look for the most recent timestamp
• Service logs are JSON formatted - use jq for pretty printing:

  cat pooler-1/multipooler.log | jq '.'
  

Tips

• Unit tests are fast (~seconds) - run them frequently while developing
• Integration tests are slow (~minutes) - run them before committing
• Use -short flag on unit tests to skip slow tests during rapid iteration
• Use -v flag when debugging to see which test is running
• Use -race flag before committing to catch concurrency bugs
• Use -count=10 to verify test stability (per CLAUDE.md guidelines)
• Use -count=1 to bypass cached results after environment or build changes
• If build fails, check that you've committed all generated files
• Coverage reports help identify untested code: -coverprofile=coverage.out
• Most integration test failures indicate real bugs, not flaky tests
• When integration tests fail, see "Debugging Integration Test Failures" section above for systematic debugging approach

Implementation Notes for Claude

Argument Parsing Logic

1. First argument (required):

   • For unit tests: package path or "all"
   • For integration tests: package name or "all"

2. Second argument (optional):

   • If starts with Test: use as -run <TestName> argument
   • If starts with -: treat as flag
   • Otherwise: error, show help

3. Additional arguments:

   • Parse as flags and append to go test command
   • Preserve order and formatting

Command Construction

Unit tests:

go test [flags] [-run TestName] <package-path>

Integration tests:

make build && scripts/portpool.sh start && MULTIGRES_PORT_POOL_ADDR=/tmp/multigres-port-pool.sock go test [flags] [-run TestName] ./go/test/endtoend/<package>/...

Output Handling

• Capture both stdout and stderr
• Summarize results: "X passed, Y failed in Z seconds"
• If failures, show failed test names and error messages
• For verbose output, provide condensed summary
• Offer to show full output if summary isn't sufficient