Project Go Backend

Go Backend Patterns

Stack Versions

Always pin to the latest stable Go release. Update Go toolchain (go.mod go directive + CI runner) when a new minor release is GA and has been out for 2+ weeks.

Dependency policy: prefer stdlib. When a library is needed, prefer the canonical pure-Go option:

Adding any new dependency requires a one-line note in the SOW under ## Decisions.

Concurrency Patterns

Goroutine ownership

Every goroutine has a clear owner who is responsible for starting and stopping it. Document in the constructor godoc:

// NewIngester constructs an ingester. The caller MUST call Close to stop
// the internal worker goroutines.
func NewIngester(...) *Ingester { ... }

Context propagation

• Every blocking function takes ctx context.Context as the first parameter.
• Every goroutine respects ctx.Done() for shutdown.
• HTTP handlers use r.Context() and pass it through to store/query calls.
• Registered REST handlers (mux.HandleFunc("/api/...", p.<handler>)) keep their method gate directly visible in the handler body as r.Method != http.Method... comparisons. Do not extract the method guard itself into a helper: scripts/spec-drift.sh intentionally fails closed when it cannot prove the registered verb/path contract from the handler body. Extract the error writer or response/query work instead.

Channels

• Direction: prefer <-chan T / chan<- T in function signatures to make ownership explicit.
• Buffer sizes: documented in code with a comment explaining the choice and the drop policy when full.
• Close: only the sender closes. Document who the sender is.

SQLite Patterns

Schema migrations

internal/store/migrations/NNNN_description.sql. Run in alphabetical order at startup; tracked in schema_meta. Migrations are idempotent (CREATE TABLE IF NOT EXISTS ...) where possible; non-idempotent migrations are guarded by version checks.

Prepared statements

Hold prepared statements in a package-level *sql.Stmt cache initialized once. Re-preparing on every call is forbidden.

Transactions

• Ingest pipeline batches writes in transactions of up to 100 rows or 500 ms, whichever first.
• Keep source-scoped derived-table repair chunks at the same 100-row writer budget unless a later SOW proves a larger chunk still leaves Tail heartbeat/stale-tail writes within their 30 s liveness budget.
• Server-side reads do not use transactions unless multiple queries must be consistent (rare).
• Always defer tx.Rollback() immediately after Begin; the deferred call is a no-op if Commit succeeded.

Connection pool

• Ingester uses single-writer (max-open=1) on the write connection; readers can be a separate pool.
• Server uses read-only pool (max-open=8). Tunable later if profiling shows contention.

fsnotify Patterns

w, err := fsnotify.NewWatcher()
if err != nil { return err }
defer w.Close()

if err := w.Add(rootDir); err != nil { return err }

// For subdirs created at runtime: re-Add when CREATE event fires on a dir.

for {
    select {
    case <-ctx.Done():
        return ctx.Err()
    case ev, ok := <-w.Events:
        if !ok { return nil }
        handle(ev)
    case err, ok := <-w.Errors:
        if !ok { return nil }
        opts.OnError(err)
    }
}

• Use w.Add(dir) not file paths. Filter events in handle.
• Recurse manually: walk the tree once at startup, then Add each new dir on CREATE.
• Atomic rename pattern (x.tmp → x): listen for Rename/Create on the target, not Write.
• Inotify watcher limits: on Linux, default user limit is 8192 watches. If the source tree exceeds, log a clear error pointing to sysctl fs.inotify.max_user_watches.

SSE Patterns

stdlib only. Pattern:

func sseHandler(w http.ResponseWriter, r *http.Request) {
    flusher, ok := w.(http.Flusher)
    if !ok {
        http.Error(w, "streaming unsupported", http.StatusInternalServerError)
        return
    }
    w.Header().Set("Content-Type", "text/event-stream")
    w.Header().Set("Cache-Control", "no-cache")
    w.Header().Set("Connection", "keep-alive")
    w.Header().Set("X-Accel-Buffering", "no") // disable nginx buffering

    sub, err := hub.Subscribe(...)
    if err != nil { http.Error(...); return }
    defer hub.Unsubscribe(sub)

    keepalive := time.NewTicker(15 * time.Second)
    defer keepalive.Stop()

    for {
        select {
        case <-r.Context().Done():
            return
        case ev := <-sub.Events:
            fmt.Fprintf(w, "id: %d\nevent: %s\ndata: %s\n\n", ev.ID, ev.Type, ev.JSON)
            flusher.Flush()
        case <-keepalive.C:
            fmt.Fprint(w, ": keepalive\n\n")
            flusher.Flush()
        }
    }
}

Logging Patterns

logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo}))
logger = logger.With("subsystem", "ingest")

logger.Info("source scan complete",
    "source_id", srcID,
    "events", n,
    "duration_us", dur.Microseconds(),
)

logger.Warn("parse error",
    "source_id", srcID,
    "file", path,
    "offset", off,
    "err", err,
)

• Never log raw user content from snapshots (PII concerns).
• Always include the subsystem.
• Errors at WARN; system failures at ERROR.

Error Patterns

// Wrap with context using %w
return fmt.Errorf("scan source %s: %w", srcID, err)

// Sentinel errors for control flow
var ErrCursorBehind = errors.New("cursor behind oldest available record")

// Adapter parse errors call opts.OnError, do not return
opts.OnError(fmt.Errorf("parse record at %s:%d: %w", file, off, err))
continue

Testing Patterns

func TestScanHappyPath(t *testing.T) {
    t.Parallel()

    fixDir := filepath.Join("testdata", "happy_path")
    out := make(chan canonical.Event, 100)
    errs := []error{}
    a := New(fixDir, canonical.AdapterOptions{
        OnError: func(e error) { errs = append(errs, e) },
    })

    if err := a.Scan(context.Background(), canonical.Cursor{}, out); err != nil {
        t.Fatalf("scan: %v", err)
    }
    close(out)

    got := drain(out)
    want := readGolden(t, filepath.Join(fixDir, "expected.jsonl"))
    if diff := cmp.Diff(want, got); diff != "" {
        t.Fatalf("events mismatch (-want +got):\n%s", diff)
    }
    if len(errs) > 0 { t.Fatalf("unexpected errors: %v", errs) }
}

• t.Parallel() on every test that doesn't share state.
• Use table-driven tests for input variations.
• Use cmp for struct/slice comparisons (clearer diffs than reflect.DeepEqual).
• Golden files in testdata/<name>/expected.jsonl; update with -update-golden flag.

Linting Configuration

.golangci.yml (v2) enables the strict set — errcheck, govet, ineffassign, staticcheck, unused, errorlint, gocritic, revive, gocyclo (min-complexity 25), misspell, nilerr, prealloc, unconvert, unparam, whitespace, bodyclose, noctx — plus the gofmt/goimports/gofumpt formatters. gosimple is folded into staticcheck (v2); gosec runs standalone (not as a golangci linter). Zero-warnings policy. The authoritative gate catalog is .agents/sow/specs/quality-gates.md "Go — Lint" and the project-quality-gates skill; run locally via scripts/lint.sh.