Forge AI

forge-ai: AI-enabled features

Engine: Hybrid — Forge + wshobson, Google, Cloudflare, Sentry

Purpose

Audit model boundaries, prompt injection, tool authority, data handling, output validation, evaluation, fallback, and cost.

Deterministic runtime composition

Before loading any provider procedure, run:

Resolve ../../runtime/cli/src/composition-entry.js relative to this SKILL.md, then run:

node "<resolved-absolute-runner-path>" ai compose --workflow audit --root "<repository-root>" --dry-run --json

Add one repeatable --request <provider-or-source> flag for each explicit user request. Add --condition <task-condition> or --risk-surface <surface> only for a task fact you directly proved; never infer one from generic wording. The command above is the default for this audit-oriented module; for implementation use --workflow build, and for a fix, retest, or release gate use --workflow fix, verify, or ship respectively. Read the JSON response, keep the Forge contract at index zero, and resolve paths against the absolute runtime_root reported in that response. Read eager[].runtimePath when entering the module. The full selected[] list is availability/provenance; load only deferred[].runtimePath when the task reaches that concern, in tier order. Refuse any path that escapes the root. Respect every reported suppression and context budget. If missing is non-empty, stop and report the installation as damaged; do not improvise a prose fallback. The runner and specialist content may live in a plugin cache or global installation; never assume they are inside the audited repository.

Resolve and read ../fullstack-forge/references/shared/module-contract.md (applicability, execution, mutation, verification, completion) and ../fullstack-forge/references/shared/evidence-rules.md (statuses, standards, tools, findings via ../fullstack-forge/references/PROTOCOL.md) relative to this module SKILL.md before reporting.

Never hide failed checks or claim that an operation ran when it did not.

Automatic activation signals

Activate when a request or direct repository evidence involves ai-enabled features, when the user explicitly names forge-ai, or when discovery proves an applicable boundary.

• LLM, embedding, classifier, agent, retrieval, or generative-media features

When not to activate

• No model inference or model-derived decision

Automated support

Relevant discovery inputs are:

• AI provider inventory
• prompts and tool definitions
• retrieval, evaluation, and moderation code

Deterministic support, bounded evidence only:

• scan-secret-patterns
• inspect-routes

Agent inspection procedure

1. Map every model boundary: inputs, system instructions, tools, outputs, and the privileges each tool grants.
2. Trace untrusted content (user text, documents, web, retrieval) into prompts and verify it is isolated as data, not instructions.
3. Verify output handling: schema validation, independent recomputation of identifiers and totals, and no direct path from model output to irreversible actions without deterministic authorization and recorded confirmation.
4. Check tenant isolation of context and retrieval, rate limits, token budgets, cost controls, and logging redaction.
5. Inspect evaluation coverage for injection resistance and task quality, and verify fallback and model-change behavior.

Manual inspection requirements:

• Adversarially test indirect injection and excessive-agency scenarios
• Review high-impact decisions and human oversight

Stack-specific guidance:

• Treat model output and retrieved content as untrusted; enforce controls outside the prompt

Evidence to collect

Standards used as criteria:

• OWASP LLM Prompt Injection Prevention Cheat Sheet
• OWASP AI Agent Security Cheat Sheet
• NIST AI RMF

Common production failures

• Map data, instructions, model, retrieval, tools, outputs, users, and trust boundaries
• Inspect prompt injection, instruction/data separation, tool allowlists, per-object authorization, argument validation, confirmation, sandboxing, and output encoding
• Review model/version pinning, privacy, retention, training opt-outs, evaluation sets, hallucination handling, moderation, fallback, rate limits, and cost bounds

Missing-control checks

Each item needs direct evidence or one reasoned status.

• Direct prompt injection
• Indirect prompt injection
• Uploaded-document injection
• Web-content injection
• Tool permissions
• Data leakage
• Tenant isolation
• Output-schema validation
• Hallucination-sensitive workflows
• Independent validation
• Human confirmation
• Irreversible actions
• Model fallbacks
• Timeouts
• Rate limits
• Token budgets
• Cost controls
• Logging
• Redaction
• Model-version changes
• Evaluation coverage
• Retrieval poisoning
• Tool-result validation
• Unsafe generated code
• Excessive tool privileges
• Document text treated as hostile data
• Document instructions never overriding system behavior
• Strict structured output
• Independent validation of totals and identifiers
• Restricted tool access
• Human confirmation before stock, accounting, debt, payment, permission, or other irreversible changes
• Original file hash and review history

Commands and tools

• Run forge ai audit --json or fullstack-forge ai audit --json when an explicit audit is requested and the CLI is installed. Normal feature work does not require it.

Safe fixes

• Constrain tool schemas, redact sensitive context, and encode output at its sink
• Add deterministic evaluation cases and token limits

Approval-required changes

• Granting new tool authority, changing model provider, sending new sensitive data, or automating high-impact decisions

Verification

• Run versioned benign, adversarial, multilingual, and failure evaluation sets
• Confirm unauthorized tool and data requests are denied at execution time

Completion contract

Follow fullstack-forge/references/shared/completion.md and the limitations below.

Known limitations

• Model behavior is probabilistic; report evaluation scope and residual risk