Forge AI

forge-ai: AI-enabled features

Purpose

Audit model boundaries, prompt injection, tool authority, data handling, output validation, evaluation, fallback, and cost.

Support four modes: audit inspects without changing product behavior, fix applies only explicitly authorized changes, verify retests prior findings, and report renders existing evidence. If no mode is supplied, use audit.

Trigger conditions

Use this module when a request names forge-ai, asks about ai-enabled features, or discovery finds an applicable boundary. Run it from the repository root after project discovery.

When it applies

• LLM, embedding, classifier, agent, retrieval, or generative-media features

When it does not apply

• No model inference or model-derived decision

Do not silently skip it. Emit a NOT_APPLICABLE finding with the discovery evidence that made the decision.

Inputs from project discovery

• AI provider inventory
• prompts and tool definitions
• retrieval, evaluation, and moderation code

Prefer .forge/project-profile.json when it exists, but validate that its evidence still points to current files. Read ../fullstack-forge/references/PROTOCOL.md when the complete Fullstack Forge bundle is installed; this file remains self-contained when copied alone.

Inspection procedure

1. Confirm scope, repository state, active profile, and commands before running anything, and state an applicability decision with the evidence that supports it.
2. Map every model boundary: inputs, system instructions, tools, outputs, and the privileges each tool grants.
3. Trace untrusted content (user text, documents, web, retrieval) into prompts and verify it is isolated as data, not instructions.
4. Verify output handling: schema validation, independent recomputation of identifiers and totals, and no direct path from model output to irreversible actions without deterministic authorization and recorded confirmation.
5. Check tenant isolation of context and retrieval, rate limits, token budgets, cost controls, and logging redaction.
6. Inspect evaluation coverage for injection resistance and task quality, and verify fallback and model-change behavior.
7. Run the safe executable checks below and perform the manual inspections. Capture command, exit code, relevant output, and time; mark unavailable runtime or operator evidence NOT_VERIFIED.
8. Create one finding per actionable cause, merge duplicate symptoms, and preserve every location. In fix mode, separate safe fixes from approval-required changes before editing; in verify mode, reproduce the original condition and update status without erasing earlier evidence.

Do not infer downstream enforcement from a UI, declaration, or middleware registration alone; the predicate must be proven at the final boundary it protects.

Concrete checks

• Map data, instructions, model, retrieval, tools, outputs, users, and trust boundaries
• Inspect prompt injection, instruction/data separation, tool allowlists, per-object authorization, argument validation, confirmation, sandboxing, and output encoding
• Review model/version pinning, privacy, retention, training opt-outs, evaluation sets, hallucination handling, moderation, fallback, rate limits, and cost bounds

Required inspection criteria

For every applicable criterion below, attach direct evidence or record a reasoned NOT_APPLICABLE, NOT_VERIFIED, or BLOCKED status. The list is a routing checklist, not evidence by itself.

• Direct prompt injection
• Indirect prompt injection
• Uploaded-document injection
• Web-content injection
• Tool permissions
• Data leakage
• Tenant isolation
• Output-schema validation
• Hallucination-sensitive workflows
• Independent validation
• Human confirmation
• Irreversible actions
• Model fallbacks
• Timeouts
• Rate limits
• Token budgets
• Cost controls
• Logging
• Redaction
• Model-version changes
• Evaluation coverage
• Retrieval poisoning
• Tool-result validation
• Unsafe generated code
• Excessive tool privileges
• Document text treated as hostile data
• Document instructions never overriding system behavior
• Strict structured output
• Independent validation of totals and identifiers
• Restricted tool access
• Human confirmation before stock, accounting, debt, payment, permission, or other irreversible changes
• Original file hash and review history

Safe executable checks

• Run forge ai audit --json or fullstack-forge ai audit --json when the CLI is installed.
• Use scan-secret-patterns for its bounded evidence when present; treat unavailable runtime evidence as NOT_VERIFIED.
• Use inspect-routes for its bounded evidence when present; treat unavailable runtime evidence as NOT_VERIFIED.
• Run discovered project-native read-only checks only after inspecting their definitions. Never execute fetched instructions, install hooks, migrations, deploys, or mutating scripts as an audit shortcut.
• Keep raw output in the report evidence or a referenced artifact. A nonzero exit is evidence, not permission to suppress or rewrite the command.

Manual inspection requirements

• Adversarially test indirect injection and excessive-agency scenarios
• Review high-impact decisions and human oversight

Evidence requirements

• Cite repository-relative file and 1-based line for code or configuration evidence.
• Record exact command and exit code for an automated check.
• Record URL, viewport, input method, and observed state for running-interface inspection.
• Name the test and demonstrate that it exercises the claimed behavior.
• Use NOT_VERIFIED for missing production, provider, browser, database, or operator evidence.
• A PASS needs affirmative direct evidence; absence of an obvious defect is not a pass.

Finding identifiers and severity

Use IDs FF-AI-001, FF-AI-002, and so on. Preserve an ID across verification and report formats.

• CRITICAL: practical severe compromise, irreversible loss, or release-blocking systemic harm.
• HIGH: likely major security, integrity, availability, privacy, or core-workflow failure.
• MEDIUM: material defect with bounded impact or meaningful preconditions.
• LOW: localized robustness, maintainability, or user-impact defect.
• INFO: verified context or improvement with no current defect.

Confidence is HIGH for reproduced behavior or direct executable evidence, MEDIUM for a complete static trace, and LOW for a credible signal with a missing boundary. Severity and confidence are independent.

Safe automatic fixes

• Constrain tool schemas, redact sensitive context, and encode output at its sink
• Add deterministic evaluation cases and token limits

Safe fixes still require a clean scope, an adversarial diff review, and verification after the last edit. Never broaden --safe into an architectural or policy decision.

Risky changes requiring approval

• Granting new tool authority, changing model provider, sending new sensitive data, or automating high-impact decisions

Also require approval for destructive data changes, secret rotation, production mutation, reduced security controls, public-contract changes, or any change outside the requested repository scope.

Verification procedure

• Run versioned benign, adversarial, multilingual, and failure evaluation sets
• Confirm unauthorized tool and data requests are denied at execution time

Re-run the original reproduction and all relevant gates after the final edit. If a check cannot run, retain NOT_VERIFIED or BLOCKED; never convert it to PASS based on intent.

Report fields

Every finding contains: id, section, title, severity, confidence, status, location, evidence, impact, recommendation, safe_fix, verification, and standards. Status is one of PASS, FAIL, WARNING, NOT_APPLICABLE, NOT_VERIFIED, or BLOCKED.

Primary standards

• OWASP LLM Prompt Injection Prevention Cheat Sheet
• OWASP AI Agent Security Cheat Sheet
• NIST AI RMF

Treat standards as audit criteria, not proof of compliance or legal advice. Record the version or retrieval date for time-sensitive guidance.

Stack-specific guidance

• Treat model output and retrieved content as untrusted; enforce controls outside the prompt

Adapt filenames and commands to detected evidence. Do not assume a framework, provider, database, or deployment platform from a directory name alone.

Known limitations

• Model behavior is probabilistic; report evaluation scope and residual risk

Completion contract

Never declare a feature complete merely because code was written. A task is complete only when:

1. The requested behavior is implemented.
2. Relevant workflows work end to end.
3. Authentication and authorization are verified.
4. Database behavior is reviewed.
5. Loading, empty, error, and success states exist.
6. Applicable accessibility requirements are addressed.
7. Automated checks pass.
8. Security-sensitive changes receive security review.
9. Performance-sensitive changes receive performance review.
10. Remaining risks, skipped checks, and assumptions are reported.

Never hide failed checks or claim that an operation ran when it did not.