SaaS Admin / Back-Office Tooling
Acknowledgement: Shared by Peter Bamuhigire, techguypeter.com, +256 784 464178.
Required Inputs
| Input |
Required |
Use |
| Tenant, product, and lifecycle scope |
yes |
Bound the SaaS decision |
| Current architecture, plans, policies, and constraints |
yes |
Preserve enforceable behaviour |
| Production data or verified evidence |
conditional |
Validate thresholds and migrations |
Capability and permission contract
Default to read-only analysis. Change configuration, billing, identity, tenant data, infrastructure, or customer communications only with explicit authority, least-privilege credentials, tenant scope, rollback, and auditable approval. Never expose secrets or cross tenant boundaries.
Degraded mode
If production access, policy, telemetry, or authoritative records are unavailable, produce a labelled design or dry-run plan. Do not claim deployment, reconciliation, deletion, delivery, or measured outcomes; list missing evidence and verification.
Decision rules
| Condition |
Action |
Stop condition |
| Tenant isolation, money, identity, or deletion is affected |
Require approval and rollback evidence |
Scope or authority is ambiguous |
| Evidence supports a reversible change |
Stage, test, and record it |
Acceptance checks fail |
| Only partial context is available |
Return assumptions and validation |
A production claim cannot be verified |
Domain Anti-Patterns
- Applying one tenant's policy or data to another. Fix: enforce tenant scope at every boundary.
- Mutating production from an advisory request. Fix: remain read-only until authority is explicit.
- Inventing limits, prices, metrics, or compliance claims. Fix: use authoritative records or mark them unresolved.
- Shipping without rollback and audit evidence. Fix: stage and retain before/after proof.
- Treating a missing dependency as successful. Fix: name the blocked verification.
Use When
- Designing the internal app that the SaaS provider's staff use to operate the platform — tenant ops, billing ops, support ops, compliance ops.
- Replacing SSH + ad-hoc SQL + Stripe Dashboard work with an audited, scriptable, role-based console.
- Adding tenant impersonation to a SaaS (so support can see what the tenant sees, audited and time-boxed).
- Building bulk operations (plan migration of a cohort, mass invite, region migration).
- Implementing the SOC2 / GDPR controls around privileged access.
Do Not Use When
- The task is the customer-facing super-admin or tenant-admin panel (inside one tenant) — use
multi-tenant-saas-architecture three-panel pattern.
- The task is the control-plane services architecture — use
saas-control-plane-engineering; this skill is the UI/UX layer on top.
- The task is the audit log schema itself — use
saas-control-plane-engineering §6.
- The task is general RBAC — use
dual-auth-rbac and multi-tenant-saas-architecture.
Required Inputs
- Control-plane service inventory from
saas-control-plane-engineering.
- Audit log spec from
saas-control-plane-engineering.
- Tenant lifecycle states from
saas-control-plane-engineering.
- Compliance posture (SOC2, ISO27001, HIPAA?) — determines required guardrails.
- Internal roles (super admin, support engineer, billing ops, finance, security, customer success).
Workflow
- Read this
SKILL.md.
- Map the operations the back-office must support (§2) — tenant ops, billing ops, support ops, compliance ops.
- Design the role model (§3) — internal staff roles + privileged-access controls.
- Build the audited mutation pipeline (§4) — every action passes through guarded handlers.
- Implement impersonation correctly (§5) — time-boxed, justified, audited, visible.
- Build bulk operations safely (§6) — dry-run + idempotency + rate limit + rollback.
- Apply the privileged-access workflow (§7) — break-glass, MFA, approval chains for high-risk actions.
- Apply anti-patterns (§8).
Quality Standards
- Every back-office mutation passes through a single audit-logging middleware. Direct DB writes from staff are forbidden in production.
- Every privileged action requires MFA for the staff user.
- Impersonation is time-boxed (default 30 minutes), justified (free-text required), and visible to the tenant (banner + email notification optionally).
- Bulk operations have a dry-run mode by default; production-mode requires a second approval.
- Every action emits an event to the audit log and to the staff-Slack channel (for transparency among the team).
Anti-Patterns
- Staff doing tenant ops via SSH +
mysql shell — no audit trail.
- Impersonation that masquerades silently — tenant has no idea their account was accessed.
- Refunds issued directly in Stripe Dashboard — diverges from internal records, no justification captured.
- "Admin can do anything" — no role separation; finance role shouldn't be able to delete tenants; support role shouldn't be able to refund.
- Bulk operations without dry-run — one bad query suspends 500 wrong tenants.
- Privileged actions without MFA — single password compromise = platform compromise.
- Feature-flag overrides per tenant scattered through code instead of in a single override table.
Outputs
- Back-office app design (routes, role map, mutation handlers).
- Impersonation workflow + guardrails.
- Bulk operations playbook (dry-run + approval + rollback).
- Internal roles + permissions matrix.
- Audit-log integration spec.
Evidence Produced
| Category |
Artifact |
Format |
Example |
| Correctness |
Back-office app spec |
Markdown doc with routes + role map |
docs/saas/backoffice-app.md |
| Security |
Privileged-access workflow |
Markdown doc |
docs/saas/privileged-access.md |
| Operability |
Bulk-operations playbook |
Markdown doc |
docs/saas/bulk-ops-playbook.md |
References
references/impersonation-design.md — time-boxed, justified, visible impersonation.
references/bulk-operations.md — dry-run + approval + rollback patterns.
references/internal-roles-and-permissions.md — typical role matrix.
- Companion:
saas-control-plane-engineering, multi-tenant-saas-architecture, dual-auth-rbac, vibe-security-skill, saas-sso-scim-enterprise-auth.
- AI incident console: when the platform ships AI features, the back-office must include an AI incident console with: feature kill-switch, agent task kill-switch, model-pin, prompt-pin, retrieval index-pin, tool-pin / tool-disable, gateway routing pin, per-tenant feature pause, quota cap, and the evidence-bundle exporter. Each control writes to
ai_incident_mitigation_log with (actor, ts, primitive, scope, reason, ticket_id). The reason field is mandatory. See ai-incident-response-runbook §3 for the primitive contract and ai-incident-recovery-and-rollback/references/rollback-patterns.md for the un-pin path.
§1 Why the Back-Office Console Matters
Trio book: between $1M and $10M ARR, the manual-SSH-and-Stripe-Dashboard model collapses. Staff make mistakes; nothing is audited; customer trust erodes. The back-office console is how operations becomes a discipline rather than a hero activity.
For SOC2 and similar: every privileged access must be logged, justified, time-boxed, and traceable to a human. A console enforces this; a shell does not.
§2 The Operations Surface
| Domain |
Operations |
| Tenant ops |
Search; view detail; suspend; restore; archive; hard-delete; transfer ownership; change plan/tier/region; toggle feature-flag overrides; impersonate |
| User ops |
Search; view memberships; reset password; force logout; revoke sessions; un-suspend account; delete user (GDPR) |
| Billing ops |
Issue refund; issue credit; apply discount; override plan price; change billing email; pause subscription; reactivate cancelled subscription |
| Support ops |
View tickets; merge accounts; export tenant data on customer request; bulk re-invite team members; reset onboarding |
| Compliance ops |
Initiate GDPR erasure; export audit log for tenant; sign DPA; record subprocessor change |
| Bulk ops |
Plan-migration cohort; region-migration cohort; mass-suspend (e.g., fraud cluster); mass-email broadcast (rare, regulated) |
§3 Internal Role Model
Don't have "is_super_admin" boolean. Have roles:
| Role |
Can |
Cannot |
| support_l1 |
Search, view, basic password reset, view tickets |
Refund, suspend, impersonate |
| support_l2 |
+ impersonate (with justification), suspend with reason, issue $50 credit |
Hard delete, refund > $200 |
| billing_ops |
Refund / credit / plan override / change billing email |
Suspend / impersonate / delete |
| engineering_oncall |
Feature-flag overrides, restart workers, replay events |
Refund / billing changes |
| security |
Force logout, revoke sessions, audit-log review, GDPR erasure |
Refund / plan override |
| super_admin |
Anything, but MFA + co-sign on highest-risk (hard delete, bulk-suspend) |
— |
Each role's permissions live in a permissions table; the UI hides irrelevant controls; the API double-checks on every request.
§4 The Audited Mutation Pipeline
Every back-office mutation passes through the same pipeline:
HTTP request → AuthN (staff session + MFA) → AuthZ (role + permission)
→ Captures: actor_user_id, actor_ip, justification (required for high-risk)
→ Idempotency check (Idempotency-Key header)
→ Begin DB transaction
→ Pre-state capture (before_state JSON)
→ Mutation
→ Post-state capture (after_state JSON)
→ Audit log INSERT (same transaction)
→ Commit
→ Emit event (Slack notification, downstream services)
A direct DB write that bypasses the pipeline is a policy violation, not just a bug.
§5 Impersonation
The most-sensitive feature.
Rules:
- Requires high-trust role (support_l2, super_admin).
- Requires
justification (free text, audited).
- Time-boxed (default 30 min, max 4 hours).
- Subject tenant sees a banner: "Support is currently viewing your workspace" (configurable per plan — enterprise often wants this visible).
- Optional notification email to tenant primary admin (configurable per plan).
- Impersonation start + end events in audit log.
- All actions during impersonation tagged with
acting_as_user_id.
Implementation pattern:
def start_impersonation(staff_user_id, target_user_id, justification, duration_min=30):
require_role(staff_user_id, 'support_l2_or_above')
require_mfa(staff_user_id)
require_non_empty(justification)
assert duration_min <= 240
session = ImpersonationSession.create(
staff_user_id=staff_user_id,
target_user_id=target_user_id,
justification=justification,
expires_at=now() + duration_min * 60,
)
audit_log('IMPERSONATION_START', actor=staff_user_id, target=target_user_id,
reason=justification, target_tenant=target_user.tenant_id)
notify_target_tenant_admin(target_user_id) # configurable
return session_token
Every action while impersonation is active is double-stamped:
audit_log(actor=staff_user_id, acting_as=target_user_id, action='...', target_tenant_id=...)
§6 Bulk Operations
Bulk operations are blast-radius nightmares. Pattern:
1. Staff selects cohort (SQL query, CSV upload, saved segment).
2. Dry-run runs the operation in a transaction-marked-readonly; produces a report (what would change, on how many tenants).
3. Staff reviews report; submits production-run with a second-approver MFA confirmation.
4. Production-run executes in batched chunks (e.g., 50 tenants per batch); pauseable; with progress reporting.
5. Per-tenant audit log entries.
6. Rollback snapshot — pre-state captured so undo is possible within N hours.
7. Slack broadcast at start, midpoint, completion.
Examples of safe bulk ops:
- Plan migration: "Move all
legacy_pro to pro_v2 with same price-locked Stripe Subscription."
- Region migration: "Move tenants in EU pod from
eu-west-1 to eu-central-1."
- Feature-flag rollout: "Enable
new_dashboard for 10% of pro tenants."
Examples of dangerous bulk ops that need extra scrutiny:
- Mass-suspend (fraud cluster) — requires security sign-off + co-sign.
- Mass-delete — requires CEO/CFO sign-off + delay window.
§7 Privileged-Access Workflow
| Action |
Approval |
MFA |
Co-sign |
Time-window |
| Login to back-office |
MFA |
Yes |
No |
Per session |
| View tenant |
role |
Yes |
No |
— |
| Suspend tenant |
role |
Yes |
No |
— |
| Impersonate |
role |
Yes |
No |
30 min default |
| Refund < $100 |
billing_ops |
Yes |
No |
— |
| Refund $100-$1000 |
billing_ops |
Yes |
Co-sign by another billing_ops |
— |
| Refund > $1000 |
billing_ops |
Yes |
Co-sign by finance lead |
— |
| Hard-delete tenant |
super_admin |
Yes |
Co-sign + 24h cooldown |
— |
| Mass-suspend cohort |
super_admin |
Yes |
Co-sign by security |
— |
Co-sign mechanism: second staff user approves in the back-office UI within a time window; otherwise the action expires.
§8 Anti-Patterns
admin.html page on the same app behind a JWT claim — same auth surface, same code path, same risk; back-office should be a separate auth domain, separate deploy, separate network segment.
- Impersonation invisible to tenant — discovered later; trust catastrophe.
- No co-sign on high-risk actions — one bad day, one compromised laptop, platform compromised.
- Justifications optional — left blank by everyone; audit log is useless.
- No internal Slack mirror of audit events — team has no informal accountability layer.
- Bulk ops without dry-run — one bad SELECT kills 500 tenants.
- Refunds via Stripe Dashboard — bypasses platform record; ledger drift; tax-reporting headaches.
- Same role for support and billing — supports cancellation of others' accounts; supports cross-team mistakes.
§9 Build Stack Suggestions
Depending on the SaaS stack:
- Retool / Internal / Tooljet / Forest Admin / Appsmith — fast to build, audit-friendly, role-based; great for $1M-$10M ARR stage.
- Custom React/Vue app talking to control-plane APIs — once team scales, custom gives the best UX and integration.
- Django admin + customization — works for Django stacks; needs extension for audit/impersonation/co-sign.
- Laravel Nova / Filament — same for Laravel/PHP stacks.
Regardless of UI tool, the API surface is the contract — UI is just one client; CLI / Slack-bot / scripts are equally valid clients of the back-office API.
§10 Agent Ops Console
When agentic features ship, the back-office gains a new surface: the Agent Ops Console. Without it, operating live agents in a multi-tenant SaaS is impossible — a runaway task or a tenant-specific safety incident has no fast remediation.
Required views
| View |
Purpose |
| Live agent tasks (per-tenant + global) |
All in-flight tasks: tenant, feature, state, age, cost so far, last activity |
| Pending approvals (per-tenant + global) |
All awaiting-approval tasks; staff can override on behalf of tenant admin |
| Task viewer (drill-down) |
Per-task trace + step list + tool I/O (redacted) + replay link — see ai-agent-observability-and-replay |
| Tenant agent config |
View / edit per-tenant tool allow-list, budgets, entitlement overrides; see audit |
| Tenant kill-switch panel |
Pause / resume agent for a tenant; per-feature pause; reason required |
| Agent budget overrides |
Time-boxed overrides with expires_at, reason, actor |
| Safety incidents |
Live feed of agent.injection.detected, agent.exfil.attempt, agent.cross_tenant.attempt, agent.escalation.blocked |
| Stuck tasks / Abandonment queue |
Tasks that exceeded stall thresholds; resume / abandon / kill |
| Agent cost dashboard |
Per-tenant cost per completed task; wasted spend share; outliers |
Required actions
| Action |
What it does |
Audit + safeguards |
| Kill task |
Transitions one task to KILLED; runs reversibility compensations |
Reason required; emits agent.task.killed; tenant notified |
| Force-pause task |
Transitions to AWAITING_APPROVAL (paused-by-staff); resumes when staff or tenant resumes |
Reason required; tenant notified |
| Tenant-wide agent kill-switch |
Sets agent_enabled=false; in-flight tasks complete current step and stop |
Reason; tenant notified; expires unless made permanent |
| Feature-specific kill-switch |
Pauses one agent feature for one tenant |
Reason; auto-expire default 24h |
| Approval override |
Approve / reject an approval on behalf of a tenant admin |
Co-sign required for irreversible; visible to tenant |
| Resume abandoned task |
Restart from last persisted step |
Audit |
| Re-run with new config |
Replay task with candidate prompt/model/tools |
Sandbox only — never re-execute side effects in prod |
| Memory inspection / forget |
Surface per-tenant or per-user agent memories; force-forget on legal request |
GDPR audit row written |
| Tool registry view |
See which tools each tenant has, which versions pinned, deprecations |
Read-only here; mutations via control-plane API |
Kill-Switch Speed Requirement
Per-tenant agent kill-switch must flip in < 5 seconds wall-time from staff click to runtime enforcement. Implementation: write to a Redis key the runtime checks before each step; DB updated asynchronously for audit.
# Runtime check before each step
def check_kill_switch(tenant_id):
if redis.get(f"agent:killed:{tenant_id}") == "1":
return KillSwitch("tenant_kill_switch")
return None
The kill-with-rollback runbook (ai-agent-reversibility-and-blast-radius §4) describes the compensation cascade. The console wires the button.
Required co-signs
- Tenant-wide agent kill-switch on Enterprise: 2 staff (support engineer + tenant CSM or higher).
- Approval override of irreversible: 2 staff.
- Bulk kill of > 10 tasks: 2 staff + dry-run.
- Memory force-forget: 1 staff + legal ticket reference.
Cross-references
ai-agent-runtime-architecture — provides the task list + state machine the console drives.
ai-agent-observability-and-replay — provides the task viewer + replay link.
ai-agent-action-approval-and-hitl — approval objects the staff override.
ai-agent-safety-and-red-team — safety incident feed.
ai-agent-reversibility-and-blast-radius — kill-with-rollback procedure.
ai-agent-memory — memory inspection / forget surface.
§11 Read Next
saas-control-plane-engineering — the services this console drives.
multi-tenant-saas-architecture — tenant data model + cross-tenant access rules.
subscription-billing — billing operations this console invokes.
saas-tenant-data-portability-and-erasure — GDPR workflows this console initiates.
dual-auth-rbac — internal auth + MFA underpinning.
vibe-security-skill — security baseline for the back-office app.
ai-agent-runtime-architecture — agent task control-plane.
ai-agent-observability-and-replay — task viewer + replay surfaces.
§12 SLA-Credit Override + Dispute-Resolution Console (Enhancement)
For agent products, the back-office needs two purpose-built surfaces beyond the standard tenant/billing views: an SLA-credit override console and a dispute-resolution console. Both share the principle that every keystroke is dual-authed, audit-logged, and links to the verdict / evidence pack.
SLA-credit override console
Reads from the credit-case ledger produced by ai-agent-sla-credit-automation. For each open case the operator sees:
- The breach event (incident link or detection signal).
- The eligibility decision (auto-approved, auto-denied, needs-review).
- The proposed credit amount (formula + inputs).
- The trace bundle + verdict + evidence pack.
- The per-tenant credit cap remaining for the period.
Actions:
| Action |
Requires |
Audit fields |
| Approve as-proposed |
single-auth (operator role) |
approved_by, approved_at |
| Adjust amount |
dual-auth (operator + manager) |
original_amount, adjusted_amount, reason_code, manager_id |
| Deny |
dual-auth |
reason_code (from enumerated list: customer_caused / force_majeure / excluded_class / cap_reached), denial_text |
| Issue mass credit |
dual-auth + IC approval link |
incident_id, tenant_filter_sql, formula_ref |
Every action emits to the credit-case audit log; the customer-facing dashboard reflects the outcome in < 60 seconds.
Dispute-resolution console
Reads from the dispute queue produced by ai-agent-task-success-tracking/references/dispute-resolution.md. For each open dispute the operator sees:
- Customer-asserted outcome (success or failure).
- System verdict (
task.success.verdict) and evidence pack.
- The judge rationale.
- Per-tenant dispute volume in the rolling window (abuse signal).
- The proposed resolution from the rebuttal pipeline.
Actions:
| Action |
Effect |
| Uphold system verdict |
dispute closed; no commercial action |
| Overturn (success → failed) |
refund triggered (ai-agent-abandonment-and-refund-policy), revenue de-recognized in current period |
| Overturn (failed → success) |
rebill at resolution price; customer notified |
| Escalate |
dual-auth review queue for the head of CS |
Rate-limit safeguard: if a tenant's overturn-in-customer-favor rate in the rolling 90-day window exceeds the configured chargeback ceiling, the console refuses single-auth overturns and forces escalation.
Cross-links
ai-agent-sla-credit-automation — case data source.
ai-agent-task-success-tracking — dispute data source.
ai-agent-abandonment-and-refund-policy — refund execution invoked from overturns.
dual-auth-rbac — auth model for the override actions.
Compliance Console (Enhancement)
The back-office adds a Compliance Console surface for control owners and (read-only) for auditors.
| Screen |
Purpose |
Backing service |
| Control Status |
One row per SOC 2 / ISO 27001 / HIPAA control; latest evidence pack, cadence health, owner, open exceptions |
ai-agent-soc2-controls, ai-agent-iso27001-controls, ai-agent-hipaa-security-controls |
| Run Evidence Collection |
Trigger an ad-hoc collector run (auditor sample request) |
ai-agent-evidence-automation |
| Integrity Verification |
Run a chain-witness over a custom window; show drift positions |
ai-agent-audit-log-integrity |
| Drill Cadence |
All drill classes with last pass date, next-due, status |
ai-agent-drill-evidence-and-cadence |
| Approval Completeness |
Latest PI1.1 report; open gaps |
ai-agent-approval-audit-completeness |
| Erasure Requests |
Open and recent erasure requests with proof-pack links |
ai-agent-memory-erasure-proof, saas-tenant-data-portability-and-erasure |
| Exception Register |
All open compliance exceptions sortable by control / severity / target-close |
ai-agent-soc2-controls |
| Auditor Portal |
Read-only auditor sub-surface (separate auth realm, scoped to packs) |
ai-agent-evidence-automation |
All Compliance Console actions are themselves logged onto the action audit log with event_class=compliance_console. Auditor-portal access is logged separately into auditor_access_log (evidence for CC6.1).
Cross-links: ai-agent-evidence-automation, ai-agent-soc2-controls, ai-agent-iso27001-controls, ai-agent-hipaa-security-controls, ai-agent-audit-log-integrity, ai-agent-drill-evidence-and-cadence, ai-agent-approval-audit-completeness, ai-agent-memory-erasure-proof, ai-agent-control-testing-and-attestation.
1---2name: saas-admin-backoffice-tooling3description: Use when designing audited SaaS back-office impersonation, tenant lifecycle, billing overrides, bulk actions, or audit controls.4---56# SaaS Admin / Back-Office Tooling7Acknowledgement: Shared by Peter Bamuhigire, techguypeter.com, +256 784 464178.8910## Required Inputs1112| Input | Required | Use |13|---|---|---|14| Tenant, product, and lifecycle scope | yes | Bound the SaaS decision |15| Current architecture, plans, policies, and constraints | yes | Preserve enforceable behaviour |16| Production data or verified evidence | conditional | Validate thresholds and migrations |1718## Capability and permission contract1920Default to read-only analysis. Change configuration, billing, identity, tenant data, infrastructure, or customer communications only with explicit authority, least-privilege credentials, tenant scope, rollback, and auditable approval. Never expose secrets or cross tenant boundaries.2122## Degraded mode2324If production access, policy, telemetry, or authoritative records are unavailable, produce a labelled design or dry-run plan. Do not claim deployment, reconciliation, deletion, delivery, or measured outcomes; list missing evidence and verification.2526## Decision rules2728| Condition | Action | Stop condition |29|---|---|---|30| Tenant isolation, money, identity, or deletion is affected | Require approval and rollback evidence | Scope or authority is ambiguous |31| Evidence supports a reversible change | Stage, test, and record it | Acceptance checks fail |32| Only partial context is available | Return assumptions and validation | A production claim cannot be verified |3334## Domain Anti-Patterns3536- Applying one tenant's policy or data to another. Fix: enforce tenant scope at every boundary.37- Mutating production from an advisory request. Fix: remain read-only until authority is explicit.38- Inventing limits, prices, metrics, or compliance claims. Fix: use authoritative records or mark them unresolved.39- Shipping without rollback and audit evidence. Fix: stage and retain before/after proof.40- Treating a missing dependency as successful. Fix: name the blocked verification.414243<!-- dual-compat-start -->44## Use When4546- Designing the internal app that the SaaS provider's staff use to operate the platform — tenant ops, billing ops, support ops, compliance ops.47- Replacing SSH + ad-hoc SQL + Stripe Dashboard work with an audited, scriptable, role-based console.48- Adding tenant impersonation to a SaaS (so support can see what the tenant sees, audited and time-boxed).49- Building bulk operations (plan migration of a cohort, mass invite, region migration).50- Implementing the SOC2 / GDPR controls around privileged access.5152## Do Not Use When5354- The task is the customer-facing super-admin or tenant-admin panel (inside one tenant) — use `multi-tenant-saas-architecture` three-panel pattern.55- The task is the control-plane services architecture — use `saas-control-plane-engineering`; this skill is the UI/UX layer on top.56- The task is the audit log schema itself — use `saas-control-plane-engineering` §6.57- The task is general RBAC — use `dual-auth-rbac` and `multi-tenant-saas-architecture`.5859## Required Inputs6061- Control-plane service inventory from `saas-control-plane-engineering`.62- Audit log spec from `saas-control-plane-engineering`.63- Tenant lifecycle states from `saas-control-plane-engineering`.64- Compliance posture (SOC2, ISO27001, HIPAA?) — determines required guardrails.65- Internal roles (super admin, support engineer, billing ops, finance, security, customer success).6667## Workflow68691. Read this `SKILL.md`.702. Map the operations the back-office must support (§2) — tenant ops, billing ops, support ops, compliance ops.713. Design the role model (§3) — internal staff roles + privileged-access controls.724. Build the audited mutation pipeline (§4) — every action passes through guarded handlers.735. Implement impersonation correctly (§5) — time-boxed, justified, audited, visible.746. Build bulk operations safely (§6) — dry-run + idempotency + rate limit + rollback.757. Apply the privileged-access workflow (§7) — break-glass, MFA, approval chains for high-risk actions.768. Apply anti-patterns (§8).7778## Quality Standards7980- Every back-office mutation passes through a single audit-logging middleware. Direct DB writes from staff are forbidden in production.81- Every privileged action requires MFA for the staff user.82- Impersonation is time-boxed (default 30 minutes), justified (free-text required), and visible to the tenant (banner + email notification optionally).83- Bulk operations have a dry-run mode by default; production-mode requires a second approval.84- Every action emits an event to the audit log and to the staff-Slack channel (for transparency among the team).8586## Anti-Patterns8788- Staff doing tenant ops via SSH + `mysql` shell — no audit trail.89- Impersonation that masquerades silently — tenant has no idea their account was accessed.90- Refunds issued directly in Stripe Dashboard — diverges from internal records, no justification captured.91- "Admin can do anything" — no role separation; finance role shouldn't be able to delete tenants; support role shouldn't be able to refund.92- Bulk operations without dry-run — one bad query suspends 500 wrong tenants.93- Privileged actions without MFA — single password compromise = platform compromise.94- Feature-flag overrides per tenant scattered through code instead of in a single override table.9596## Outputs9798- Back-office app design (routes, role map, mutation handlers).99- Impersonation workflow + guardrails.100- Bulk operations playbook (dry-run + approval + rollback).101- Internal roles + permissions matrix.102- Audit-log integration spec.103104## Evidence Produced105106| Category | Artifact | Format | Example |107|----------|----------|--------|---------|108| Correctness | Back-office app spec | Markdown doc with routes + role map | `docs/saas/backoffice-app.md` |109| Security | Privileged-access workflow | Markdown doc | `docs/saas/privileged-access.md` |110| Operability | Bulk-operations playbook | Markdown doc | `docs/saas/bulk-ops-playbook.md` |111112## References113114- `references/impersonation-design.md` — time-boxed, justified, visible impersonation.115- `references/bulk-operations.md` — dry-run + approval + rollback patterns.116- `references/internal-roles-and-permissions.md` — typical role matrix.117- Companion: `saas-control-plane-engineering`, `multi-tenant-saas-architecture`, `dual-auth-rbac`, `vibe-security-skill`, `saas-sso-scim-enterprise-auth`.118- AI incident console: when the platform ships AI features, the back-office must include an **AI incident console** with: feature kill-switch, agent task kill-switch, model-pin, prompt-pin, retrieval index-pin, tool-pin / tool-disable, gateway routing pin, per-tenant feature pause, quota cap, and the evidence-bundle exporter. Each control writes to `ai_incident_mitigation_log` with `(actor, ts, primitive, scope, reason, ticket_id)`. The reason field is mandatory. See `ai-incident-response-runbook` §3 for the primitive contract and `ai-incident-recovery-and-rollback/references/rollback-patterns.md` for the un-pin path.119120<!-- dual-compat-end -->121122## §1 Why the Back-Office Console Matters123124Trio book: between $1M and $10M ARR, the manual-SSH-and-Stripe-Dashboard model collapses. Staff make mistakes; nothing is audited; customer trust erodes. The back-office console is **how operations becomes a discipline rather than a hero activity**.125126For SOC2 and similar: every privileged access must be logged, justified, time-boxed, and traceable to a human. A console enforces this; a shell does not.127128## §2 The Operations Surface129130| Domain | Operations |131|---|---|132| **Tenant ops** | Search; view detail; suspend; restore; archive; hard-delete; transfer ownership; change plan/tier/region; toggle feature-flag overrides; impersonate |133| **User ops** | Search; view memberships; reset password; force logout; revoke sessions; un-suspend account; delete user (GDPR) |134| **Billing ops** | Issue refund; issue credit; apply discount; override plan price; change billing email; pause subscription; reactivate cancelled subscription |135| **Support ops** | View tickets; merge accounts; export tenant data on customer request; bulk re-invite team members; reset onboarding |136| **Compliance ops** | Initiate GDPR erasure; export audit log for tenant; sign DPA; record subprocessor change |137| **Bulk ops** | Plan-migration cohort; region-migration cohort; mass-suspend (e.g., fraud cluster); mass-email broadcast (rare, regulated) |138139## §3 Internal Role Model140141Don't have "is_super_admin" boolean. Have roles:142143| Role | Can | Cannot |144|---|---|---|145| **support_l1** | Search, view, basic password reset, view tickets | Refund, suspend, impersonate |146| **support_l2** | + impersonate (with justification), suspend with reason, issue $50 credit | Hard delete, refund > $200 |147| **billing_ops** | Refund / credit / plan override / change billing email | Suspend / impersonate / delete |148| **engineering_oncall** | Feature-flag overrides, restart workers, replay events | Refund / billing changes |149| **security** | Force logout, revoke sessions, audit-log review, GDPR erasure | Refund / plan override |150| **super_admin** | Anything, but MFA + co-sign on highest-risk (hard delete, bulk-suspend) | — |151152Each role's permissions live in a permissions table; the UI hides irrelevant controls; the API double-checks on every request.153154## §4 The Audited Mutation Pipeline155156Every back-office mutation passes through the same pipeline:157```158HTTP request → AuthN (staff session + MFA) → AuthZ (role + permission)159 → Captures: actor_user_id, actor_ip, justification (required for high-risk)160 → Idempotency check (Idempotency-Key header)161 → Begin DB transaction162 → Pre-state capture (before_state JSON)163 → Mutation164 → Post-state capture (after_state JSON)165 → Audit log INSERT (same transaction)166 → Commit167 → Emit event (Slack notification, downstream services)168```169170A direct DB write that bypasses the pipeline is a **policy violation**, not just a bug.171172## §5 Impersonation173174The most-sensitive feature.175176**Rules:**177- Requires high-trust role (support_l2, super_admin).178- Requires `justification` (free text, audited).179- Time-boxed (default 30 min, max 4 hours).180- Subject tenant sees a banner: "Support is currently viewing your workspace" (configurable per plan — enterprise often wants this visible).181- Optional notification email to tenant primary admin (configurable per plan).182- Impersonation start + end events in audit log.183- All actions during impersonation tagged with `acting_as_user_id`.184185**Implementation pattern:**186```python187def start_impersonation(staff_user_id, target_user_id, justification, duration_min=30):188 require_role(staff_user_id, 'support_l2_or_above')189 require_mfa(staff_user_id)190 require_non_empty(justification)191 assert duration_min <= 240192 session = ImpersonationSession.create(193 staff_user_id=staff_user_id,194 target_user_id=target_user_id,195 justification=justification,196 expires_at=now() + duration_min * 60,197 )198 audit_log('IMPERSONATION_START', actor=staff_user_id, target=target_user_id,199 reason=justification, target_tenant=target_user.tenant_id)200 notify_target_tenant_admin(target_user_id) # configurable201 return session_token202```203204Every action while impersonation is active is double-stamped:205```206audit_log(actor=staff_user_id, acting_as=target_user_id, action='...', target_tenant_id=...)207```208209## §6 Bulk Operations210211Bulk operations are blast-radius nightmares. Pattern:212213```2141. Staff selects cohort (SQL query, CSV upload, saved segment).2152. Dry-run runs the operation in a transaction-marked-readonly; produces a report (what would change, on how many tenants).2163. Staff reviews report; submits production-run with a second-approver MFA confirmation.2174. Production-run executes in batched chunks (e.g., 50 tenants per batch); pauseable; with progress reporting.2185. Per-tenant audit log entries.2196. Rollback snapshot — pre-state captured so undo is possible within N hours.2207. Slack broadcast at start, midpoint, completion.221```222223Examples of safe bulk ops:224- Plan migration: "Move all `legacy_pro` to `pro_v2` with same price-locked Stripe Subscription."225- Region migration: "Move tenants in EU pod from `eu-west-1` to `eu-central-1`."226- Feature-flag rollout: "Enable `new_dashboard` for 10% of `pro` tenants."227228Examples of dangerous bulk ops that need extra scrutiny:229- Mass-suspend (fraud cluster) — requires security sign-off + co-sign.230- Mass-delete — requires CEO/CFO sign-off + delay window.231232## §7 Privileged-Access Workflow233234| Action | Approval | MFA | Co-sign | Time-window |235|---|---|---|---|---|236| Login to back-office | MFA | Yes | No | Per session |237| View tenant | role | Yes | No | — |238| Suspend tenant | role | Yes | No | — |239| Impersonate | role | Yes | No | 30 min default |240| Refund < $100 | billing_ops | Yes | No | — |241| Refund $100-$1000 | billing_ops | Yes | Co-sign by another billing_ops | — |242| Refund > $1000 | billing_ops | Yes | Co-sign by finance lead | — |243| Hard-delete tenant | super_admin | Yes | Co-sign + 24h cooldown | — |244| Mass-suspend cohort | super_admin | Yes | Co-sign by security | — |245246Co-sign mechanism: second staff user approves in the back-office UI within a time window; otherwise the action expires.247248## §8 Anti-Patterns249250- **`admin.html` page on the same app behind a JWT claim** — same auth surface, same code path, same risk; back-office should be a separate auth domain, separate deploy, separate network segment.251- **Impersonation invisible to tenant** — discovered later; trust catastrophe.252- **No co-sign on high-risk actions** — one bad day, one compromised laptop, platform compromised.253- **Justifications optional** — left blank by everyone; audit log is useless.254- **No internal Slack mirror of audit events** — team has no informal accountability layer.255- **Bulk ops without dry-run** — one bad SELECT kills 500 tenants.256- **Refunds via Stripe Dashboard** — bypasses platform record; ledger drift; tax-reporting headaches.257- **Same role for support and billing** — supports cancellation of others' accounts; supports cross-team mistakes.258259## §9 Build Stack Suggestions260261Depending on the SaaS stack:262- **Retool / Internal / Tooljet / Forest Admin / Appsmith** — fast to build, audit-friendly, role-based; great for $1M-$10M ARR stage.263- **Custom React/Vue app talking to control-plane APIs** — once team scales, custom gives the best UX and integration.264- **Django admin + customization** — works for Django stacks; needs extension for audit/impersonation/co-sign.265- **Laravel Nova / Filament** — same for Laravel/PHP stacks.266267Regardless of UI tool, **the API surface is the contract** — UI is just one client; CLI / Slack-bot / scripts are equally valid clients of the back-office API.268269## §10 Agent Ops Console270271When agentic features ship, the back-office gains a new surface: the **Agent Ops Console**. Without it, operating live agents in a multi-tenant SaaS is impossible — a runaway task or a tenant-specific safety incident has no fast remediation.272273### Required views274275| View | Purpose |276|---|---|277| **Live agent tasks** (per-tenant + global) | All in-flight tasks: tenant, feature, state, age, cost so far, last activity |278| **Pending approvals** (per-tenant + global) | All awaiting-approval tasks; staff can override on behalf of tenant admin |279| **Task viewer** (drill-down) | Per-task trace + step list + tool I/O (redacted) + replay link — see `ai-agent-observability-and-replay` |280| **Tenant agent config** | View / edit per-tenant tool allow-list, budgets, entitlement overrides; see audit |281| **Tenant kill-switch panel** | Pause / resume agent for a tenant; per-feature pause; reason required |282| **Agent budget overrides** | Time-boxed overrides with `expires_at`, `reason`, `actor` |283| **Safety incidents** | Live feed of `agent.injection.detected`, `agent.exfil.attempt`, `agent.cross_tenant.attempt`, `agent.escalation.blocked` |284| **Stuck tasks / Abandonment queue** | Tasks that exceeded stall thresholds; resume / abandon / kill |285| **Agent cost dashboard** | Per-tenant cost per completed task; wasted spend share; outliers |286287### Required actions288289| Action | What it does | Audit + safeguards |290|---|---|---|291| **Kill task** | Transitions one task to `KILLED`; runs reversibility compensations | Reason required; emits `agent.task.killed`; tenant notified |292| **Force-pause task** | Transitions to `AWAITING_APPROVAL` (paused-by-staff); resumes when staff or tenant resumes | Reason required; tenant notified |293| **Tenant-wide agent kill-switch** | Sets agent_enabled=false; in-flight tasks complete current step and stop | Reason; tenant notified; expires unless made permanent |294| **Feature-specific kill-switch** | Pauses one agent feature for one tenant | Reason; auto-expire default 24h |295| **Approval override** | Approve / reject an approval on behalf of a tenant admin | Co-sign required for irreversible; visible to tenant |296| **Resume abandoned task** | Restart from last persisted step | Audit |297| **Re-run with new config** | Replay task with candidate prompt/model/tools | Sandbox only — never re-execute side effects in prod |298| **Memory inspection / forget** | Surface per-tenant or per-user agent memories; force-forget on legal request | GDPR audit row written |299| **Tool registry view** | See which tools each tenant has, which versions pinned, deprecations | Read-only here; mutations via control-plane API |300301### Kill-Switch Speed Requirement302303Per-tenant agent kill-switch must flip in **< 5 seconds** wall-time from staff click to runtime enforcement. Implementation: write to a Redis key the runtime checks before each step; DB updated asynchronously for audit.304305```python306# Runtime check before each step307def check_kill_switch(tenant_id):308 if redis.get(f"agent:killed:{tenant_id}") == "1":309 return KillSwitch("tenant_kill_switch")310 return None311```312313The kill-with-rollback runbook (`ai-agent-reversibility-and-blast-radius` §4) describes the compensation cascade. The console wires the button.314315### Required co-signs316317- Tenant-wide agent kill-switch on Enterprise: 2 staff (support engineer + tenant CSM or higher).318- Approval override of irreversible: 2 staff.319- Bulk kill of > 10 tasks: 2 staff + dry-run.320- Memory force-forget: 1 staff + legal ticket reference.321322### Cross-references323324- `ai-agent-runtime-architecture` — provides the task list + state machine the console drives.325- `ai-agent-observability-and-replay` — provides the task viewer + replay link.326- `ai-agent-action-approval-and-hitl` — approval objects the staff override.327- `ai-agent-safety-and-red-team` — safety incident feed.328- `ai-agent-reversibility-and-blast-radius` — kill-with-rollback procedure.329- `ai-agent-memory` — memory inspection / forget surface.330331## §11 Read Next332333- `saas-control-plane-engineering` — the services this console drives.334- `multi-tenant-saas-architecture` — tenant data model + cross-tenant access rules.335- `subscription-billing` — billing operations this console invokes.336- `saas-tenant-data-portability-and-erasure` — GDPR workflows this console initiates.337- `dual-auth-rbac` — internal auth + MFA underpinning.338- `vibe-security-skill` — security baseline for the back-office app.339- `ai-agent-runtime-architecture` — agent task control-plane.340- `ai-agent-observability-and-replay` — task viewer + replay surfaces.341342## §12 SLA-Credit Override + Dispute-Resolution Console (Enhancement)343344For agent products, the back-office needs two purpose-built surfaces beyond the standard tenant/billing views: an **SLA-credit override** console and a **dispute-resolution** console. Both share the principle that every keystroke is dual-authed, audit-logged, and links to the verdict / evidence pack.345346### SLA-credit override console347348Reads from the credit-case ledger produced by `ai-agent-sla-credit-automation`. For each open case the operator sees:349350- The breach event (incident link or detection signal).351- The eligibility decision (auto-approved, auto-denied, needs-review).352- The proposed credit amount (formula + inputs).353- The trace bundle + verdict + evidence pack.354- The per-tenant credit cap remaining for the period.355356Actions:357358| Action | Requires | Audit fields |359|---|---|---|360| Approve as-proposed | single-auth (operator role) | `approved_by`, `approved_at` |361| Adjust amount | dual-auth (operator + manager) | `original_amount`, `adjusted_amount`, `reason_code`, `manager_id` |362| Deny | dual-auth | `reason_code` (from enumerated list: `customer_caused` / `force_majeure` / `excluded_class` / `cap_reached`), `denial_text` |363| Issue mass credit | dual-auth + IC approval link | `incident_id`, `tenant_filter_sql`, `formula_ref` |364365Every action emits to the credit-case audit log; the customer-facing dashboard reflects the outcome in < 60 seconds.366367### Dispute-resolution console368369Reads from the dispute queue produced by `ai-agent-task-success-tracking/references/dispute-resolution.md`. For each open dispute the operator sees:370371- Customer-asserted outcome (success or failure).372- System verdict (`task.success.verdict`) and evidence pack.373- The judge rationale.374- Per-tenant dispute volume in the rolling window (abuse signal).375- The proposed resolution from the rebuttal pipeline.376377Actions:378379| Action | Effect |380|---|---|381| Uphold system verdict | dispute closed; no commercial action |382| Overturn (success → failed) | refund triggered (`ai-agent-abandonment-and-refund-policy`), revenue de-recognized in current period |383| Overturn (failed → success) | rebill at resolution price; customer notified |384| Escalate | dual-auth review queue for the head of CS |385386Rate-limit safeguard: if a tenant's overturn-in-customer-favor rate in the rolling 90-day window exceeds the configured chargeback ceiling, the console refuses single-auth overturns and forces escalation.387388### Cross-links389390- `ai-agent-sla-credit-automation` — case data source.391- `ai-agent-task-success-tracking` — dispute data source.392- `ai-agent-abandonment-and-refund-policy` — refund execution invoked from overturns.393- `dual-auth-rbac` — auth model for the override actions.394395---396397## Compliance Console (Enhancement)398399The back-office adds a **Compliance Console** surface for control owners and (read-only) for auditors.400401| Screen | Purpose | Backing service |402|---|---|---|403| **Control Status** | One row per SOC 2 / ISO 27001 / HIPAA control; latest evidence pack, cadence health, owner, open exceptions | `ai-agent-soc2-controls`, `ai-agent-iso27001-controls`, `ai-agent-hipaa-security-controls` |404| **Run Evidence Collection** | Trigger an ad-hoc collector run (auditor sample request) | `ai-agent-evidence-automation` |405| **Integrity Verification** | Run a chain-witness over a custom window; show drift positions | `ai-agent-audit-log-integrity` |406| **Drill Cadence** | All drill classes with last pass date, next-due, status | `ai-agent-drill-evidence-and-cadence` |407| **Approval Completeness** | Latest PI1.1 report; open gaps | `ai-agent-approval-audit-completeness` |408| **Erasure Requests** | Open and recent erasure requests with proof-pack links | `ai-agent-memory-erasure-proof`, `saas-tenant-data-portability-and-erasure` |409| **Exception Register** | All open compliance exceptions sortable by control / severity / target-close | `ai-agent-soc2-controls` |410| **Auditor Portal** | Read-only auditor sub-surface (separate auth realm, scoped to packs) | `ai-agent-evidence-automation` |411412All Compliance Console actions are themselves logged onto the action audit log with `event_class=compliance_console`. Auditor-portal access is logged separately into `auditor_access_log` (evidence for CC6.1).413414Cross-links: `ai-agent-evidence-automation`, `ai-agent-soc2-controls`, `ai-agent-iso27001-controls`, `ai-agent-hipaa-security-controls`, `ai-agent-audit-log-integrity`, `ai-agent-drill-evidence-and-cadence`, `ai-agent-approval-audit-completeness`, `ai-agent-memory-erasure-proof`, `ai-agent-control-testing-and-attestation`.