SaaS Admin / Back-Office Tooling
Acknowledgement: Shared by Peter Bamuhigire, techguypeter.com, +256 784 464178.
Required Inputs
| Input |
Required |
Use |
| Tenant, product, and lifecycle scope |
yes |
Bound the SaaS decision |
| Current architecture, plans, policies, and constraints |
yes |
Preserve enforceable behaviour |
| Production data or verified evidence |
conditional |
Validate thresholds and migrations |
Capability and permission contract
Default to read-only analysis. Change configuration, billing, identity, tenant data, infrastructure, or customer communications only with explicit authority, least-privilege credentials, tenant scope, rollback, and auditable approval. Never expose secrets or cross tenant boundaries.
Degraded mode
If production access, policy, telemetry, or authoritative records are unavailable, produce a labelled design or dry-run plan. Do not claim deployment, reconciliation, deletion, delivery, or measured outcomes; list missing evidence and verification.
Decision rules
| Condition |
Action |
Stop condition |
| Tenant isolation, money, identity, or deletion is affected |
Require approval and rollback evidence |
Scope or authority is ambiguous |
| Evidence supports a reversible change |
Stage, test, and record it |
Acceptance checks fail |
| Only partial context is available |
Return assumptions and validation |
A production claim cannot be verified |
Domain Anti-Patterns
- Applying one tenant's policy or data to another. Fix: enforce tenant scope at every boundary.
- Mutating production from an advisory request. Fix: remain read-only until authority is explicit.
- Inventing limits, prices, metrics, or compliance claims. Fix: use authoritative records or mark them unresolved.
- Shipping without rollback and audit evidence. Fix: stage and retain before/after proof.
- Treating a missing dependency as successful. Fix: name the blocked verification.
Use When
- Designing the internal app that the SaaS provider's staff use to operate the platform — tenant ops, billing ops, support ops, compliance ops.
- Replacing SSH + ad-hoc SQL + Stripe Dashboard work with an audited, scriptable, role-based console.
- Adding tenant impersonation to a SaaS (so support can see what the tenant sees, audited and time-boxed).
- Building bulk operations (plan migration of a cohort, mass invite, region migration).
- Implementing the SOC2 / GDPR controls around privileged access.
Do Not Use When
- The task is the customer-facing super-admin or tenant-admin panel (inside one tenant) — use
multi-tenant-saas-architecture three-panel pattern.
- The task is the control-plane services architecture — use
saas-control-plane-engineering; this skill is the UI/UX layer on top.
- The task is the audit log schema itself — use
saas-control-plane-engineering §6.
- The task is general RBAC — use
dual-auth-rbac and multi-tenant-saas-architecture.
Required Inputs
- Control-plane service inventory from
saas-control-plane-engineering.
- Audit log spec from
saas-control-plane-engineering.
- Tenant lifecycle states from
saas-control-plane-engineering.
- Compliance posture (SOC2, ISO27001, HIPAA?) — determines required guardrails.
- Internal roles (super admin, support engineer, billing ops, finance, security, customer success).
Workflow
- Read this
SKILL.md.
- Map the operations the back-office must support (§2) — tenant ops, billing ops, support ops, compliance ops.
- Design the role model (§3) — internal staff roles + privileged-access controls.
- Build the audited mutation pipeline (§4) — every action passes through guarded handlers.
- Implement impersonation correctly (§5) — time-boxed, justified, audited, visible.
- Build bulk operations safely (§6) — dry-run + idempotency + rate limit + rollback.
- Apply the privileged-access workflow (§7) — break-glass, MFA, approval chains for high-risk actions.
- Apply anti-patterns (§8).
Quality Standards
- Every back-office mutation passes through a single audit-logging middleware. Direct DB writes from staff are forbidden in production.
- Every privileged action requires MFA for the staff user.
- Impersonation is time-boxed (default 30 minutes), justified (free-text required), and visible to the tenant (banner + email notification optionally).
- Bulk operations have a dry-run mode by default; production-mode requires a second approval.
- Every action emits an event to the audit log and to the staff-Slack channel (for transparency among the team).
Anti-Patterns
- Staff doing tenant ops via SSH +
mysql shell — no audit trail.
- Impersonation that masquerades silently — tenant has no idea their account was accessed.
- Refunds issued directly in Stripe Dashboard — diverges from internal records, no justification captured.
- "Admin can do anything" — no role separation; finance role shouldn't be able to delete tenants; support role shouldn't be able to refund.
- Bulk operations without dry-run — one bad query suspends 500 wrong tenants.
- Privileged actions without MFA — single password compromise = platform compromise.
- Feature-flag overrides per tenant scattered through code instead of in a single override table.
Outputs
- Back-office app design (routes, role map, mutation handlers).
- Impersonation workflow + guardrails.
- Bulk operations playbook (dry-run + approval + rollback).
- Internal roles + permissions matrix.
- Audit-log integration spec.
Evidence Produced
| Category |
Artifact |
Format |
Example |
| Correctness |
Back-office app spec |
Markdown doc with routes + role map |
docs/saas/backoffice-app.md |
| Security |
Privileged-access workflow |
Markdown doc |
docs/saas/privileged-access.md |
| Operability |
Bulk-operations playbook |
Markdown doc |
docs/saas/bulk-ops-playbook.md |
References
references/impersonation-design.md — time-boxed, justified, visible impersonation.
references/bulk-operations.md — dry-run + approval + rollback patterns.
references/internal-roles-and-permissions.md — typical role matrix.
- Companion:
saas-control-plane-engineering, multi-tenant-saas-architecture, dual-auth-rbac, vibe-security-skill, saas-sso-scim-enterprise-auth.
- AI incident console: when the platform ships AI features, the back-office must include an AI incident console with: feature kill-switch, agent task kill-switch, model-pin, prompt-pin, retrieval index-pin, tool-pin / tool-disable, gateway routing pin, per-tenant feature pause, quota cap, and the evidence-bundle exporter. Each control writes to
ai_incident_mitigation_log with (actor, ts, primitive, scope, reason, ticket_id). The reason field is mandatory. See ai-incident-response-runbook §3 for the primitive contract and ai-incident-recovery-and-rollback/references/rollback-patterns.md for the un-pin path.
§1 Why the Back-Office Console Matters
Trio book: between $1M and $10M ARR, the manual-SSH-and-Stripe-Dashboard model collapses. Staff make mistakes; nothing is audited; customer trust erodes. The back-office console is how operations becomes a discipline rather than a hero activity.
For SOC2 and similar: every privileged access must be logged, justified, time-boxed, and traceable to a human. A console enforces this; a shell does not.
§2 The Operations Surface
| Domain |
Operations |
| Tenant ops |
Search; view detail; suspend; restore; archive; hard-delete; transfer ownership; change plan/tier/region; toggle feature-flag overrides; impersonate |
| User ops |
Search; view memberships; reset password; force logout; revoke sessions; un-suspend account; delete user (GDPR) |
| Billing ops |
Issue refund; issue credit; apply discount; override plan price; change billing email; pause subscription; reactivate cancelled subscription |
| Support ops |
View tickets; merge accounts; export tenant data on customer request; bulk re-invite team members; reset onboarding |
| Compliance ops |
Initiate GDPR erasure; export audit log for tenant; sign DPA; record subprocessor change |
| Bulk ops |
Plan-migration cohort; region-migration cohort; mass-suspend (e.g., fraud cluster); mass-email broadcast (rare, regulated) |
§3 Internal Role Model
Don't have "is_super_admin" boolean. Have roles:
| Role |
Can |
Cannot |
| support_l1 |
Search, view, basic password reset, view tickets |
Refund, suspend, impersonate |
| support_l2 |
+ impersonate (with justification), suspend with reason, issue $50 credit |
Hard delete, refund > $200 |
| billing_ops |
Refund / credit / plan override / change billing email |
Suspend / impersonate / delete |
| engineering_oncall |
Feature-flag overrides, restart workers, replay events |
Refund / billing changes |
| security |
Force logout, revoke sessions, audit-log review, GDPR erasure |
Refund / plan override |
| super_admin |
Anything, but MFA + co-sign on highest-risk (hard delete, bulk-suspend) |
— |
Each role's permissions live in a permissions table; the UI hides irrelevant controls; the API double-checks on every request.
§4 The Audited Mutation Pipeline
Every back-office mutation passes through the same pipeline:
HTTP request → AuthN (staff session + MFA) → AuthZ (role + permission)
→ Captures: actor_user_id, actor_ip, justification (required for high-risk)
→ Idempotency check (Idempotency-Key header)
→ Begin DB transaction
→ Pre-state capture (before_state JSON)
→ Mutation
→ Post-state capture (after_state JSON)
→ Audit log INSERT (same transaction)
→ Commit
→ Emit event (Slack notification, downstream services)
A direct DB write that bypasses the pipeline is a policy violation, not just a bug.
§5 Impersonation
The most-sensitive feature.
Rules:
- Requires high-trust role (support_l2, super_admin).
- Requires
justification (free text, audited).
- Time-boxed (default 30 min, max 4 hours).
- Subject tenant sees a banner: "Support is currently viewing your workspace" (configurable per plan — enterprise often wants this visible).
- Optional notification email to tenant primary admin (configurable per plan).
- Impersonation start + end events in audit log.
- All actions during impersonation tagged with
acting_as_user_id.
Implementation pattern:
def start_impersonation(staff_user_id, target_user_id, justification, duration_min=30):
require_role(staff_user_id, 'support_l2_or_above')
require_mfa(staff_user_id)
require_non_empty(justification)
assert duration_min <= 240
session = ImpersonationSession.create(
staff_user_id=staff_user_id,
target_user_id=target_user_id,
justification=justification,
expires_at=now() + duration_min * 60,
)
audit_log('IMPERSONATION_START', actor=staff_user_id, target=target_user_id,
reason=justification, target_tenant=target_user.tenant_id)
notify_target_tenant_admin(target_user_id) # configurable
return session_token
Every action while impersonation is active is double-stamped:
audit_log(actor=staff_user_id, acting_as=target_user_id, action='...', target_tenant_id=...)
§6 Bulk Operations
Bulk operations are blast-radius nightmares. Pattern:
1. Staff selects cohort (SQL query, CSV upload, saved segment).
2. Dry-run runs the operation in a transaction-marked-readonly; produces a report (what would change, on how many tenants).
3. Staff reviews report; submits production-run with a second-approver MFA confirmation.
4. Production-run executes in batched chunks (e.g., 50 tenants per batch); pauseable; with progress reporting.
5. Per-tenant audit log entries.
6. Rollback snapshot — pre-state captured so undo is possible within N hours.
7. Slack broadcast at start, midpoint, completion.
Examples of safe bulk ops:
- Plan migration: "Move all
legacy_pro to pro_v2 with same price-locked Stripe Subscription."
- Region migration: "Move tenants in EU pod from
eu-west-1 to eu-central-1."
- Feature-flag rollout: "Enable
new_dashboard for 10% of pro tenants."
Examples of dangerous bulk ops that need extra scrutiny:
- Mass-suspend (fraud cluster) — requires security sign-off + co-sign.
- Mass-delete — requires CEO/CFO sign-off + delay window.
§7 Privileged-Access Workflow
| Action |
Approval |
MFA |
Co-sign |
Time-window |
| Login to back-office |
MFA |
Yes |
No |
Per session |
| View tenant |
role |
Yes |
No |
— |
| Suspend tenant |
role |
Yes |
No |
— |
| Impersonate |
role |
Yes |
No |
30 min default |
| Refund < $100 |
billing_ops |
Yes |
No |
— |
| Refund $100-$1000 |
billing_ops |
Yes |
Co-sign by another billing_ops |
— |
| Refund > $1000 |
billing_ops |
Yes |
Co-sign by finance lead |
— |
| Hard-delete tenant |
super_admin |
Yes |
Co-sign + 24h cooldown |
— |
| Mass-suspend cohort |
super_admin |
Yes |
Co-sign by security |
— |
Co-sign mechanism: second staff user approves in the back-office UI within a time window; otherwise the action expires.
§8 Anti-Patterns
admin.html page on the same app behind a JWT claim — same auth surface, same code path, same risk; back-office should be a separate auth domain, separate deploy, separate network segment.
- Impersonation invisible to tenant — discovered later; trust catastrophe.
- No co-sign on high-risk actions — one bad day, one compromised laptop, platform compromised.
- Justifications optional — left blank by everyone; audit log is useless.
- No internal Slack mirror of audit events — team has no informal accountability layer.
- Bulk ops without dry-run — one bad SELECT kills 500 tenants.
- Refunds via Stripe Dashboard — bypasses platform record; ledger drift; tax-reporting headaches.
- Same role for support and billing — supports cancellation of others' accounts; supports cross-team mistakes.
§9 Build Stack Suggestions
Depending on the SaaS stack:
- Retool / Internal / Tooljet / Forest Admin / Appsmith — fast to build, audit-friendly, role-based; great for $1M-$10M ARR stage.
- Custom React/Vue app talking to control-plane APIs — once team scales, custom gives the best UX and integration.
- Django admin + customization — works for Django stacks; needs extension for audit/impersonation/co-sign.
- Laravel Nova / Filament — same for Laravel/PHP stacks.
Regardless of UI tool, the API surface is the contract — UI is just one client; CLI / Slack-bot / scripts are equally valid clients of the back-office API.
§10 Agent Ops Console
When agentic features ship, the back-office gains a new surface: the Agent Ops Console. Without it, operating live agents in a multi-tenant SaaS is impossible — a runaway task or a tenant-specific safety incident has no fast remediation.
Required views
| View |
Purpose |
| Live agent tasks (per-tenant + global) |
All in-flight tasks: tenant, feature, state, age, cost so far, last activity |
| Pending approvals (per-tenant + global) |
All awaiting-approval tasks; staff can override on behalf of tenant admin |
| Task viewer (drill-down) |
Per-task trace + step list + tool I/O (redacted) + replay link — see ai-agent-observability-and-replay |
| Tenant agent config |
View / edit per-tenant tool allow-list, budgets, entitlement overrides; see audit |
| Tenant kill-switch panel |
Pause / resume agent for a tenant; per-feature pause; reason required |
| Agent budget overrides |
Time-boxed overrides with expires_at, reason, actor |
| Safety incidents |
Live feed of agent.injection.detected, agent.exfil.attempt, agent.cross_tenant.attempt, agent.escalation.blocked |
| Stuck tasks / Abandonment queue |
Tasks that exceeded stall thresholds; resume / abandon / kill |
| Agent cost dashboard |
Per-tenant cost per completed task; wasted spend share; outliers |
Required actions
| Action |
What it does |
Audit + safeguards |
| Kill task |
Transitions one task to KILLED; runs reversibility compensations |
Reason required; emits agent.task.killed; tenant notified |
| Force-pause task |
Transitions to AWAITING_APPROVAL (paused-by-staff); resumes when staff or tenant resumes |
Reason required; tenant notified |
| Tenant-wide agent kill-switch |
Sets agent_enabled=false; in-flight tasks complete current step and stop |
Reason; tenant notified; expires unless made permanent |
| Feature-specific kill-switch |
Pauses one agent feature for one tenant |
Reason; auto-expire default 24h |
| Approval override |
Approve / reject an approval on behalf of a tenant admin |
Co-sign required for irreversible; visible to tenant |
| Resume abandoned task |
Restart from last persisted step |
Audit |
| Re-run with new config |
Replay task with candidate prompt/model/tools |
Sandbox only — never re-execute side effects in prod |
| Memory inspection / forget |
Surface per-tenant or per-user agent memories; force-forget on legal request |
GDPR audit row written |
| Tool registry view |
See which tools each tenant has, which versions pinned, deprecations |
Read-only here; mutations via control-plane API |
Kill-Switch Speed Requirement
Per-tenant agent kill-switch must flip in < 5 seconds wall-time from staff click to runtime enforcement. Implementation: write to a Redis key the runtime checks before each step; DB updated asynchronously for audit.
# Runtime check before each step
def check_kill_switch(tenant_id):
if redis.get(f"agent:killed:{tenant_id}") == "1":
return KillSwitch("tenant_kill_switch")
return None
The kill-with-rollback runbook (ai-agent-reversibility-and-blast-radius §4) describes the compensation cascade. The console wires the button.
Required co-signs
- Tenant-wide agent kill-switch on Enterprise: 2 staff (support engineer + tenant CSM or higher).
- Approval override of irreversible: 2 staff.
- Bulk kill of > 10 tasks: 2 staff + dry-run.
- Memory force-forget: 1 staff + legal ticket reference.
Cross-references
ai-agent-runtime-architecture — provides the task list + state machine the console drives.
ai-agent-observability-and-replay — provides the task viewer + replay link.
ai-agent-action-approval-and-hitl — approval objects the staff override.
ai-agent-safety-and-red-team — safety incident feed.
ai-agent-reversibility-and-blast-radius — kill-with-rollback procedure.
ai-agent-memory — memory inspection / forget surface.
§11 Read Next
saas-control-plane-engineering — the services this console drives.
multi-tenant-saas-architecture — tenant data model + cross-tenant access rules.
subscription-billing — billing operations this console invokes.
saas-tenant-data-portability-and-erasure — GDPR workflows this console initiates.
dual-auth-rbac — internal auth + MFA underpinning.
vibe-security-skill — security baseline for the back-office app.
ai-agent-runtime-architecture — agent task control-plane.
ai-agent-observability-and-replay — task viewer + replay surfaces.
§12 SLA-Credit Override + Dispute-Resolution Console (Enhancement)
For agent products, the back-office needs two purpose-built surfaces beyond the standard tenant/billing views: an SLA-credit override console and a dispute-resolution console. Both share the principle that every keystroke is dual-authed, audit-logged, and links to the verdict / evidence pack.
SLA-credit override console
Reads from the credit-case ledger produced by ai-agent-sla-credit-automation. For each open case the operator sees:
- The breach event (incident link or detection signal).
- The eligibility decision (auto-approved, auto-denied, needs-review).
- The proposed credit amount (formula + inputs).
- The trace bundle + verdict + evidence pack.
- The per-tenant credit cap remaining for the period.
Actions:
| Action |
Requires |
Audit fields |
| Approve as-proposed |
single-auth (operator role) |
approved_by, approved_at |
| Adjust amount |
dual-auth (operator + manager) |
original_amount, adjusted_amount, reason_code, manager_id |
| Deny |
dual-auth |
reason_code (from enumerated list: customer_caused / force_majeure / excluded_class / cap_reached), denial_text |
| Issue mass credit |
dual-auth + IC approval link |
incident_id, tenant_filter_sql, formula_ref |
Every action emits to the credit-case audit log; the customer-facing dashboard reflects the outcome in < 60 seconds.
Dispute-resolution console
Reads from the dispute queue produced by ai-agent-task-success-tracking/references/dispute-resolution.md. For each open dispute the operator sees:
- Customer-asserted outcome (success or failure).
- System verdict (
task.success.verdict) and evidence pack.
- The judge rationale.
- Per-tenant dispute volume in the rolling window (abuse signal).
- The proposed resolution from the rebuttal pipeline.
Actions:
| Action |
Effect |
| Uphold system verdict |
dispute closed; no commercial action |
| Overturn (success → failed) |
refund triggered (ai-agent-abandonment-and-refund-policy), revenue de-recognized in current period |
| Overturn (failed → success) |
rebill at resolution price; customer notified |
| Escalate |
dual-auth review queue for the head of CS |
Rate-limit safeguard: if a tenant's overturn-in-customer-favor rate in the rolling 90-day window exceeds the configured chargeback ceiling, the console refuses single-auth overturns and forces escalation.
Cross-links
ai-agent-sla-credit-automation — case data source.
ai-agent-task-success-tracking — dispute data source.
ai-agent-abandonment-and-refund-policy — refund execution invoked from overturns.
dual-auth-rbac — auth model for the override actions.
Compliance Console (Enhancement)
The back-office adds a Compliance Console surface for control owners and (read-only) for auditors.
| Screen |
Purpose |
Backing service |
| Control Status |
One row per SOC 2 / ISO 27001 / HIPAA control; latest evidence pack, cadence health, owner, open exceptions |
ai-agent-soc2-controls, ai-agent-iso27001-controls, ai-agent-hipaa-security-controls |
| Run Evidence Collection |
Trigger an ad-hoc collector run (auditor sample request) |
ai-agent-evidence-automation |
| Integrity Verification |
Run a chain-witness over a custom window; show drift positions |
ai-agent-audit-log-integrity |
| Drill Cadence |
All drill classes with last pass date, next-due, status |
ai-agent-drill-evidence-and-cadence |
| Approval Completeness |
Latest PI1.1 report; open gaps |
ai-agent-approval-audit-completeness |
| Erasure Requests |
Open and recent erasure requests with proof-pack links |
ai-agent-memory-erasure-proof, saas-tenant-data-portability-and-erasure |
| Exception Register |
All open compliance exceptions sortable by control / severity / target-close |
ai-agent-soc2-controls |
| Auditor Portal |
Read-only auditor sub-surface (separate auth realm, scoped to packs) |
ai-agent-evidence-automation |
All Compliance Console actions are themselves logged onto the action audit log with event_class=compliance_console. Auditor-portal access is logged separately into auditor_access_log (evidence for CC6.1).
Cross-links: ai-agent-evidence-automation, ai-agent-soc2-controls, ai-agent-iso27001-controls, ai-agent-hipaa-security-controls, ai-agent-audit-log-integrity, ai-agent-drill-evidence-and-cadence, ai-agent-approval-audit-completeness, ai-agent-memory-erasure-proof, ai-agent-control-testing-and-attestation.
1---2name: saas-admin-backoffice-tooling3description: Use when designing audited SaaS back-office impersonation, tenant lifecycle, billing overrides, bulk actions, or audit controls.4metadata:5 portable: true6 compatible_with:7 - claude-code8 - codex9---1011# SaaS Admin / Back-Office Tooling12Acknowledgement: Shared by Peter Bamuhigire, techguypeter.com, +256 784 464178.131415## Required Inputs1617| Input | Required | Use |18|---|---|---|19| Tenant, product, and lifecycle scope | yes | Bound the SaaS decision |20| Current architecture, plans, policies, and constraints | yes | Preserve enforceable behaviour |21| Production data or verified evidence | conditional | Validate thresholds and migrations |2223## Capability and permission contract2425Default to read-only analysis. Change configuration, billing, identity, tenant data, infrastructure, or customer communications only with explicit authority, least-privilege credentials, tenant scope, rollback, and auditable approval. Never expose secrets or cross tenant boundaries.2627## Degraded mode2829If production access, policy, telemetry, or authoritative records are unavailable, produce a labelled design or dry-run plan. Do not claim deployment, reconciliation, deletion, delivery, or measured outcomes; list missing evidence and verification.3031## Decision rules3233| Condition | Action | Stop condition |34|---|---|---|35| Tenant isolation, money, identity, or deletion is affected | Require approval and rollback evidence | Scope or authority is ambiguous |36| Evidence supports a reversible change | Stage, test, and record it | Acceptance checks fail |37| Only partial context is available | Return assumptions and validation | A production claim cannot be verified |3839## Domain Anti-Patterns4041- Applying one tenant's policy or data to another. Fix: enforce tenant scope at every boundary.42- Mutating production from an advisory request. Fix: remain read-only until authority is explicit.43- Inventing limits, prices, metrics, or compliance claims. Fix: use authoritative records or mark them unresolved.44- Shipping without rollback and audit evidence. Fix: stage and retain before/after proof.45- Treating a missing dependency as successful. Fix: name the blocked verification.464748<!-- dual-compat-start -->49## Use When5051- Designing the internal app that the SaaS provider's staff use to operate the platform — tenant ops, billing ops, support ops, compliance ops.52- Replacing SSH + ad-hoc SQL + Stripe Dashboard work with an audited, scriptable, role-based console.53- Adding tenant impersonation to a SaaS (so support can see what the tenant sees, audited and time-boxed).54- Building bulk operations (plan migration of a cohort, mass invite, region migration).55- Implementing the SOC2 / GDPR controls around privileged access.5657## Do Not Use When5859- The task is the customer-facing super-admin or tenant-admin panel (inside one tenant) — use `multi-tenant-saas-architecture` three-panel pattern.60- The task is the control-plane services architecture — use `saas-control-plane-engineering`; this skill is the UI/UX layer on top.61- The task is the audit log schema itself — use `saas-control-plane-engineering` §6.62- The task is general RBAC — use `dual-auth-rbac` and `multi-tenant-saas-architecture`.6364## Required Inputs6566- Control-plane service inventory from `saas-control-plane-engineering`.67- Audit log spec from `saas-control-plane-engineering`.68- Tenant lifecycle states from `saas-control-plane-engineering`.69- Compliance posture (SOC2, ISO27001, HIPAA?) — determines required guardrails.70- Internal roles (super admin, support engineer, billing ops, finance, security, customer success).7172## Workflow73741. Read this `SKILL.md`.752. Map the operations the back-office must support (§2) — tenant ops, billing ops, support ops, compliance ops.763. Design the role model (§3) — internal staff roles + privileged-access controls.774. Build the audited mutation pipeline (§4) — every action passes through guarded handlers.785. Implement impersonation correctly (§5) — time-boxed, justified, audited, visible.796. Build bulk operations safely (§6) — dry-run + idempotency + rate limit + rollback.807. Apply the privileged-access workflow (§7) — break-glass, MFA, approval chains for high-risk actions.818. Apply anti-patterns (§8).8283## Quality Standards8485- Every back-office mutation passes through a single audit-logging middleware. Direct DB writes from staff are forbidden in production.86- Every privileged action requires MFA for the staff user.87- Impersonation is time-boxed (default 30 minutes), justified (free-text required), and visible to the tenant (banner + email notification optionally).88- Bulk operations have a dry-run mode by default; production-mode requires a second approval.89- Every action emits an event to the audit log and to the staff-Slack channel (for transparency among the team).9091## Anti-Patterns9293- Staff doing tenant ops via SSH + `mysql` shell — no audit trail.94- Impersonation that masquerades silently — tenant has no idea their account was accessed.95- Refunds issued directly in Stripe Dashboard — diverges from internal records, no justification captured.96- "Admin can do anything" — no role separation; finance role shouldn't be able to delete tenants; support role shouldn't be able to refund.97- Bulk operations without dry-run — one bad query suspends 500 wrong tenants.98- Privileged actions without MFA — single password compromise = platform compromise.99- Feature-flag overrides per tenant scattered through code instead of in a single override table.100101## Outputs102103- Back-office app design (routes, role map, mutation handlers).104- Impersonation workflow + guardrails.105- Bulk operations playbook (dry-run + approval + rollback).106- Internal roles + permissions matrix.107- Audit-log integration spec.108109## Evidence Produced110111| Category | Artifact | Format | Example |112|----------|----------|--------|---------|113| Correctness | Back-office app spec | Markdown doc with routes + role map | `docs/saas/backoffice-app.md` |114| Security | Privileged-access workflow | Markdown doc | `docs/saas/privileged-access.md` |115| Operability | Bulk-operations playbook | Markdown doc | `docs/saas/bulk-ops-playbook.md` |116117## References118119- `references/impersonation-design.md` — time-boxed, justified, visible impersonation.120- `references/bulk-operations.md` — dry-run + approval + rollback patterns.121- `references/internal-roles-and-permissions.md` — typical role matrix.122- Companion: `saas-control-plane-engineering`, `multi-tenant-saas-architecture`, `dual-auth-rbac`, `vibe-security-skill`, `saas-sso-scim-enterprise-auth`.123- AI incident console: when the platform ships AI features, the back-office must include an **AI incident console** with: feature kill-switch, agent task kill-switch, model-pin, prompt-pin, retrieval index-pin, tool-pin / tool-disable, gateway routing pin, per-tenant feature pause, quota cap, and the evidence-bundle exporter. Each control writes to `ai_incident_mitigation_log` with `(actor, ts, primitive, scope, reason, ticket_id)`. The reason field is mandatory. See `ai-incident-response-runbook` §3 for the primitive contract and `ai-incident-recovery-and-rollback/references/rollback-patterns.md` for the un-pin path.124125<!-- dual-compat-end -->126127## §1 Why the Back-Office Console Matters128129Trio book: between $1M and $10M ARR, the manual-SSH-and-Stripe-Dashboard model collapses. Staff make mistakes; nothing is audited; customer trust erodes. The back-office console is **how operations becomes a discipline rather than a hero activity**.130131For SOC2 and similar: every privileged access must be logged, justified, time-boxed, and traceable to a human. A console enforces this; a shell does not.132133## §2 The Operations Surface134135| Domain | Operations |136|---|---|137| **Tenant ops** | Search; view detail; suspend; restore; archive; hard-delete; transfer ownership; change plan/tier/region; toggle feature-flag overrides; impersonate |138| **User ops** | Search; view memberships; reset password; force logout; revoke sessions; un-suspend account; delete user (GDPR) |139| **Billing ops** | Issue refund; issue credit; apply discount; override plan price; change billing email; pause subscription; reactivate cancelled subscription |140| **Support ops** | View tickets; merge accounts; export tenant data on customer request; bulk re-invite team members; reset onboarding |141| **Compliance ops** | Initiate GDPR erasure; export audit log for tenant; sign DPA; record subprocessor change |142| **Bulk ops** | Plan-migration cohort; region-migration cohort; mass-suspend (e.g., fraud cluster); mass-email broadcast (rare, regulated) |143144## §3 Internal Role Model145146Don't have "is_super_admin" boolean. Have roles:147148| Role | Can | Cannot |149|---|---|---|150| **support_l1** | Search, view, basic password reset, view tickets | Refund, suspend, impersonate |151| **support_l2** | + impersonate (with justification), suspend with reason, issue $50 credit | Hard delete, refund > $200 |152| **billing_ops** | Refund / credit / plan override / change billing email | Suspend / impersonate / delete |153| **engineering_oncall** | Feature-flag overrides, restart workers, replay events | Refund / billing changes |154| **security** | Force logout, revoke sessions, audit-log review, GDPR erasure | Refund / plan override |155| **super_admin** | Anything, but MFA + co-sign on highest-risk (hard delete, bulk-suspend) | — |156157Each role's permissions live in a permissions table; the UI hides irrelevant controls; the API double-checks on every request.158159## §4 The Audited Mutation Pipeline160161Every back-office mutation passes through the same pipeline:162```163HTTP request → AuthN (staff session + MFA) → AuthZ (role + permission)164 → Captures: actor_user_id, actor_ip, justification (required for high-risk)165 → Idempotency check (Idempotency-Key header)166 → Begin DB transaction167 → Pre-state capture (before_state JSON)168 → Mutation169 → Post-state capture (after_state JSON)170 → Audit log INSERT (same transaction)171 → Commit172 → Emit event (Slack notification, downstream services)173```174175A direct DB write that bypasses the pipeline is a **policy violation**, not just a bug.176177## §5 Impersonation178179The most-sensitive feature.180181**Rules:**182- Requires high-trust role (support_l2, super_admin).183- Requires `justification` (free text, audited).184- Time-boxed (default 30 min, max 4 hours).185- Subject tenant sees a banner: "Support is currently viewing your workspace" (configurable per plan — enterprise often wants this visible).186- Optional notification email to tenant primary admin (configurable per plan).187- Impersonation start + end events in audit log.188- All actions during impersonation tagged with `acting_as_user_id`.189190**Implementation pattern:**191```python192def start_impersonation(staff_user_id, target_user_id, justification, duration_min=30):193 require_role(staff_user_id, 'support_l2_or_above')194 require_mfa(staff_user_id)195 require_non_empty(justification)196 assert duration_min <= 240197 session = ImpersonationSession.create(198 staff_user_id=staff_user_id,199 target_user_id=target_user_id,200 justification=justification,201 expires_at=now() + duration_min * 60,202 )203 audit_log('IMPERSONATION_START', actor=staff_user_id, target=target_user_id,204 reason=justification, target_tenant=target_user.tenant_id)205 notify_target_tenant_admin(target_user_id) # configurable206 return session_token207```208209Every action while impersonation is active is double-stamped:210```211audit_log(actor=staff_user_id, acting_as=target_user_id, action='...', target_tenant_id=...)212```213214## §6 Bulk Operations215216Bulk operations are blast-radius nightmares. Pattern:217218```2191. Staff selects cohort (SQL query, CSV upload, saved segment).2202. Dry-run runs the operation in a transaction-marked-readonly; produces a report (what would change, on how many tenants).2213. Staff reviews report; submits production-run with a second-approver MFA confirmation.2224. Production-run executes in batched chunks (e.g., 50 tenants per batch); pauseable; with progress reporting.2235. Per-tenant audit log entries.2246. Rollback snapshot — pre-state captured so undo is possible within N hours.2257. Slack broadcast at start, midpoint, completion.226```227228Examples of safe bulk ops:229- Plan migration: "Move all `legacy_pro` to `pro_v2` with same price-locked Stripe Subscription."230- Region migration: "Move tenants in EU pod from `eu-west-1` to `eu-central-1`."231- Feature-flag rollout: "Enable `new_dashboard` for 10% of `pro` tenants."232233Examples of dangerous bulk ops that need extra scrutiny:234- Mass-suspend (fraud cluster) — requires security sign-off + co-sign.235- Mass-delete — requires CEO/CFO sign-off + delay window.236237## §7 Privileged-Access Workflow238239| Action | Approval | MFA | Co-sign | Time-window |240|---|---|---|---|---|241| Login to back-office | MFA | Yes | No | Per session |242| View tenant | role | Yes | No | — |243| Suspend tenant | role | Yes | No | — |244| Impersonate | role | Yes | No | 30 min default |245| Refund < $100 | billing_ops | Yes | No | — |246| Refund $100-$1000 | billing_ops | Yes | Co-sign by another billing_ops | — |247| Refund > $1000 | billing_ops | Yes | Co-sign by finance lead | — |248| Hard-delete tenant | super_admin | Yes | Co-sign + 24h cooldown | — |249| Mass-suspend cohort | super_admin | Yes | Co-sign by security | — |250251Co-sign mechanism: second staff user approves in the back-office UI within a time window; otherwise the action expires.252253## §8 Anti-Patterns254255- **`admin.html` page on the same app behind a JWT claim** — same auth surface, same code path, same risk; back-office should be a separate auth domain, separate deploy, separate network segment.256- **Impersonation invisible to tenant** — discovered later; trust catastrophe.257- **No co-sign on high-risk actions** — one bad day, one compromised laptop, platform compromised.258- **Justifications optional** — left blank by everyone; audit log is useless.259- **No internal Slack mirror of audit events** — team has no informal accountability layer.260- **Bulk ops without dry-run** — one bad SELECT kills 500 tenants.261- **Refunds via Stripe Dashboard** — bypasses platform record; ledger drift; tax-reporting headaches.262- **Same role for support and billing** — supports cancellation of others' accounts; supports cross-team mistakes.263264## §9 Build Stack Suggestions265266Depending on the SaaS stack:267- **Retool / Internal / Tooljet / Forest Admin / Appsmith** — fast to build, audit-friendly, role-based; great for $1M-$10M ARR stage.268- **Custom React/Vue app talking to control-plane APIs** — once team scales, custom gives the best UX and integration.269- **Django admin + customization** — works for Django stacks; needs extension for audit/impersonation/co-sign.270- **Laravel Nova / Filament** — same for Laravel/PHP stacks.271272Regardless of UI tool, **the API surface is the contract** — UI is just one client; CLI / Slack-bot / scripts are equally valid clients of the back-office API.273274## §10 Agent Ops Console275276When agentic features ship, the back-office gains a new surface: the **Agent Ops Console**. Without it, operating live agents in a multi-tenant SaaS is impossible — a runaway task or a tenant-specific safety incident has no fast remediation.277278### Required views279280| View | Purpose |281|---|---|282| **Live agent tasks** (per-tenant + global) | All in-flight tasks: tenant, feature, state, age, cost so far, last activity |283| **Pending approvals** (per-tenant + global) | All awaiting-approval tasks; staff can override on behalf of tenant admin |284| **Task viewer** (drill-down) | Per-task trace + step list + tool I/O (redacted) + replay link — see `ai-agent-observability-and-replay` |285| **Tenant agent config** | View / edit per-tenant tool allow-list, budgets, entitlement overrides; see audit |286| **Tenant kill-switch panel** | Pause / resume agent for a tenant; per-feature pause; reason required |287| **Agent budget overrides** | Time-boxed overrides with `expires_at`, `reason`, `actor` |288| **Safety incidents** | Live feed of `agent.injection.detected`, `agent.exfil.attempt`, `agent.cross_tenant.attempt`, `agent.escalation.blocked` |289| **Stuck tasks / Abandonment queue** | Tasks that exceeded stall thresholds; resume / abandon / kill |290| **Agent cost dashboard** | Per-tenant cost per completed task; wasted spend share; outliers |291292### Required actions293294| Action | What it does | Audit + safeguards |295|---|---|---|296| **Kill task** | Transitions one task to `KILLED`; runs reversibility compensations | Reason required; emits `agent.task.killed`; tenant notified |297| **Force-pause task** | Transitions to `AWAITING_APPROVAL` (paused-by-staff); resumes when staff or tenant resumes | Reason required; tenant notified |298| **Tenant-wide agent kill-switch** | Sets agent_enabled=false; in-flight tasks complete current step and stop | Reason; tenant notified; expires unless made permanent |299| **Feature-specific kill-switch** | Pauses one agent feature for one tenant | Reason; auto-expire default 24h |300| **Approval override** | Approve / reject an approval on behalf of a tenant admin | Co-sign required for irreversible; visible to tenant |301| **Resume abandoned task** | Restart from last persisted step | Audit |302| **Re-run with new config** | Replay task with candidate prompt/model/tools | Sandbox only — never re-execute side effects in prod |303| **Memory inspection / forget** | Surface per-tenant or per-user agent memories; force-forget on legal request | GDPR audit row written |304| **Tool registry view** | See which tools each tenant has, which versions pinned, deprecations | Read-only here; mutations via control-plane API |305306### Kill-Switch Speed Requirement307308Per-tenant agent kill-switch must flip in **< 5 seconds** wall-time from staff click to runtime enforcement. Implementation: write to a Redis key the runtime checks before each step; DB updated asynchronously for audit.309310```python311# Runtime check before each step312def check_kill_switch(tenant_id):313 if redis.get(f"agent:killed:{tenant_id}") == "1":314 return KillSwitch("tenant_kill_switch")315 return None316```317318The kill-with-rollback runbook (`ai-agent-reversibility-and-blast-radius` §4) describes the compensation cascade. The console wires the button.319320### Required co-signs321322- Tenant-wide agent kill-switch on Enterprise: 2 staff (support engineer + tenant CSM or higher).323- Approval override of irreversible: 2 staff.324- Bulk kill of > 10 tasks: 2 staff + dry-run.325- Memory force-forget: 1 staff + legal ticket reference.326327### Cross-references328329- `ai-agent-runtime-architecture` — provides the task list + state machine the console drives.330- `ai-agent-observability-and-replay` — provides the task viewer + replay link.331- `ai-agent-action-approval-and-hitl` — approval objects the staff override.332- `ai-agent-safety-and-red-team` — safety incident feed.333- `ai-agent-reversibility-and-blast-radius` — kill-with-rollback procedure.334- `ai-agent-memory` — memory inspection / forget surface.335336## §11 Read Next337338- `saas-control-plane-engineering` — the services this console drives.339- `multi-tenant-saas-architecture` — tenant data model + cross-tenant access rules.340- `subscription-billing` — billing operations this console invokes.341- `saas-tenant-data-portability-and-erasure` — GDPR workflows this console initiates.342- `dual-auth-rbac` — internal auth + MFA underpinning.343- `vibe-security-skill` — security baseline for the back-office app.344- `ai-agent-runtime-architecture` — agent task control-plane.345- `ai-agent-observability-and-replay` — task viewer + replay surfaces.346347## §12 SLA-Credit Override + Dispute-Resolution Console (Enhancement)348349For agent products, the back-office needs two purpose-built surfaces beyond the standard tenant/billing views: an **SLA-credit override** console and a **dispute-resolution** console. Both share the principle that every keystroke is dual-authed, audit-logged, and links to the verdict / evidence pack.350351### SLA-credit override console352353Reads from the credit-case ledger produced by `ai-agent-sla-credit-automation`. For each open case the operator sees:354355- The breach event (incident link or detection signal).356- The eligibility decision (auto-approved, auto-denied, needs-review).357- The proposed credit amount (formula + inputs).358- The trace bundle + verdict + evidence pack.359- The per-tenant credit cap remaining for the period.360361Actions:362363| Action | Requires | Audit fields |364|---|---|---|365| Approve as-proposed | single-auth (operator role) | `approved_by`, `approved_at` |366| Adjust amount | dual-auth (operator + manager) | `original_amount`, `adjusted_amount`, `reason_code`, `manager_id` |367| Deny | dual-auth | `reason_code` (from enumerated list: `customer_caused` / `force_majeure` / `excluded_class` / `cap_reached`), `denial_text` |368| Issue mass credit | dual-auth + IC approval link | `incident_id`, `tenant_filter_sql`, `formula_ref` |369370Every action emits to the credit-case audit log; the customer-facing dashboard reflects the outcome in < 60 seconds.371372### Dispute-resolution console373374Reads from the dispute queue produced by `ai-agent-task-success-tracking/references/dispute-resolution.md`. For each open dispute the operator sees:375376- Customer-asserted outcome (success or failure).377- System verdict (`task.success.verdict`) and evidence pack.378- The judge rationale.379- Per-tenant dispute volume in the rolling window (abuse signal).380- The proposed resolution from the rebuttal pipeline.381382Actions:383384| Action | Effect |385|---|---|386| Uphold system verdict | dispute closed; no commercial action |387| Overturn (success → failed) | refund triggered (`ai-agent-abandonment-and-refund-policy`), revenue de-recognized in current period |388| Overturn (failed → success) | rebill at resolution price; customer notified |389| Escalate | dual-auth review queue for the head of CS |390391Rate-limit safeguard: if a tenant's overturn-in-customer-favor rate in the rolling 90-day window exceeds the configured chargeback ceiling, the console refuses single-auth overturns and forces escalation.392393### Cross-links394395- `ai-agent-sla-credit-automation` — case data source.396- `ai-agent-task-success-tracking` — dispute data source.397- `ai-agent-abandonment-and-refund-policy` — refund execution invoked from overturns.398- `dual-auth-rbac` — auth model for the override actions.399400---401402## Compliance Console (Enhancement)403404The back-office adds a **Compliance Console** surface for control owners and (read-only) for auditors.405406| Screen | Purpose | Backing service |407|---|---|---|408| **Control Status** | One row per SOC 2 / ISO 27001 / HIPAA control; latest evidence pack, cadence health, owner, open exceptions | `ai-agent-soc2-controls`, `ai-agent-iso27001-controls`, `ai-agent-hipaa-security-controls` |409| **Run Evidence Collection** | Trigger an ad-hoc collector run (auditor sample request) | `ai-agent-evidence-automation` |410| **Integrity Verification** | Run a chain-witness over a custom window; show drift positions | `ai-agent-audit-log-integrity` |411| **Drill Cadence** | All drill classes with last pass date, next-due, status | `ai-agent-drill-evidence-and-cadence` |412| **Approval Completeness** | Latest PI1.1 report; open gaps | `ai-agent-approval-audit-completeness` |413| **Erasure Requests** | Open and recent erasure requests with proof-pack links | `ai-agent-memory-erasure-proof`, `saas-tenant-data-portability-and-erasure` |414| **Exception Register** | All open compliance exceptions sortable by control / severity / target-close | `ai-agent-soc2-controls` |415| **Auditor Portal** | Read-only auditor sub-surface (separate auth realm, scoped to packs) | `ai-agent-evidence-automation` |416417All Compliance Console actions are themselves logged onto the action audit log with `event_class=compliance_console`. Auditor-portal access is logged separately into `auditor_access_log` (evidence for CC6.1).418419Cross-links: `ai-agent-evidence-automation`, `ai-agent-soc2-controls`, `ai-agent-iso27001-controls`, `ai-agent-hipaa-security-controls`, `ai-agent-audit-log-integrity`, `ai-agent-drill-evidence-and-cadence`, `ai-agent-approval-audit-completeness`, `ai-agent-memory-erasure-proof`, `ai-agent-control-testing-and-attestation`.