Temporal Durable Execution & Workflow Orchestration AI Skill Guide
Overview & Engine Architecture
Temporal provides durable execution: workflow state is event-sourced; workers replay history to recover. Workflow code must be deterministic; Activities hold all I/O (HTTP, DB, messaging). The server (Frontend/History/Matching) persists to a primary database; Visibility is a separate search index (eventually consistent).
Client -> Temporal Server (gRPC, auth/mTLS)
-> History (event log) -> Persistence (Postgres/MySQL/Cassandra)
-> Matching (task queues)
Workers (poll task queues)
-> Workflow sandbox (replay)
-> Activities (retries, timeouts, heartbeats)
When to use / when not to
Use when
- Long-running processes (hours/days), sagas, human-in-the-loop, cron with strong reliability
- Need automatic retries, timers, signals, and audit-grade history
Do not use when
- Simple fire-and-forget cron without state (
@airflow task)
- All logic fits a single DB transaction with no external calls
- Team refuses determinism constraints—workflows will break on deploy
Operational Capabilities & Agent Directives
- Determinism: No
Date.now(), Math.random(), or undisciplined UUIDs in workflow code—use SDK workflow.now(), workflow.uuid4(), etc. All I/O in activities.
- Activity contracts: Always set
start_to_close_timeout; retries with backoff; heartbeats for long activities; activities must be idempotent (retries are normal).
- Versioning: Use
workflow.patched() or Worker Versioning before changing command order; run replay tests on representative histories before prod deploy (safe deployments).
- Backpressure: Scale workers per task queue; split activity vs workflow workers; monitor schedule-to-start latency; avoid unbounded local activity parallelism.
- Auth: mTLS + namespace for Temporal Cloud; rotate client certs; never commit keys.
- Backup/restore: Backup persistence DB for execution truth; Visibility (Elasticsearch/OpenSearch) does not rebuild from primary—backup visibility too if search history matters; closed workflows may stay stale after partial restore (community thread).
- Exactly-once myth: Activities run at-least-once; workflows replay; side effects require idempotency keys or compensations (sagas).
Production examples
Saga with compensation stack (TypeScript)
import { proxyActivities, defineQuery, setHandler } from "@temporalio/workflow";
import type * as activities from "./activities";
const { chargePayment, reserveInventory, cancelPayment, releaseInventory } =
proxyActivities<typeof activities>({
startToCloseTimeout: "30 seconds",
retry: { initialInterval: "1s", backoffCoefficient: 2, maximumAttempts: 5 },
});
export const orderStatusQuery = defineQuery<string>("orderStatus");
export async function processOrderWorkflow(input: OrderInput): Promise<string> {
const compensations: Array<() => Promise<void>> = [];
let status = "INITIALIZING";
setHandler(orderStatusQuery, () => status);
try {
status = "CHARGING";
await chargePayment(input.customerId, input.amount);
compensations.push(() => cancelPayment(input.customerId, input.amount));
status = "RESERVING";
await reserveInventory(input.sku, 1);
compensations.push(() => releaseInventory(input.sku, 1));
status = "COMPLETED";
return `Order ${input.orderId} processed`;
} catch (err) {
status = "ROLLING_BACK";
for (const compensate of compensations.reverse()) {
try { await compensate(); } catch { /* alert + manual runbook */ }
}
status = "FAILED";
throw err;
}
}
Activity idempotency (sketch)
export async function chargePayment(customerId: string, amount: number) {
await payments.charge({ idempotencyKey: workflowInfo().workflowId, customerId, amount });
}
CLI operations
temporal workflow start --task-queue order-processing-queue \
--type processOrderWorkflow --workflow-id order-ord-9901 \
--input '{"orderId":"ord-9901","customerId":"cust-42","amount":1500,"sku":"WIDGET-01"}'
temporal workflow query --workflow-id order-ord-9901 --name orderStatus
Technical troubleshooting matrix
| Failure signature |
Root cause |
Diagnostic & fix |
NonDeterminismError on deploy |
Workflow code changed without patch |
Roll back worker; add patched(); replay test |
| Activity schedule-to-start timeout |
No workers / wrong queue |
Verify task queue; scale workers |
| History size limit |
Unbounded loop/events |
continueAsNew(); batch signals |
| Duplicate charges |
Non-idempotent activity |
Idempotency keys; dedupe table |
| Visibility stale vs history |
Async index |
Use DescribeWorkflowExecution for truth |
| Restore search broken |
ES not restored |
Backup visibility store; accept stale closed WF |
Best practices
- One workflow type per business process; keep workflows thin orchestrators.
- Use signals for human approval; queries for read-only status—not polling visibility from workflow code (visibility guidance).
- Encrypt payloads at rest when PII present; redact in UI exports.
- Monitor
temporal_workflow_task_execution_failed, task lag, and worker restarts.
Limitations
- Workflow code changes are deployment events—plan versioning like schema migrations.
- High-cardinality child workflows can stress history service.
- This skill does not replace Temporal Cloud SLA/support contracts.
Related skills
@postgresql — common persistence backend
@kafka — event ingestion triggering workflows
@ai-human-handoff-contract — human steps via signals
Agent Operational Directive
MANDATORY: No non-deterministic APIs in workflow functions. Every activity has timeouts and idempotent side effects. Run replay validation before shipping workflow changes. Do not use Visibility queries as strongly consistent workflow state. Backup plan must include visibility if search-dependent ops exist.
Source anchors (research)
1---2name: temporal3description: "Author deterministic Temporal workflows and idempotent activities; deploy with versioning/replay tests; configure mTLS auth; understand visibility backup limits and saga compensation—not in-memory orchestration."4category: development5risk: safe6source: self7source_type: self8date_added: "2026-09-13"9tags: ["temporal", "workflow", "activities", "determinism", "sagas", "worker-versioning", "replay", "visibility", "backpressure"]10tools: ["claude", "cursor", "gemini", "codex"]11metadata:12 source: https://skillmd.com/skills/alivirgo/temporal13---1415# Temporal Durable Execution & Workflow Orchestration AI Skill Guide1617## Overview & Engine Architecture1819Temporal provides **durable execution**: workflow state is event-sourced; workers replay history to recover. **Workflow code must be deterministic**; **Activities** hold all I/O (HTTP, DB, messaging). The server (Frontend/History/Matching) persists to a primary database; **Visibility** is a separate search index (eventually consistent).2021```22Client -> Temporal Server (gRPC, auth/mTLS)23 -> History (event log) -> Persistence (Postgres/MySQL/Cassandra)24 -> Matching (task queues)25Workers (poll task queues)26 -> Workflow sandbox (replay)27 -> Activities (retries, timeouts, heartbeats)28```2930---3132## When to use / when not to3334**Use when**3536- Long-running processes (hours/days), sagas, human-in-the-loop, cron with strong reliability37- Need automatic retries, timers, signals, and audit-grade history3839**Do not use when**4041- Simple fire-and-forget cron without state (`@airflow` task)42- All logic fits a single DB transaction with no external calls43- Team refuses determinism constraints—workflows will break on deploy4445---4647## Operational Capabilities & Agent Directives48491. **Determinism**: No `Date.now()`, `Math.random()`, or undisciplined UUIDs in workflow code—use SDK `workflow.now()`, `workflow.uuid4()`, etc. All I/O in activities.502. **Activity contracts**: Always set `start_to_close_timeout`; retries with backoff; **heartbeats** for long activities; activities must be **idempotent** (retries are normal).513. **Versioning**: Use `workflow.patched()` or **Worker Versioning** before changing command order; run **replay tests** on representative histories before prod deploy ([safe deployments](https://docs.temporal.io/develop/safe-deployments)).524. **Backpressure**: Scale workers per **task queue**; split activity vs workflow workers; monitor schedule-to-start latency; avoid unbounded local activity parallelism.535. **Auth**: mTLS + namespace for Temporal Cloud; rotate client certs; never commit keys.546. **Backup/restore**: Backup **persistence DB** for execution truth; **Visibility** (Elasticsearch/OpenSearch) does **not** rebuild from primary—backup visibility too if search history matters; closed workflows may stay stale after partial restore ([community thread](https://community.temporal.io/t/backup-restore-with-advanced-visibility/7423)).557. **Exactly-once myth**: Activities run **at-least-once**; workflows replay; side effects require idempotency keys or compensations (sagas).5657---5859## Production examples6061### Saga with compensation stack (TypeScript)6263```typescript64import { proxyActivities, defineQuery, setHandler } from "@temporalio/workflow";65import type * as activities from "./activities";6667const { chargePayment, reserveInventory, cancelPayment, releaseInventory } =68 proxyActivities<typeof activities>({69 startToCloseTimeout: "30 seconds",70 retry: { initialInterval: "1s", backoffCoefficient: 2, maximumAttempts: 5 },71 });7273export const orderStatusQuery = defineQuery<string>("orderStatus");7475export async function processOrderWorkflow(input: OrderInput): Promise<string> {76 const compensations: Array<() => Promise<void>> = [];77 let status = "INITIALIZING";78 setHandler(orderStatusQuery, () => status);7980 try {81 status = "CHARGING";82 await chargePayment(input.customerId, input.amount);83 compensations.push(() => cancelPayment(input.customerId, input.amount));8485 status = "RESERVING";86 await reserveInventory(input.sku, 1);87 compensations.push(() => releaseInventory(input.sku, 1));8889 status = "COMPLETED";90 return `Order ${input.orderId} processed`;91 } catch (err) {92 status = "ROLLING_BACK";93 for (const compensate of compensations.reverse()) {94 try { await compensate(); } catch { /* alert + manual runbook */ }95 }96 status = "FAILED";97 throw err;98 }99}100```101102### Activity idempotency (sketch)103104```typescript105export async function chargePayment(customerId: string, amount: number) {106 await payments.charge({ idempotencyKey: workflowInfo().workflowId, customerId, amount });107}108```109110### CLI operations111112```bash113temporal workflow start --task-queue order-processing-queue \114 --type processOrderWorkflow --workflow-id order-ord-9901 \115 --input '{"orderId":"ord-9901","customerId":"cust-42","amount":1500,"sku":"WIDGET-01"}'116117temporal workflow query --workflow-id order-ord-9901 --name orderStatus118```119120---121122## Technical troubleshooting matrix123124| Failure signature | Root cause | Diagnostic & fix |125| :--- | :--- | :--- |126| `NonDeterminismError` on deploy | Workflow code changed without patch | Roll back worker; add `patched()`; replay test |127| Activity schedule-to-start timeout | No workers / wrong queue | Verify task queue; scale workers |128| History size limit | Unbounded loop/events | `continueAsNew()`; batch signals |129| Duplicate charges | Non-idempotent activity | Idempotency keys; dedupe table |130| Visibility stale vs history | Async index | Use `DescribeWorkflowExecution` for truth |131| Restore search broken | ES not restored | Backup visibility store; accept stale closed WF |132133---134135## Best practices136137- One workflow type per business process; keep workflows thin orchestrators.138- Use signals for human approval; queries for read-only status—**not** polling visibility from workflow code ([visibility guidance](https://docs.temporal.io/visibility)).139- Encrypt payloads at rest when PII present; redact in UI exports.140- Monitor `temporal_workflow_task_execution_failed`, task lag, and worker restarts.141142---143144## Limitations145146- Workflow code changes are deployment events—plan versioning like schema migrations.147- High-cardinality child workflows can stress history service.148- This skill does not replace Temporal Cloud SLA/support contracts.149150---151152## Related skills153154- `@postgresql` — common persistence backend155- `@kafka` — event ingestion triggering workflows156- `@ai-human-handoff-contract` — human steps via signals157158---159160## Agent Operational Directive161162> **MANDATORY**: No non-deterministic APIs in workflow functions. Every activity has timeouts and idempotent side effects. Run replay validation before shipping workflow changes. Do not use Visibility queries as strongly consistent workflow state. Backup plan must include visibility if search-dependent ops exist.163164---165166## Source anchors (research)167168- [Temporal — Workflow determinism](https://docs.temporal.io/workflow-definition)169- [Safe deployments & replay testing](https://docs.temporal.io/develop/safe-deployments)170- [Troubleshooting execution failures](https://docs.temporal.io/troubleshooting/execution-failures)171- [Visibility (eventually consistent)](https://docs.temporal.io/visibility)172- [Backup/restore with advanced visibility](https://community.temporal.io/t/backup-restore-with-advanced-visibility/7423)