Design Azure API Management as the runtime AI gateway for model and tool traffic, including token-per-minute controls, token limits, quotas, multi-backend load-balanced backend pools, circuit breakers, semantic caching, token metrics, managed identity, and content safety. Use when fronting model deployments, enforcing token budgets, adding semantic cache, load balancing LLM endpoints, or governing tool API calls at runtime.
Design Azure API Management as the runtime gateway between agents, applications, model deployments, and tool APIs so traffic is governed, resilient, cached, metered, and secured without changing every caller.
When to invoke
"Put Azure API Management in front of our model deployments."
"Design token limits and semantic caching for agent traffic."
"Load balance Azure OpenAI backends with circuit breakers."
"Govern tool API calls at runtime with an AI gateway."
Inputs
Use $ARGUMENTS as the runtime gateway scenario: what callers invoke, which model or tool backends exist, tenant or feature dimensions, token budget goals, cache needs, and network/security constraints. If details are missing, state assumptions before designing.
Prerequisites and context
Verify current Azure API Management policy names and behavior on Microsoft Learn before committing exact XML policy syntax.
The Developer tier of API Management cannot be created with publicNetworkAccess=Disabled; create it enabled and harden later if that tier is required.
Use the installed azure-aigateway skill for deep policy examples when available; this skill defines the primitive design shape.
Gateway capabilities
Capability
Policy area
Why it matters
Token rate limiting
token limit per key or subscription
Protect budgets and prevent runaway spend.
Token quotas
renewable token quota
Enforce per-tenant, per-user, or per-feature ceilings.
Load balancing
backend pool with weights and priority
Spread traffic across model deployments, models, and regions.
Resilience
circuit breaker on backends
Bypass throttled, unhealthy, or down deployments.
Semantic caching
semantic cache lookup and store
Reuse answers for similar prompts through a cache backend such as Azure Managed Redis.
Cost attribution
emit token metric
Track usage by model, tenant, feature, and caller in Azure Monitor.
Identity
managed identity to backend
Remove backend keys from application code and use AAD where supported.
Safety
content safety integration
Screen requests and responses at the edge.
Reference pattern
Agents / GitHub Copilot / apps
|
Azure API Management (AI gateway)
- authN (managed identity, subscription keys for callers)
- token limit + quota
- semantic cache lookup --> hit returns cached response
- load balance + circuit breaker
| |
Model deployment A Model deployment B (multi-region, multi-model)
|
emit token metric --> Azure Monitor (cost attribution)
Procedure
Define backends. Register each model deployment or tool API as a backend. Group related model deployments into backend pools with weights and priorities.
Set token controls. Apply token limit policies for rate and token quota policies for renewable ceilings. Key limits by caller, tenant, subscription, feature, or product tier.
Add resilience. Configure backend circuit breaker rules so throttled or failing deployments are skipped before callers experience repeated failures.
Add semantic cache. Use semantic cache lookup before backend routing and semantic cache store after successful responses. Tune similarity thresholds conservatively to avoid unsafe reuse.
Attribute cost. Emit token metric dimensions such as model, deployment, tenant, caller, and feature to Azure Monitor dashboards and alerts.
Secure the path. Use managed identity from API Management to the model backend where supported. Restrict callers with subscription keys, AAD, networking, and content safety based on data sensitivity.
Validate with live probes. Test cache hit/miss behavior, token budget enforcement, backend failover, and metric emission before documenting the gateway as production-ready.
Design boundaries
Boundary
Decision rule
Governance vs runtime
azure-api-center catalogs APIs, tools, and MCP servers; APIM enforces and routes calls at runtime.
Caching
Gateway semantic caching is the lowest-change option; app-level caching belongs in azure-managed-redis-cache.
Routing
Centralizing model calls behind APIM makes routing tiers from agentic-architecture-patterns enforceable in one place.
Tool calls
Treat governed tool APIs like any other backend: authenticate callers, enforce quotas, monitor errors, and document ownership.
Gotchas
Semantic cache correctness is a safety issue: overly loose similarity thresholds can return plausible but wrong answers across tenants or tasks.
Token limits need dimensions: a global limit protects the service but does not control per-tenant cost.
Circuit breakers need health evidence: failover without metrics can hide systemic backend failure.
Managed identity support varies by backend: verify the target service before promising keyless auth.
Related primitives
Name
Type
Use it when
azure-aigateway
skill
You need exact APIM AI policy syntax or current Microsoft Learn examples.
azure-api-center
skill
You need catalog, governance, discovery, or linter rules rather than runtime enforcement.
azure-managed-redis-cache
skill
You need the cache backend design behind semantic caching.
agentic-architecture-patterns
skill
You need agent routing and tool-use architecture decisions.
Open Horizons integration
Scope AI gateway work to the Agent IDP objective and current Horizon stage.
Preserve Open Horizons Azure ownership, managed-identity, network, policy, and evidence boundaries where applicable.
Route cross-domain sequencing through open-horizons-orchestration (skill).
Output template
## APIM AI gateway design
**Scenario:** <what is fronted>
**Gateway role:** runtime enforcement for model traffic | tool traffic | both
| Concern | Design decision | Policy or service area | Validation |
| --- | --- | --- | --- |
| Token budget | <limit/quota dimensions> | token limit, token quota | <test> |
| Routing | <backend pool, weights, priority> | backends | <failover test> |
| Cache | <lookup/store and threshold> | semantic cache | <hit/miss test> |
| Identity | <caller and backend auth> | managed identity, subscription keys, AAD | <auth test> |
| Cost | <dimensions> | emit token metric, Azure Monitor | <dashboard or query> |
### Risks and mitigations
- <risk>: <mitigation>
Quality gate
Current APIM AI gateway policy names and limitations were verified against Microsoft Learn.
Token limits and quotas are scoped by caller, tenant, subscription, feature, or another explicit dimension.
Backend pools, priorities, weights, and circuit breaker behavior are defined.
Semantic cache lookup and store behavior includes a conservative threshold and tenant-safe keying.
Managed identity, caller authentication, network hardening, and content safety are addressed.
Token metric emission includes dimensions for cost attribution.
The Developer tier publicNetworkAccess=Disabled limitation is considered when relevant.
1---2name: azure-apim-ai-gateway-23description: Design Azure API Management as the runtime AI gateway for model and tool traffic, including token-per-minute controls, token limits, quotas, multi-backend load-balanced backend pools, circuit breakers, semantic caching, token metrics, managed identity, and content safety. Use when fronting model deployments, enforcing token budgets, adding semantic cache, load balancing LLM endpoints, or governing tool API calls at runtime.4---56# APIM AI gateway78Design Azure API Management as the runtime gateway between agents, applications, model deployments, and tool APIs so traffic is governed, resilient, cached, metered, and secured without changing every caller.910## When to invoke1112- "Put Azure API Management in front of our model deployments."13- "Design token limits and semantic caching for agent traffic."14- "Load balance Azure OpenAI backends with circuit breakers."15- "Govern tool API calls at runtime with an AI gateway."1617## Inputs1819Use `$ARGUMENTS` as the runtime gateway scenario: what callers invoke, which model or tool backends exist, tenant or feature dimensions, token budget goals, cache needs, and network/security constraints. If details are missing, state assumptions before designing.2021## Prerequisites and context2223- Verify current Azure API Management policy names and behavior on Microsoft Learn before committing exact XML policy syntax.24- The Developer tier of API Management cannot be created with `publicNetworkAccess=Disabled`; create it enabled and harden later if that tier is required.25- Use the installed `azure-aigateway` skill for deep policy examples when available; this skill defines the primitive design shape.2627## Gateway capabilities2829| Capability | Policy area | Why it matters |30| --- | --- | --- |31| Token rate limiting | token limit per key or subscription | Protect budgets and prevent runaway spend. |32| Token quotas | renewable token quota | Enforce per-tenant, per-user, or per-feature ceilings. |33| Load balancing | backend pool with weights and priority | Spread traffic across model deployments, models, and regions. |34| Resilience | circuit breaker on backends | Bypass throttled, unhealthy, or down deployments. |35| Semantic caching | semantic cache lookup and store | Reuse answers for similar prompts through a cache backend such as Azure Managed Redis. |36| Cost attribution | emit token metric | Track usage by model, tenant, feature, and caller in Azure Monitor. |37| Identity | managed identity to backend | Remove backend keys from application code and use AAD where supported. |38| Safety | content safety integration | Screen requests and responses at the edge. |3940## Reference pattern4142```text43Agents / GitHub Copilot / apps44 |45 Azure API Management (AI gateway)46 - authN (managed identity, subscription keys for callers)47 - token limit + quota48 - semantic cache lookup --> hit returns cached response49 - load balance + circuit breaker50 | |51 Model deployment A Model deployment B (multi-region, multi-model)52 |53 emit token metric --> Azure Monitor (cost attribution)54```5556## Procedure57581. Define backends. Register each model deployment or tool API as a backend. Group related model deployments into backend pools with weights and priorities.592. Set token controls. Apply token limit policies for rate and token quota policies for renewable ceilings. Key limits by caller, tenant, subscription, feature, or product tier.603. Add resilience. Configure backend circuit breaker rules so throttled or failing deployments are skipped before callers experience repeated failures.614. Add semantic cache. Use semantic cache lookup before backend routing and semantic cache store after successful responses. Tune similarity thresholds conservatively to avoid unsafe reuse.625. Attribute cost. Emit token metric dimensions such as model, deployment, tenant, caller, and feature to Azure Monitor dashboards and alerts.636. Secure the path. Use managed identity from API Management to the model backend where supported. Restrict callers with subscription keys, AAD, networking, and content safety based on data sensitivity.647. Validate with live probes. Test cache hit/miss behavior, token budget enforcement, backend failover, and metric emission before documenting the gateway as production-ready.6566## Design boundaries6768| Boundary | Decision rule |69| --- | --- |70| Governance vs runtime | `azure-api-center` catalogs APIs, tools, and MCP servers; APIM enforces and routes calls at runtime. |71| Caching | Gateway semantic caching is the lowest-change option; app-level caching belongs in `azure-managed-redis-cache`. |72| Routing | Centralizing model calls behind APIM makes routing tiers from `agentic-architecture-patterns` enforceable in one place. |73| Tool calls | Treat governed tool APIs like any other backend: authenticate callers, enforce quotas, monitor errors, and document ownership. |7475## Gotchas7677- **Semantic cache correctness is a safety issue**: overly loose similarity thresholds can return plausible but wrong answers across tenants or tasks.78- **Token limits need dimensions**: a global limit protects the service but does not control per-tenant cost.79- **Circuit breakers need health evidence**: failover without metrics can hide systemic backend failure.80- **Managed identity support varies by backend**: verify the target service before promising keyless auth.8182## Related primitives8384| Name | Type | Use it when |85| --- | --- | --- |86| `azure-aigateway` | skill | You need exact APIM AI policy syntax or current Microsoft Learn examples. |87| `azure-api-center` | skill | You need catalog, governance, discovery, or linter rules rather than runtime enforcement. |88| `azure-managed-redis-cache` | skill | You need the cache backend design behind semantic caching. |89| `agentic-architecture-patterns` | skill | You need agent routing and tool-use architecture decisions. |9091## Open Horizons integration9293- Scope AI gateway work to the Agent IDP objective and current Horizon stage.94- Preserve Open Horizons Azure ownership, managed-identity, network, policy, and evidence boundaries where applicable.95- Route cross-domain sequencing through `open-horizons-orchestration` (`skill`).9697## Output template9899```markdown100## APIM AI gateway design101102**Scenario:** <what is fronted>103**Gateway role:** runtime enforcement for model traffic | tool traffic | both104105| Concern | Design decision | Policy or service area | Validation |106| --- | --- | --- | --- |107| Token budget | <limit/quota dimensions> | token limit, token quota | <test> |108| Routing | <backend pool, weights, priority> | backends | <failover test> |109| Cache | <lookup/store and threshold> | semantic cache | <hit/miss test> |110| Identity | <caller and backend auth> | managed identity, subscription keys, AAD | <auth test> |111| Cost | <dimensions> | emit token metric, Azure Monitor | <dashboard or query> |112113### Risks and mitigations114- <risk>: <mitigation>115```116117## Quality gate118119- [ ] Current APIM AI gateway policy names and limitations were verified against Microsoft Learn.120- [ ] Token limits and quotas are scoped by caller, tenant, subscription, feature, or another explicit dimension.121- [ ] Backend pools, priorities, weights, and circuit breaker behavior are defined.122- [ ] Semantic cache lookup and store behavior includes a conservative threshold and tenant-safe keying.123- [ ] Managed identity, caller authentication, network hardening, and content safety are addressed.124- [ ] Token metric emission includes dimensions for cost attribution.125- [ ] The Developer tier `publicNetworkAccess=Disabled` limitation is considered when relevant.126127## References128129- [Azure API Management AI gateway capabilities](https://learn.microsoft.com/azure/api-management/genai-gateway-capabilities)130- [Token limit policy](https://learn.microsoft.com/azure/api-management/llm-token-limit-policy)131- [Semantic caching policy](https://learn.microsoft.com/azure/api-management/azure-openai-semantic-cache-lookup-policy)132- [Load balancing and circuit breaker for backends](https://learn.microsoft.com/azure/api-management/backends)
Run npx skillmds@latest add paulasilvatech/azure-apim-ai-gateway-2 in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.
Design Azure API Management as the runtime AI gateway for model and tool traffic, including token-per-minute controls, token limits, quotas, multi-backend load-balanced backend pools, circuit breakers, semantic caching, token metrics, managed identity, and content safety. Use when fronting model deployments, enforcing token budgets, adding semantic cache, load balancing LLM endpoints, or governing tool API calls at runtime. It is listed under DevOps & Infra on SkillMD.
This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.
This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.
Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.
paulasilvatech (@paulasilvatech) published this skill. Their other Agent Skills are listed on their SkillMD profile.
paulasilvatech