Azure OpenAI

Azure OpenAI Service

Shared Workflow Routing

• Use the shared workflow spec for deterministic multi-plugin routing: workflows/multi-plugin-workflows.md.
• Apply the trigger phrases, handoff contracts, auth prerequisites, validation checkpoints, and stop conditions before escalating to the next plugin.

1. Azure OpenAI Overview

Azure OpenAI Service provides REST API access to OpenAI's language models including GPT-4o, GPT-4, GPT-3.5-Turbo, Embeddings (text-embedding-ada-002, text-embedding-3-small, text-embedding-3-large), DALL-E 3, Whisper, and TTS. Models run on Azure infrastructure with enterprise security, compliance, and regional data residency.

Azure OpenAI vs OpenAI API:

Pricing models:

Supported models (as of 2026):

Regional availability: Not all models are available in all regions. Check the Azure OpenAI model availability matrix before selecting a deployment region. Key regions with broad model support: East US, East US 2, West US, West US 3, Sweden Central, UK South, France Central, Japan East, Australia East.

2. Resource & Deployment Management

Create an Azure OpenAI Resource

# Create resource group
az group create --name rg-openai --location eastus

# Create Azure OpenAI cognitive account
az cognitiveservices account create \
  --name my-openai-resource \
  --resource-group rg-openai \
  --kind OpenAI \
  --sku S0 \
  --location eastus \
  --custom-domain my-openai-resource

# Verify the resource
az cognitiveservices account show \
  --name my-openai-resource \
  --resource-group rg-openai \
  --query "{Name:name, Endpoint:properties.endpoint, ProvisioningState:properties.provisioningState}"

# List all OpenAI resources in subscription
az cognitiveservices account list \
  --query "[?kind=='OpenAI'].{Name:name, RG:resourceGroup, Location:location, State:properties.provisioningState}" \
  --output table

# Get keys
az cognitiveservices account keys list \
  --name my-openai-resource \
  --resource-group rg-openai

# Regenerate key
az cognitiveservices account keys regenerate \
  --name my-openai-resource \
  --resource-group rg-openai \
  --key-name key1

Custom domain is required for Azure AD authentication. Set --custom-domain at creation time (cannot be changed later).

Deploy Models

Deployment types comparison:

# Deploy a GPT-4o model (Standard)
az cognitiveservices account deployment create \
  --name my-openai-resource \
  --resource-group rg-openai \
  --deployment-name gpt4o-deployment \
  --model-name gpt-4o \
  --model-version "2024-08-06" \
  --model-format OpenAI \
  --sku-name Standard \
  --sku-capacity 30

# Deploy GPT-4o-mini (Standard)
az cognitiveservices account deployment create \
  --name my-openai-resource \
  --resource-group rg-openai \
  --deployment-name gpt4o-mini-deploy \
  --model-name gpt-4o-mini \
  --model-version "2024-07-18" \
  --model-format OpenAI \
  --sku-name Standard \
  --sku-capacity 50

# Deploy with Provisioned Throughput (PTU)
az cognitiveservices account deployment create \
  --name my-openai-resource \
  --resource-group rg-openai \
  --deployment-name gpt4o-ptu \
  --model-name gpt-4o \
  --model-version "2024-08-06" \
  --model-format OpenAI \
  --sku-name ProvisionedManaged \
  --sku-capacity 50

# Deploy Global Standard
az cognitiveservices account deployment create \
  --name my-openai-resource \
  --resource-group rg-openai \
  --deployment-name gpt4o-global \
  --model-name gpt-4o \
  --model-version "2024-08-06" \
  --model-format OpenAI \
  --sku-name GlobalStandard \
  --sku-capacity 80

# Deploy an embeddings model
az cognitiveservices account deployment create \
  --name my-openai-resource \
  --resource-group rg-openai \
  --deployment-name embedding-deploy \
  --model-name text-embedding-3-small \
  --model-version "1" \
  --model-format OpenAI \
  --sku-name Standard \
  --sku-capacity 120

# Deploy DALL-E 3
az cognitiveservices account deployment create \
  --name my-openai-resource \
  --resource-group rg-openai \
  --deployment-name dalle3-deploy \
  --model-name dall-e-3 \
  --model-version "3.0" \
  --model-format OpenAI \
  --sku-name Standard \
  --sku-capacity 1

# Deploy Whisper
az cognitiveservices account deployment create \
  --name my-openai-resource \
  --resource-group rg-openai \
  --deployment-name whisper-deploy \
  --model-name whisper \
  --model-version "001" \
  --model-format OpenAI \
  --sku-name Standard \
  --sku-capacity 1

# List deployments
az cognitiveservices account deployment list \
  --name my-openai-resource \
  --resource-group rg-openai \
  --query "[].{Name:name, Model:properties.model.name, Version:properties.model.version, SKU:sku.name, Capacity:sku.capacity, State:properties.provisioningState}" \
  --output table

# Update deployment capacity (scale up/down)
az cognitiveservices account deployment create \
  --name my-openai-resource \
  --resource-group rg-openai \
  --deployment-name gpt4o-deployment \
  --model-name gpt-4o \
  --model-version "2024-08-06" \
  --model-format OpenAI \
  --sku-name Standard \
  --sku-capacity 60

# Delete a deployment
az cognitiveservices account deployment delete \
  --name my-openai-resource \
  --resource-group rg-openai \
  --deployment-name gpt4o-deployment

Model Version Upgrades

Azure OpenAI models have version deprecation timelines. When a model version is deprecated, deployments using that version must be upgraded.

# List available model versions
az cognitiveservices account list-models \
  --name my-openai-resource \
  --resource-group rg-openai \
  --query "[?name=='gpt-4o'].{Name:name, Version:version, LifecycleStatus:lifecycleStatus}" \
  --output table

# Update deployment to a new model version
az cognitiveservices account deployment create \
  --name my-openai-resource \
  --resource-group rg-openai \
  --deployment-name gpt4o-deployment \
  --model-name gpt-4o \
  --model-version "2024-11-20" \
  --model-format OpenAI \
  --sku-name Standard \
  --sku-capacity 30

Auto-update policy: Deployments can be set to auto-update to the latest GA version. For production, pin to a specific version and test before upgrading.

3. Model Catalog & Selection

Model Selection Guide

Deprecation Awareness

Models follow a lifecycle: Preview -> GA -> Deprecated -> Retired. Monitor deprecation timelines:

# Check model lifecycle status
az cognitiveservices account list-models \
  --name my-openai-resource \
  --resource-group rg-openai \
  --query "[].{Name:name, Version:version, Status:lifecycleStatus, Deprecation:deprecation.inference}" \
  --output table

Always plan model version migrations 30+ days before retirement dates. Test new versions in a staging deployment before updating production.

4. Fine-Tuning

Fine-tuning customizes a base model with your own training data to improve performance on specific tasks, reduce prompt length, and lower inference costs.

Supported models for fine-tuning: gpt-4o-mini, gpt-35-turbo-0613, gpt-35-turbo-1106, gpt-35-turbo-0125.

Training Data Format

Training data must be in JSONL format with chat completion messages:

{"messages": [{"role": "system", "content": "You are a customer support agent for Contoso."}, {"role": "user", "content": "How do I reset my password?"}, {"role": "assistant", "content": "To reset your password, go to Settings > Security > Change Password. Enter your current password and your new password twice. Click Save."}]}
{"messages": [{"role": "system", "content": "You are a customer support agent for Contoso."}, {"role": "user", "content": "What are your business hours?"}, {"role": "assistant", "content": "Our business hours are Monday through Friday, 9 AM to 5 PM Eastern Time. We are closed on weekends and federal holidays."}]}

Data requirements:

• Minimum 10 examples (50-100+ recommended for good results)
• Maximum file size: 512 MB per file
• Each example must include at least one user message and one assistant message
• System messages should be consistent across examples
• Validation split is automatic (or provide a separate validation file)

Fine-Tuning Workflow via REST API

RESOURCE_NAME="my-openai-resource"
API_VERSION="2024-08-01-preview"
BASE_URL="https://${RESOURCE_NAME}.openai.azure.com/openai"

# Step 1: Upload training file
curl -X POST "${BASE_URL}/files?api-version=${API_VERSION}" \
  -H "api-key: ${AZURE_OPENAI_KEY}" \
  -F "purpose=fine-tune" \
  -F "file=@training_data.jsonl"
# Response includes file ID: "file-abc123"

# Step 2: Upload validation file (optional)
curl -X POST "${BASE_URL}/files?api-version=${API_VERSION}" \
  -H "api-key: ${AZURE_OPENAI_KEY}" \
  -F "purpose=fine-tune" \
  -F "file=@validation_data.jsonl"

# Step 3: Check file processing status
curl -X GET "${BASE_URL}/files/file-abc123?api-version=${API_VERSION}" \
  -H "api-key: ${AZURE_OPENAI_KEY}"
# Wait until status is "processed"

# Step 4: Create fine-tuning job
curl -X POST "${BASE_URL}/fine_tuning/jobs?api-version=${API_VERSION}" \
  -H "api-key: ${AZURE_OPENAI_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "training_file": "file-abc123",
    "validation_file": "file-def456",
    "hyperparameters": {
      "n_epochs": 3,
      "batch_size": 1,
      "learning_rate_multiplier": 1.0
    },
    "suffix": "contoso-support"
  }'
# Response includes job ID: "ftjob-xyz789"

# Step 5: Monitor training progress
curl -X GET "${BASE_URL}/fine_tuning/jobs/ftjob-xyz789?api-version=${API_VERSION}" \
  -H "api-key: ${AZURE_OPENAI_KEY}"

# Step 6: List fine-tuning events (training logs)
curl -X GET "${BASE_URL}/fine_tuning/jobs/ftjob-xyz789/events?api-version=${API_VERSION}" \
  -H "api-key: ${AZURE_OPENAI_KEY}"

# Step 7: Deploy the fine-tuned model
az cognitiveservices account deployment create \
  --name my-openai-resource \
  --resource-group rg-openai \
  --deployment-name contoso-support-ft \
  --model-name "gpt-4o-mini.ft-contoso-support" \
  --model-version "1" \
  --model-format OpenAI \
  --sku-name Standard \
  --sku-capacity 20

# List fine-tuning jobs
curl -X GET "${BASE_URL}/fine_tuning/jobs?api-version=${API_VERSION}" \
  -H "api-key: ${AZURE_OPENAI_KEY}"

# Cancel a running job
curl -X POST "${BASE_URL}/fine_tuning/jobs/ftjob-xyz789/cancel?api-version=${API_VERSION}" \
  -H "api-key: ${AZURE_OPENAI_KEY}"

# Delete a fine-tuned model
curl -X DELETE "${BASE_URL}/fine_tuning/jobs/ftjob-xyz789?api-version=${API_VERSION}" \
  -H "api-key: ${AZURE_OPENAI_KEY}"

Hyperparameter Tuning

Evaluation

After fine-tuning, evaluate the model:

• Compare fine-tuned model responses against base model on a held-out test set
• Check training loss and validation loss curves for convergence
• Monitor for overfitting: validation loss increasing while training loss decreases
• Test edge cases and adversarial inputs specific to your domain

5. Content Filtering

Azure OpenAI includes built-in content filtering that evaluates both prompts and completions. Filters run automatically and can be customized per deployment.

Default filter categories:

Severity levels (0-7 scale, grouped):

Custom Content Filter Policies

Content filter policies are managed via the REST API. Policies can be assigned to individual deployments.

RESOURCE_NAME="my-openai-resource"
RG="rg-openai"
SUB_ID="$(az account show --query id -o tsv)"
API_VERSION="2024-06-01-preview"
BASE_ARM="https://management.azure.com/subscriptions/${SUB_ID}/resourceGroups/${RG}/providers/Microsoft.CognitiveServices/accounts/${RESOURCE_NAME}"

# Create a custom content filter policy
curl -X PUT "${BASE_ARM}/raiPolicies/strict-policy?api-version=${API_VERSION}" \
  -H "Authorization: Bearer $(az account get-access-token --query accessToken -o tsv)" \
  -H "Content-Type: application/json" \
  -d '{
    "properties": {
      "basePolicyName": "Microsoft.DefaultV2",
      "mode": "Blocking",
      "contentFilters": [
        {"name": "hate", "blocking": true, "enabled": true, "allowedContentLevel": "Low", "source": "Prompt"},
        {"name": "hate", "blocking": true, "enabled": true, "allowedContentLevel": "Low", "source": "Completion"},
        {"name": "sexual", "blocking": true, "enabled": true, "allowedContentLevel": "Low", "source": "Prompt"},
        {"name": "sexual", "blocking": true, "enabled": true, "allowedContentLevel": "Low", "source": "Completion"},
        {"name": "violence", "blocking": true, "enabled": true, "allowedContentLevel": "Low", "source": "Prompt"},
        {"name": "violence", "blocking": true, "enabled": true, "allowedContentLevel": "Low", "source": "Completion"},
        {"name": "selfharm", "blocking": true, "enabled": true, "allowedContentLevel": "Low", "source": "Prompt"},
        {"name": "selfharm", "blocking": true, "enabled": true, "allowedContentLevel": "Low", "source": "Completion"},
        {"name": "jailbreak", "blocking": true, "enabled": true, "source": "Prompt"},
        {"name": "indirect_attack", "blocking": true, "enabled": true, "source": "Prompt"}
      ]
    }
  }'

# List content filter policies
curl -X GET "${BASE_ARM}/raiPolicies?api-version=${API_VERSION}" \
  -H "Authorization: Bearer $(az account get-access-token --query accessToken -o tsv)"

# Get a specific policy
curl -X GET "${BASE_ARM}/raiPolicies/strict-policy?api-version=${API_VERSION}" \
  -H "Authorization: Bearer $(az account get-access-token --query accessToken -o tsv)"

# Delete a policy
curl -X DELETE "${BASE_ARM}/raiPolicies/strict-policy?api-version=${API_VERSION}" \
  -H "Authorization: Bearer $(az account get-access-token --query accessToken -o tsv)"

Custom Blocklists

Blocklists allow blocking specific terms or patterns that the default filters may not catch.

# Create a blocklist
curl -X PUT "${BASE_ARM}/raiBlocklists/profanity-list?api-version=${API_VERSION}" \
  -H "Authorization: Bearer $(az account get-access-token --query accessToken -o tsv)" \
  -H "Content-Type: application/json" \
  -d '{
    "properties": {
      "description": "Custom profanity and competitor brand terms"
    }
  }'

# Add items to blocklist
curl -X PUT "${BASE_ARM}/raiBlocklists/profanity-list/raiBlocklistItems/item1?api-version=${API_VERSION}" \
  -H "Authorization: Bearer $(az account get-access-token --query accessToken -o tsv)" \
  -H "Content-Type: application/json" \
  -d '{
    "properties": {
      "pattern": "competitor-brand-name",
      "isRegex": false
    }
  }'

# Add regex-based blocklist item
curl -X PUT "${BASE_ARM}/raiBlocklists/profanity-list/raiBlocklistItems/item2?api-version=${API_VERSION}" \
  -H "Authorization: Bearer $(az account get-access-token --query accessToken -o tsv)" \
  -H "Content-Type: application/json" \
  -d '{
    "properties": {
      "pattern": "\\b(badword1|badword2|badword3)\\b",
      "isRegex": true
    }
  }'

# List blocklist items
curl -X GET "${BASE_ARM}/raiBlocklists/profanity-list/raiBlocklistItems?api-version=${API_VERSION}" \
  -H "Authorization: Bearer $(az account get-access-token --query accessToken -o tsv)"

Content Filter Annotations

When content filtering triggers, the API response includes annotations:

{
  "choices": [{
    "content_filter_results": {
      "hate": {"filtered": false, "severity": "safe"},
      "sexual": {"filtered": false, "severity": "safe"},
      "violence": {"filtered": false, "severity": "safe"},
      "self_harm": {"filtered": false, "severity": "safe"}
    }
  }],
  "prompt_filter_results": [{
    "content_filter_results": {
      "hate": {"filtered": false, "severity": "safe"},
      "sexual": {"filtered": false, "severity": "safe"},
      "violence": {"filtered": false, "severity": "safe"},
      "self_harm": {"filtered": false, "severity": "safe"},
      "jailbreak": {"filtered": false, "detected": false},
      "indirect_attack": {"filtered": false, "detected": false}
    }
  }]
}

When a filter triggers with "filtered": true, the API returns a 400 error with content_filter as the finish reason. Handle this in your application:

try:
    response = client.chat.completions.create(...)
except openai.BadRequestError as e:
    if "content_filter" in str(e):
        # Log the filtered request, return safe fallback
        logger.warning(f"Content filter triggered: {e}")
        return "I cannot assist with that request."

6. Prompt Engineering

System Message Patterns

System messages set the persona, constraints, and behavior of the model. They are the most important tool for controlling output quality.

Basic persona pattern:

You are a helpful assistant for Contoso customer support. You answer questions about
Contoso products and services. You are polite, concise, and accurate.

Rules:
- Only answer questions related to Contoso products.
- If you don't know the answer, say "I don't have that information. Please contact support@contoso.com."
- Never make up product features or pricing.
- Always include a link to relevant documentation when available.

Grounding with context pattern (for RAG):

You are a knowledgeable assistant that answers questions based on the provided context.
Use ONLY the information from the context below to answer. If the answer is not in
the context, say "I don't have enough information to answer that question."

## Context
{retrieved_documents}

## Rules
- Cite the source document for each claim using [Source: document_name].
- Do not infer or extrapolate beyond what the context states.
- If the context contains conflicting information, present both perspectives.

JSON output pattern:

You are a data extraction assistant. Extract the requested information from the input
text and return it as a JSON object matching the specified schema.

Always respond with valid JSON. Do not include markdown formatting or explanation text.
If a field cannot be determined from the input, set its value to null.

Few-Shot Examples

Few-shot examples teach the model your expected output format and reasoning:

{
  "messages": [
    {"role": "system", "content": "You classify support tickets into categories: billing, technical, account, other."},
    {"role": "user", "content": "I was charged twice for my subscription this month."},
    {"role": "assistant", "content": "{\"category\": \"billing\", \"confidence\": 0.95, \"reasoning\": \"Double charge indicates a billing issue\"}"},
    {"role": "user", "content": "I can't log into my account after changing my email."},
    {"role": "assistant", "content": "{\"category\": \"account\", \"confidence\": 0.90, \"reasoning\": \"Login failure after email change is an account management issue\"}"},
    {"role": "user", "content": "The API returns a 500 error when I call the /users endpoint."},
    {"role": "assistant", "content": "{\"category\": \"technical\", \"confidence\": 0.98, \"reasoning\": \"API error is a technical support issue\"}"},
    {"role": "user", "content": "<actual user input here>"}
  ]
}

Function Calling

Function calling lets the model generate structured arguments for functions you define:

{
  "messages": [
    {"role": "system", "content": "You help users manage their calendar."},
    {"role": "user", "content": "Schedule a meeting with John tomorrow at 2 PM for 30 minutes about project review."}
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "schedule_meeting",
        "description": "Schedule a calendar meeting",
        "parameters": {
          "type": "object",
          "properties": {
            "title": {"type": "string", "description": "Meeting title"},
            "attendees": {"type": "array", "items": {"type": "string"}, "description": "List of attendee names or emails"},
            "start_time": {"type": "string", "description": "Meeting start time in ISO 8601 format"},
            "duration_minutes": {"type": "integer", "description": "Meeting duration in minutes"},
            "description": {"type": "string", "description": "Meeting description or agenda"}
          },
          "required": ["title", "attendees", "start_time", "duration_minutes"]
        }
      }
    }
  ],
  "tool_choice": "auto"
}

Structured Output (JSON Mode)

Force the model to output valid JSON:

curl -X POST "https://${RESOURCE_NAME}.openai.azure.com/openai/deployments/${DEPLOYMENT}/chat/completions?api-version=2024-08-01-preview" \
  -H "api-key: ${AZURE_OPENAI_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "system", "content": "Extract entities from the text. Return a JSON object with arrays for people, organizations, and locations."},
      {"role": "user", "content": "Microsoft CEO Satya Nadella announced new AI features at the Seattle conference."}
    ],
    "response_format": {"type": "json_object"},
    "temperature": 0,
    "max_tokens": 500
  }'

JSON Schema mode (for strict schema validation):

{
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "entity_extraction",
      "strict": true,
      "schema": {
        "type": "object",
        "properties": {
          "people": {"type": "array", "items": {"type": "string"}},
          "organizations": {"type": "array", "items": {"type": "string"}},
          "locations": {"type": "array", "items": {"type": "string"}}
        },
        "required": ["people", "organizations", "locations"],
        "additionalProperties": false
      }
    }
  }
}

Token Optimization

Reduce token usage and cost:

• Be concise in system messages: Remove unnecessary words; use bullet points over paragraphs.
• Use few-shot wisely: 2-3 examples are usually sufficient; more adds tokens without proportional quality gain.
• Set max_tokens: Always set a reasonable max_tokens limit to prevent runaway generation.
• Use gpt-4o-mini for simple tasks: Classification, extraction, and reformatting tasks often work well with the smaller model.
• Cache common prompts: Use the same system message across conversations to benefit from prompt caching (automatic for Azure OpenAI).
• Structured output reduces retries: JSON mode avoids parsing failures that require re-prompting.

7. Batch API

The Batch API processes large volumes of requests asynchronously at a 50% discount compared to standard pricing. Ideal for bulk processing, data labeling, evaluation, and non-time-sensitive workloads.

Batch Input File Format

The input file is JSONL where each line is a request:

{"custom_id": "request-1", "method": "POST", "url": "/chat/completions", "body": {"model": "gpt-4o-mini", "messages": [{"role": "system", "content": "Classify the sentiment."}, {"role": "user", "content": "I love this product!"}], "max_tokens": 50}}
{"custom_id": "request-2", "method": "POST", "url": "/chat/completions", "body": {"model": "gpt-4o-mini", "messages": [{"role": "system", "content": "Classify the sentiment."}, {"role": "user", "content": "This is terrible and broken."}], "max_tokens": 50}}
{"custom_id": "request-3", "method": "POST", "url": "/chat/completions", "body": {"model": "gpt-4o-mini", "messages": [{"role": "system", "content": "Classify the sentiment."}, {"role": "user", "content": "It works fine, nothing special."}], "max_tokens": 50}}

Requirements:

• Each line must have custom_id, method, url, and body
• custom_id must be unique within the file
• Maximum 100,000 requests per batch
• Maximum input file size: 200 MB
• model field in body must match the deployment name

Batch Workflow

RESOURCE_NAME="my-openai-resource"
API_VERSION="2024-08-01-preview"
BASE_URL="https://${RESOURCE_NAME}.openai.azure.com/openai"

# Step 1: Upload input file
curl -X POST "${BASE_URL}/files?api-version=${API_VERSION}" \
  -H "api-key: ${AZURE_OPENAI_KEY}" \
  -F "purpose=batch" \
  -F "file=@batch_input.jsonl"
# Returns file_id: "file-batch123"

# Step 2: Create batch job
curl -X POST "${BASE_URL}/batches?api-version=${API_VERSION}" \
  -H "api-key: ${AZURE_OPENAI_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "input_file_id": "file-batch123",
    "endpoint": "/chat/completions",
    "completion_window": "24h"
  }'
# Returns batch_id: "batch_abc789"

# Step 3: Monitor batch status
curl -X GET "${BASE_URL}/batches/batch_abc789?api-version=${API_VERSION}" \
  -H "api-key: ${AZURE_OPENAI_KEY}"
# status: "validating" -> "in_progress" -> "completed"

# Step 4: Retrieve results
# Get output_file_id from batch status response
curl -X GET "${BASE_URL}/files/file-output456/content?api-version=${API_VERSION}" \
  -H "api-key: ${AZURE_OPENAI_KEY}" \
  --output batch_results.jsonl

# Step 5: List all batches
curl -X GET "${BASE_URL}/batches?api-version=${API_VERSION}" \
  -H "api-key: ${AZURE_OPENAI_KEY}"

# Cancel a batch
curl -X POST "${BASE_URL}/batches/batch_abc789/cancel?api-version=${API_VERSION}" \
  -H "api-key: ${AZURE_OPENAI_KEY}"

Batch Output Format

Each line in the output JSONL file:

{"id": "response-1", "custom_id": "request-1", "response": {"status_code": 200, "body": {"choices": [{"message": {"role": "assistant", "content": "Positive"}, "finish_reason": "stop"}], "usage": {"prompt_tokens": 25, "completion_tokens": 2, "total_tokens": 27}}}}
{"id": "response-2", "custom_id": "request-2", "response": {"status_code": 200, "body": {"choices": [{"message": {"role": "assistant", "content": "Negative"}, "finish_reason": "stop"}], "usage": {"prompt_tokens": 27, "completion_tokens": 2, "total_tokens": 29}}}}

Failed requests include error details in the response body. Always check status_code for each result.

8. Embeddings

Deploy and Use Embeddings

# Deploy embedding model
az cognitiveservices account deployment create \
  --name my-openai-resource \
  --resource-group rg-openai \
  --deployment-name embeddings \
  --model-name text-embedding-3-small \
  --model-version "1" \
  --model-format OpenAI \
  --sku-name Standard \
  --sku-capacity 120

# Generate embeddings
curl -X POST "https://${RESOURCE_NAME}.openai.azure.com/openai/deployments/embeddings/embeddings?api-version=2024-08-01-preview" \
  -H "api-key: ${AZURE_OPENAI_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "input": ["Azure OpenAI provides enterprise-grade AI models", "Machine learning on Azure cloud platform"],
    "dimensions": 1536
  }'

Model comparison:

Dimension reduction: Use the dimensions parameter to reduce vector size. Lower dimensions = smaller storage, faster search, slightly lower quality. text-embedding-3-small with 512 dimensions is a good balance for most RAG applications.

Integration with Azure AI Search

Embeddings power vector search in Azure AI Search:

1. Generate embeddings for documents during indexing
2. Generate embeddings for user queries at search time
3. Use cosine similarity for ranking
4. Combine with keyword search for hybrid retrieval (recommended)

# Index documents with embeddings (AI Search REST API)
curl -X POST "https://my-search.search.windows.net/indexes/my-index/docs/index?api-version=2024-07-01" \
  -H "api-key: ${SEARCH_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "value": [
      {
        "@search.action": "upload",
        "id": "doc1",
        "content": "Azure OpenAI overview document...",
        "contentVector": [0.0023, -0.0142, ...]
      }
    ]
  }'

9. Quota & Rate Limits

Understanding Quotas

Azure OpenAI uses Tokens Per Minute (TPM) as the primary quota unit. Requests Per Minute (RPM) is derived from TPM.

TPM to RPM conversion (approximate):

Viewing and Managing Quotas

# View quota usage for a subscription
az cognitiveservices usage list \
  --location eastus \
  --query "[?name.value=='OpenAI.Standard.gpt-4o'].{Model:name.value, Current:currentValue, Limit:limit}" \
  --output table

# View deployment-level metrics via Azure Monitor
az monitor metrics list \
  --resource "/subscriptions/${SUB_ID}/resourceGroups/rg-openai/providers/Microsoft.CognitiveServices/accounts/my-openai-resource" \
  --metric "TokenTransaction" \
  --dimension "ModelDeploymentName" \
  --interval PT1H \
  --start-time "$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --output table

# View active tokens per minute
az monitor metrics list \
  --resource "/subscriptions/${SUB_ID}/resourceGroups/rg-openai/providers/Microsoft.CognitiveServices/accounts/my-openai-resource" \
  --metric "ActiveTokens" \
  --dimension "ModelDeploymentName" \
  --interval PT1M \
  --output table

Handling 429 Rate Limits

When you exceed your quota, the API returns HTTP 429 with a Retry-After header.

Retry pattern with exponential backoff:

import time
import openai

def call_with_retry(client, messages, deployment, max_retries=5):
    for attempt in range(max_retries):
        try:
            return client.chat.completions.create(
                model=deployment,
                messages=messages,
            )
        except openai.RateLimitError as e:
            retry_after = int(e.response.headers.get("Retry-After", 2 ** attempt))
            if attempt == max_retries - 1:
                raise
            time.sleep(min(retry_after, 60))
        except openai.APITimeoutError:
            if attempt == max_retries - 1:
                raise
            time.sleep(2 ** attempt)

Strategies for high-throughput workloads:

1. Multiple deployments: Spread load across multiple deployments of the same model
2. Multiple resources: Create OpenAI resources in different regions for aggregate quota
3. Batch API: Use for non-time-sensitive workloads (50% cheaper, separate quota)
4. Provisioned throughput: Purchase PTUs for guaranteed capacity
5. Request queuing: Implement a queue with rate limiting to smooth traffic spikes

10. Monitoring & Diagnostics

Azure Monitor Metrics

Key metrics for Azure OpenAI resources:

# Create an alert for high error rate
az monitor metrics alert create \
  --name "openai-high-errors" \
  --resource-group rg-openai \
  --scopes "/subscriptions/${SUB_ID}/resourceGroups/rg-openai/providers/Microsoft.CognitiveServices/accounts/my-openai-resource" \
  --condition "total ClientErrors > 100" \
  --window-size 5m \
  --evaluation-frequency 1m \
  --action "/subscriptions/${SUB_ID}/resourceGroups/rg-openai/providers/Microsoft.Insights/actionGroups/ops-team"

# Create an alert for PTU utilization
az monitor metrics alert create \
  --name "openai-ptu-high-utilization" \
  --resource-group rg-openai \
  --scopes "/subscriptions/${SUB_ID}/resourceGroups/rg-openai/providers/Microsoft.CognitiveServices/accounts/my-openai-resource" \
  --condition "avg ProvisionedManagedUtilizationV2 > 90" \
  --window-size 15m \
  --evaluation-frequency 5m \
  --action "/subscriptions/${SUB_ID}/resourceGroups/rg-openai/providers/Microsoft.Insights/actionGroups/ops-team"

Diagnostic Settings

# Enable diagnostic logging to Log Analytics
az monitor diagnostic-settings create \
  --resource "/subscriptions/${SUB_ID}/resourceGroups/rg-openai/providers/Microsoft.CognitiveServices/accounts/my-openai-resource" \
  --name "openai-diagnostics" \
  --workspace "/subscriptions/${SUB_ID}/resourceGroups/rg-openai/providers/Microsoft.OperationalInsights/workspaces/my-law" \
  --logs '[{"categoryGroup":"allLogs","enabled":true}]' \
  --metrics '[{"category":"AllMetrics","enabled":true}]'

KQL Queries for Log Analytics

// Token usage by deployment (last 24h)
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where TimeGenerated > ago(24h)
| summarize TotalTokens=sum(toint(properties_s.totalTokens))
    by DeploymentName=tostring(properties_s.modelDeploymentName)
| order by TotalTokens desc

// Error rate by deployment
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where TimeGenerated > ago(1h)
| summarize
    TotalRequests = count(),
    Errors = countif(toint(resultSignature_d) >= 400)
    by DeploymentName=tostring(properties_s.modelDeploymentName)
| extend ErrorRate = round(100.0 * Errors / TotalRequests, 2)

// P50/P95/P99 latency
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where TimeGenerated > ago(1h)
| summarize
    P50 = percentile(toint(DurationMs), 50),
    P95 = percentile(toint(DurationMs), 95),
    P99 = percentile(toint(DurationMs), 99)
    by DeploymentName=tostring(properties_s.modelDeploymentName)

// Content filter triggers
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where resultSignature_d == 400
| where properties_s contains "content_filter"
| summarize FilterTriggers = count() by bin(TimeGenerated, 1h)

Cost Tracking

# View Azure OpenAI costs for current billing period
az consumption usage list \
  --start-date "$(date -u -d '30 days ago' +%Y-%m-%d)" \
  --end-date "$(date -u +%Y-%m-%d)" \
  --query "[?contains(instanceName, 'openai')].{Resource:instanceName, Cost:pretaxCost, Currency:currency, UsageStart:usageStart}" \
  --output table

11. Security

Managed Identity Authentication (Recommended)

Use Azure AD authentication instead of API keys for production:

# Assign system-assigned managed identity
az cognitiveservices account identity assign \
  --name my-openai-resource \
  --resource-group rg-openai

# Grant "Cognitive Services OpenAI User" role to an identity
az role assignment create \
  --assignee <principal-id> \
  --role "Cognitive Services OpenAI User" \
  --scope "/subscriptions/${SUB_ID}/resourceGroups/rg-openai/providers/Microsoft.CognitiveServices/accounts/my-openai-resource"

# Grant "Cognitive Services OpenAI Contributor" for deployment management
az role assignment create \
  --assignee <principal-id> \
  --role "Cognitive Services OpenAI Contributor" \
  --scope "/subscriptions/${SUB_ID}/resourceGroups/rg-openai/providers/Microsoft.CognitiveServices/accounts/my-openai-resource"

RBAC roles:

| Role | Permissions | |----

…(truncated)