Implementing Chat Streaming

Chat Streaming Implementation

Backend: SSE Endpoint

app.MapPost("/api/chat/stream", async (
    ChatRequest request,
    AgentFrameworkService agentService,
    HttpContext httpContext,
    CancellationToken cancellationToken) =>
{
    httpContext.Response.Headers.Append("Content-Type", "text/event-stream");
    httpContext.Response.Headers.Append("Cache-Control", "no-cache");
    
    var conversationId = request.ConversationId 
        ?? await agentService.CreateConversationAsync(request.Message, cancellationToken);
    
    // Send conversation ID first
    await httpContext.Response.WriteAsync(
        $"data: {{\"type\":\"conversationId\",\"conversationId\":\"{conversationId}\"}}\n\n", 
        cancellationToken);
    await httpContext.Response.Body.FlushAsync(cancellationToken);
    
    // Stream chunks
    await foreach (var chunk in agentService.StreamMessageAsync(
        conversationId, request.Message, request.ImageDataUris, cancellationToken))
    {
        var json = JsonSerializer.Serialize(new { type = "chunk", content = chunk });
        await httpContext.Response.WriteAsync($"data: {json}\n\n", cancellationToken);
        await httpContext.Response.Body.FlushAsync(cancellationToken);
    }
    
    await httpContext.Response.WriteAsync("data: {\"type\":\"done\"}\n\n", cancellationToken);
})
.RequireAuthorization("RequireChatScope");

Backend: IAsyncEnumerable Service

Actual return type: IAsyncEnumerable<StreamChunk> (not raw strings)

Why direct SDK? Uses ProjectResponsesClient directly because we need typed access to MCP approvals, file search quotes, and citation annotations. See .github/skills/researching-azure-ai-sdk/SKILL.md for full rationale.

public async IAsyncEnumerable<StreamChunk> StreamMessageAsync(
    string conversationId,
    string message,
    List<string>? imageDataUris = null,
    [EnumeratorCancellation] CancellationToken cancellationToken = default)
{
    ObjectDisposedException.ThrowIf(_disposed, this);
    
    // Stream response - yields StreamChunk with text deltas OR annotations
    await foreach (var update in responsesClient.CreateResponseStreamingAsync(...))
    {
        if (update is StreamingResponseOutputTextDeltaUpdate deltaUpdate)
        {
            yield return StreamChunk.Text(deltaUpdate.Delta);
        }
        else if (update is StreamingResponseOutputItemDoneUpdate itemDoneUpdate)
        {
            var annotations = ExtractAnnotations(itemDoneUpdate.Item, fileSearchQuotes);
            if (annotations.Count > 0)
            {
                yield return StreamChunk.WithAnnotations(annotations);
            }
        }
    }
}

StreamChunk model (backend/WebApp.Api/Models/StreamChunk.cs):

• IsText / TextDelta - Text content
• HasAnnotations / Annotations - Citation metadata

Frontend: Action Flow

CHAT_SEND_MESSAGE 
  → CHAT_ADD_ASSISTANT_MESSAGE 
  → CHAT_START_STREAM 
  → (repeat CHAT_STREAM_CHUNK) 
  → CHAT_STREAM_ANNOTATIONS (optional, for citations)
  → CHAT_STREAM_COMPLETE (with usage metrics)

If user cancels: CHAT_CANCEL_STREAM sets status to idle.

Frontend: ChatService Pattern

See: frontend/src/services/ChatService.ts

Key patterns:

• AbortController for cancellation
• EventSource or fetch with ReadableStream
• Parse SSE data: lines
• Dispatch actions for each event type

Image Validation

Backend limits (see AzureAIAgentService.cs):

• Max 5 images per request
• Max 5MB per image (decoded)
• Allowed: image/png, image/jpeg, image/gif, image/webp

Frontend limits (see frontend/src/utils/fileAttachments.ts):

• Same limits with user-friendly error messages
• Toast notifications for validation feedback

Project-Specific: Full Endpoint Implementation

app.MapPost("/api/chat/stream", async (
    ChatRequest request,
    AgentFrameworkService agentService,
    HttpContext httpContext,
    IHostEnvironment env,
    CancellationToken cancellationToken) =>
{
    httpContext.Response.Headers.Append("Content-Type", "text/event-stream");
    httpContext.Response.Headers.Append("Cache-Control", "no-cache");
    
    var conversationId = request.ConversationId 
        ?? await agentService.CreateConversationAsync(request.Message, cancellationToken);
    
    await httpContext.Response.WriteAsync(
        $"data: {{\"type\":\"conversationId\",\"conversationId\":\"{conversationId}\"}}\n\n", 
        cancellationToken);
    await httpContext.Response.Body.FlushAsync(cancellationToken);
    
    await foreach (var chunk in agentService.StreamMessageAsync(
        conversationId, request.Message, request.ImageDataUris, cancellationToken))
    {
        var json = System.Text.Json.JsonSerializer.Serialize(new { type = "chunk", content = chunk });
        await httpContext.Response.WriteAsync($"data: {json}\n\n", cancellationToken);
        await httpContext.Response.Body.FlushAsync(cancellationToken);
    }
    
    await httpContext.Response.WriteAsync("data: {\"type\":\"done\"}\n\n", cancellationToken);
})
.RequireAuthorization("RequireChatScope")
.WithName("StreamChatMessage");

Project-Specific: Service Implementation

See: backend/WebApp.Api/Services/AgentFrameworkService.cs

Key patterns in StreamMessageAsync:

• Disposal guard before processing
• Multi-modal message support (text + image data URIs)
• IAsyncEnumerable<StreamChunk> with [EnumeratorCancellation]
• StreamingResponseOutputTextDeltaUpdate for text content
• StreamingResponseOutputItemDoneUpdate for annotations
• Collects file search quotes via FileSearchCallResponseItem for citation context
• Usage captured from StreamingResponseCompletedUpdate

Project-Specific: Frontend State Flow

CHAT_SEND_MESSAGE 
  → CHAT_ADD_ASSISTANT_MESSAGE 
  → CHAT_START_STREAM 
  → (repeat CHAT_STREAM_CHUNK) 
  → CHAT_STREAM_ANNOTATIONS (optional, for citations)
  → CHAT_STREAM_COMPLETE (with usage: promptTokens, completionTokens, totalTokens, duration)

Cancel: CHAT_CANCEL_STREAM sets status to idle and re-enables input.

Error: CHAT_ERROR with AppError containing message, optional retry action, timestamp.

SSE Event Types

Project-Specific: Dev Logging

Each state change prints (dev only):

🔄 [HH:MM:SS] ACTION_TYPE
Action: { … }
Changes: { field: before → after }

Related Skills

• writing-csharp-code - Backend coding standards and AgentFrameworkService patterns
• writing-typescript-code - Frontend React patterns and ChatService implementation
• troubleshooting-authentication - Token acquisition for authenticated streaming