Smart Session Routing

Smart Session Routing

Provide AI chat applications with an intelligent decision framework for determining whether a user's new message should continue the current conversation or trigger a new session. This prevents context pollution, reduces hallucination from irrelevant history, and improves user experience.

Core Concept

Session Routing is the process of evaluating an incoming message against the current conversation context to decide the optimal routing path:

When to Apply

Apply session routing when building any AI chat interface that supports:

• Multi-turn conversations with persistent history
• Multiple concurrent sessions or conversation threads
• Long-running sessions where context may become stale
• AI agents that consume conversation history as context

Decision Signals (Quick Reference)

Five categories of signals determine the routing decision:

For detailed scoring rules and thresholds, consult references/decision-framework.md.

Quick Decision Flow

Incoming Message
    │
    ├─ Explicit new-session intent? ──YES──→ NEW SESSION
    │
    ├─ Context window >80% full?  ──YES──→ NEW SESSION (with summary carry-over)
    │
    ├─ Semantic similarity < 0.3?  ──YES──→ PROMPT USER or NEW SESSION
    │
    ├─ Time gap > threshold?       ──YES──→ PROMPT USER
    │
    └─ None of the above          ──────→ CONTINUE

Implementation Approaches

Approach 1: AI-Self-Judgment (Prompt-Based)

Embed routing logic into the system prompt. The AI model evaluates its own conversation and decides.

Best for: Quick integration, smaller teams, prototype phase.

Inject a routing analysis step before generating the response.
The AI outputs a structured decision (continue/new/prompt) before answering.

See examples/session-router-prompt.md for a complete prompt template.

Approach 2: Middleware / Service Layer

Implement routing as a dedicated service that intercepts messages before they reach the AI model.

Best for: Production systems, high-traffic applications, fine-grained control.

Key components:

• Semantic Analyzer: Embedding-based similarity between new message and conversation summary
• Intent Detector: Regex + NLU for explicit session commands
• Context Monitor: Token counter and window state tracker
• Router: Aggregates signals and makes the final decision

See references/implementation-patterns.md for architecture details.

Approach 3: Frontend Heuristics

Lightweight client-side checks before sending the message to the backend.

Best for: Augmenting server-side routing, instant UX feedback.

Implement quick checks in the message send flow:

• Intent keyword matching (instant, no API call)
• Time gap since last message (timestamp comparison)
• Message count / token estimation (simple arithmetic)

If any local check triggers, act immediately or prompt the user. Reserve server-side checks for ambiguous cases only.

Integration Checklist

When implementing session routing:

• Define explicit intent keywords for the target language(s)
• Set semantic similarity threshold (recommended: 0.3 for cosine similarity)
• Configure context window monitoring (trigger at 80% capacity)
• Implement graceful degradation (if routing fails, default to CONTINUE)
• Add "New Session" UI affordance so users can explicitly switch
• Log routing decisions for analysis and threshold tuning
• Handle summary carry-over when auto-creating new sessions

Additional Resources

Reference Files

For detailed decision logic and implementation architecture, consult:

• references/decision-framework.md - Complete scoring model, thresholds, edge cases, and decision matrix
• references/implementation-patterns.md - Architecture patterns, design principles, monitoring and tuning

Example Files

Calibration references and templates in examples/:

• examples/conversation-scenarios.md - 14 real-world conversation scenarios showing when each routing decision applies
• examples/session-router-prompt.md - Production-ready prompt template for AI-self-judgment approach