Spring AI Chat Models

Spring AI - Chat Models & ChatClient API

Description

Comprehensive guide for implementing and configuring Chat Models and ChatClient APIs in Spring AI. Covers all supported LLM providers (OpenAI, Azure OpenAI, Bedrock, Ollama, etc.), configuration options, response handling, and streaming patterns.

When to Use

• Setting up chat interactions with language models
• Configuring ChatModel or ChatClient beans
• Implementing chat completions with specific providers
• Handling chat options and model parameters
• Working with structured outputs and tool calling
• Streaming chat responses
• Integration with Spring Boot applications

Topics Covered

1. Supported Chat Providers

• OpenAI: GPT-4o, GPT-4, GPT-3.5-turbo, o1-preview, o1-mini
• Azure OpenAI: Full integration with Azure deployments
• Anthropic Claude: Claude 3 Opus, Sonnet, Haiku
• Google Gemini: Gemini Pro, Gemini 2.0
• Amazon Bedrock: Multiple Titan, Claude, and Llama models
• Ollama: Local model inference
• Mistral AI: Open models
• HuggingFace: Community models
• Other providers: Grok, Perplexity, DeepSeek

2. ChatModel API Core

• Request: Prompt + ChatOptions
• Response: ChatResponse (generation + metadata)
• Streaming: Reactive Flux for real-time output
• Call Metrics: Token usage, cost, latency tracking
• Non-streaming vs Streaming: Performance considerations

3. ChatClient Fluent API

• High-level abstraction over ChatModel
• Method chaining for configuration
• .user(), .system(), .messages()
• .call(), .stream() for execution
• .content(), .entity() for results

4. Chat Configuration

• Model/deployment selection
• Temperature, max_tokens, frequency_penalty
• Top_p, top_k sampling parameters
• Stop sequences
• Function calling mode (auto, none, required)

5. Response Handling

• Generation with usage information
• Tool calls in response
• Finish reason interpretation (STOP, LENGTH, TOOL_CALLS, etc.)
• Metadata extraction (model info, timestamps)

6. Structured Output

• Type-safe JSON responses
• StructuredOutputConverter (JSON Schema mode)
• Jackson and Gson support
• Automatic serialization/deserialization

7. Tool Calling Integration

• Automatic tool detection and binding
• Request model vs response format handling
• Exception handling and retries
• OpenAI, Anthropic, Google, Bedrock support

Code Patterns

Basic ChatModel Usage

@Configuration
public class ChatConfig {
    @Bean
    public ChatModel chatModel(OpenAiApi openAiApi) {
        return new OpenAiChatModel(openAiApi,
            OpenAiChatOptions.builder()
                .withModel("gpt-4o")
                .withTemperature(0.7f)
                .build());
    }
}

ChatClient with Streaming

chatClient.prompt()
    .user("Question: " + userQuery)
    .stream()
    .content()
    .doOnNext(content -> logger.info("Token: {}", content))
    .blockLast();

Structured Output

var response = chatClient.prompt()
    .user("Generate a person")
    .call()
    .entity(Person.class);

Configuration via Properties

spring.ai.openai.api-key=${OPENAI_API_KEY}
spring.ai.openai.chat.options.model=gpt-4o
spring.ai.openai.chat.options.temperature=0.7
spring.ai.openai.chat.options.max-tokens=1000

Related Skills

• embeddings/SKILL.md - Vector embeddings
• rag-retrieval/SKILL.md - RAG patterns
• tools-agents/SKILL.md - Tool calling
• advisors/SKILL.md - Chain of responsibility patterns
• structured-output/SKILL.md - Type-safe responses

References

• API: /pages/api/chatmodel.adoc, /pages/api/chatclient.adoc
• Providers: /pages/api/bedrock.adoc, OpenAI, Azure OpenAI docs
• Tools: /pages/api/tools.adoc, /pages/api/tools-migration.adoc
• Structured Output: /pages/api/structured-output-converter.adoc