Spring AI Audio Image Multimodal

Spring AI - Audio, Image & Multimodal APIs

Description

Complete guide for audio (transcription, text-to-speech), image generation, and multimodal models in Spring AI. Covers supported models, streaming patterns, and integration patterns for rich media applications.

When to Use

• Speech-to-text transcription
• Text-to-speech audio generation
• Image generation from text prompts
• Vision models (image understanding)
• Multimodal applications (text + images + audio)
• Accessibility features
• Content creation automation
• Real-time audio processing

Topics Covered

1. Audio Capabilities

Speech-to-Text (Transcription)

• Providers: OpenAI Whisper, Azure, Google, others
• Input formats: MP3, WAV, M4A, WEBM, FLAC
• Languages: 99+ languages supported
• Streaming: Real-time transcription
• Quality levels: Accuracy vs speed tradeoff
• Post-processing: Punctuation, language model

Text-to-Speech (TTS)

• Providers: OpenAI, Azure, Google, Amazon Polly
• Voice selection: 100+ voices in multiple languages
• Speed control: Adjustable playback speed
• Format: MP3, WAV, AAC output
• Streaming: Real-time audio generation
• Emotion/style: Provider-specific variations

2. Image Capabilities

Image Generation

• Providers: OpenAI DALL-E 3, Azure, Bedrock, others
• Models: Latest generative models
• Quality: Resolution and detail control
• Style: Artistic styles, photorealistic
• Batch generation: Multiple images
• Editing: Image-to-image variations

Image Understanding

• Vision models: GPT-4o, Claude 3 Vision, Gemini Vision
• Tasks: Describe, classify, OCR, analysis
• Multiple images: Support for image sequences
• Chart understanding: Extract data from charts
• Document analysis: Extract text from documents

3. Multimodal Models

• GPT-4o Vision: Text + image input/output
• Claude 3 Opus/Sonnet/Haiku: Vision capabilities
• Google Gemini: Integrated multimodal
• Amazon Bedrock: Multiple multimodal models
• Mixing modalities: Text + image + structured output

4. Streaming Audio/Images

• Chunked responses: Stream audio/image bytes
• Reactive streams: Flux-based handling
• Backpressure: Flow control
• Error handling: Partial results
• Cancellation: Stop streaming mid-operation

Code Patterns

Speech-to-Text (Transcription)

@Configuration
public class AudioConfig {
    @Bean
    public SpeechApi speechApi(OpenAiApi openAiApi) {
        return openAiApi.speechApi();
    }
}

@Service
public class TranscriptionService {
    @Autowired
    private SpeechApi speechApi;

    public String transcribeAudio(File audioFile) {
        byte[] audioBytes = readFile(audioFile);

        TranscriptionRequest request = TranscriptionRequest.builder()
            .withModel("whisper-1")
            .withLanguage("en")
            .withPrompt("Context hint")
            .withTemperature(0.0f)
            .build();

        TranscriptionResponse response = speechApi.transcribe(
            audioBytes,
            request
        );

        return response.getText();
    }
}

Text-to-Speech

@Service
public class TextToSpeechService {
    @Autowired
    private SpeechApi speechApi;

    public byte[] generateSpeech(String text) {
        SpeechRequest request = SpeechRequest.builder()
            .withModel("tts-1-hd")
            .withVoice("nova")
            .withSpeed(1.0f)
            .build();

        return speechApi.textToSpeech(text, request);
    }
}

Image Generation

@Service
public class ImageGenerationService {
    @Autowired
    private ImageModel imageModel;

    public byte[] generateImage(String prompt) {
        ImagePrompt imagePrompt = new ImagePrompt(prompt,
            ImageOptions.builder()
                .withModel("dall-e-3")
                .withN(1)
                .withHeight(1024)
                .withWidth(1024)
                .withQuality("hd")
                .build());

        ImageResponse response = imageModel.call(imagePrompt);

        return downloadImage(
            response.getResult().getOutput().getB64Json()
        );
    }
}

Image Understanding (Vision)

@Service
public class ImageUnderstandingService {
    @Autowired
    private ChatClient chatClient;

    public String describeImage(String imageUrl) {
        return chatClient.prompt()
            .user("Describe this image",
                new ImageContent("image/url", imageUrl))
            .call()
            .content();
    }

    public String analyzeDocument(File documentImage) {
        byte[] imageBytes = readFile(documentImage);
        String base64 = Base64.encoder.encodeToString(imageBytes);

        return chatClient.prompt()
            .user("Extract all text from this document",
                new ImageContent("image/jpeg",
                    "data:image/jpeg;base64," + base64))
            .call()
            .content();
    }
}

Multimodal Chat (Text + Image)

@Service
public class MultimodalService {
    @Autowired
    private ChatClient chatClient;

    public String analyzeBoth(String question, String imageUrl) {
        return chatClient.prompt()
            .user(question)
            .user("And here is the image:",
                new ImageContent("image/url", imageUrl))
            .call()
            .content();
    }

    public ChatResponse multimodalAnalysis(
            String textContext,
            String imageUrl,
            String question) {

        return chatClient.prompt()
            .system("You are a multimodal analyzer")
            .user("Context: " + textContext)
            .user("Image:", new ImageContent("image/url", imageUrl))
            .user("Question: " + question)
            .call()
            .chatResponse();
    }
}

Streaming Audio

@Service
public class StreamingAudioService {
    @Autowired
    private SpeechApi speechApi;

    public Flux<byte[]> streamSpeech(String text) {
        SpeechRequest request = SpeechRequest.builder()
            .withModel("tts-1")
            .withVoice("echo")
            .withResponseFormat("mp3")
            .build();

        return Flux.create(sink -> {
            try {
                Flux<byte[]> audioStream =
                    speechApi.textToSpeechStream(text, request);

                audioStream.subscribe(
                    chunk -> sink.next(chunk),
                    error -> sink.error(error),
                    () -> sink.complete()
                );
            } catch (Exception e) {
                sink.error(e);
            }
        });
    }
}

Batch Image Generation

@Service
public class BatchImageService {
    @Autowired
    private ImageModel imageModel;

    public List<byte[]> generateImageVariations(
            String prompt,
            int count) {

        ImagePrompt imagePrompt = new ImagePrompt(prompt,
            ImageOptions.builder()
                .withModel("dall-e-3")
                .withN(count)
                .withSize("1024x1024")
                .build());

        ImageResponse response = imageModel.call(imagePrompt);

        return response.getResults().stream()
            .map(result -> downloadImage(result.getOutput().getB64Json()))
            .toList();
    }
}

Multimodal with Structured Output

@Service
public class StructuredMultimodalService {
    @Autowired
    private ChatClient chatClient;

    public ImageAnalysisResult analyzeAndStructure(
            String imageUrl) {

        return chatClient.prompt()
            .user("Analyze this image",
                new ImageContent("image/url", imageUrl))
            .call()
            .entity(ImageAnalysisResult.class);
    }
}

record ImageAnalysisResult(
    String description,
    List<String> objects,
    String dominantColor,
    float confidenceScore
) {}

Configuration

Audio Setup

@Configuration
public class AudioConfiguration {
    @Bean
    public OpenAiSpeechClient speechClient(
            OpenAiApi openAiApi) {
        return new OpenAiSpeechClient(openAiApi);
    }
}

Image Setup

@Configuration
public class ImageConfiguration {
    @Bean
    public ImageModel imageModel(OpenAiApi openAiApi) {
        return new OpenAiImageModel(openAiApi,
            OpenAiImageOptions.builder()
                .withModel("dall-e-3")
                .withQuality("standard")
                .build());
    }
}

Properties

# Audio
spring.ai.openai.speech.model=tts-1-hd
spring.ai.openai.speech.voice=nova
spring.ai.openai.speech.speed=1.0

# Images
spring.ai.openai.image.model=dall-e-3
spring.ai.openai.image.quality=hd
spring.ai.openai.image.size=1024x1024

# Streaming
spring.ai.streaming.enabled=true
spring.ai.streaming.chunk-size=1024

Best Practices

• Cache generated audio/images
• Handle streaming backpressure
• Implement appropriate timeouts
• Monitor API usage and costs
• Validate image URLs before processing
• Use appropriate models for tasks
• Implement error recovery
• Test with various media formats

Related Skills

• chat-models/SKILL.md - Chat integration
• structured-output/SKILL.md - Type-safe responses
• providers/SKILL.md - Provider setup

References

• API: /pages/api/audio.adoc, /pages/api/image.adoc
• Multimodal: /pages/api/multimodality.adoc
• Speech: /pages/api/speech.adoc, /pages/api/transcriptions.adoc