# Spring AI Audio Image Multimodal

> A complete guide for audio (transcription, text-to-speech), image generation, and multimodal models in Spring AI.

- Skill: `mat-garcia/spring-ai-audio-image-multimodal` (Agent Skill)
- Install (CLI): `npx skillmds@latest add mat-garcia/spring-ai-audio-image-multimodal`
- Raw SKILL.md: https://api.skillmd.com/api/skills/mat-garcia/spring-ai-audio-image-multimodal/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: Complete terms in LICENSE.txt
- Author: mat-garcia (https://skillmd.com/u/mat-garcia)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/mat-garcia/spring-ai-audio-image-multimodal

---


# Spring AI - Audio, Image & Multimodal APIs

## Description

Complete guide for audio (transcription, text-to-speech), image generation, and multimodal models in Spring AI. Covers supported models, streaming patterns, and integration patterns for rich media applications.

## When to Use

- Speech-to-text transcription
- Text-to-speech audio generation
- Image generation from text prompts
- Vision models (image understanding)
- Multimodal applications (text + images + audio)
- Accessibility features
- Content creation automation
- Real-time audio processing

## Topics Covered

### 1. Audio Capabilities

#### Speech-to-Text (Transcription)

- **Providers**: OpenAI Whisper, Azure, Google, others
- **Input formats**: MP3, WAV, M4A, WEBM, FLAC
- **Languages**: 99+ languages supported
- **Streaming**: Real-time transcription
- **Quality levels**: Accuracy vs speed tradeoff
- **Post-processing**: Punctuation, language model

#### Text-to-Speech (TTS)

- **Providers**: OpenAI, Azure, Google, Amazon Polly
- **Voice selection**: 100+ voices in multiple languages
- **Speed control**: Adjustable playback speed
- **Format**: MP3, WAV, AAC output
- **Streaming**: Real-time audio generation
- **Emotion/style**: Provider-specific variations

### 2. Image Capabilities

#### Image Generation

- **Providers**: OpenAI DALL-E 3, Azure, Bedrock, others
- **Models**: Latest generative models
- **Quality**: Resolution and detail control
- **Style**: Artistic styles, photorealistic
- **Batch generation**: Multiple images
- **Editing**: Image-to-image variations

#### Image Understanding

- **Vision models**: GPT-4o, Claude 3 Vision, Gemini Vision
- **Tasks**: Describe, classify, OCR, analysis
- **Multiple images**: Support for image sequences
- **Chart understanding**: Extract data from charts
- **Document analysis**: Extract text from documents

### 3. Multimodal Models

- **GPT-4o Vision**: Text + image input/output
- **Claude 3 Opus/Sonnet/Haiku**: Vision capabilities
- **Google Gemini**: Integrated multimodal
- **Amazon Bedrock**: Multiple multimodal models
- **Mixing modalities**: Text + image + structured output

### 4. Streaming Audio/Images

- **Chunked responses**: Stream audio/image bytes
- **Reactive streams**: Flux-based handling
- **Backpressure**: Flow control
- **Error handling**: Partial results
- **Cancellation**: Stop streaming mid-operation

## Code Patterns

### Speech-to-Text (Transcription)

```java
@Configuration
public class AudioConfig {
    @Bean
    public SpeechApi speechApi(OpenAiApi openAiApi) {
        return openAiApi.speechApi();
    }
}

@Service
public class TranscriptionService {
    @Autowired
    private SpeechApi speechApi;

    public String transcribeAudio(File audioFile) {
        byte[] audioBytes = readFile(audioFile);

        TranscriptionRequest request = TranscriptionRequest.builder()
            .withModel("whisper-1")
            .withLanguage("en")
            .withPrompt("Context hint")
            .withTemperature(0.0f)
            .build();

        TranscriptionResponse response = speechApi.transcribe(
            audioBytes,
            request
        );

        return response.getText();
    }
}
```

### Text-to-Speech

```java
@Service
public class TextToSpeechService {
    @Autowired
    private SpeechApi speechApi;

    public byte[] generateSpeech(String text) {
        SpeechRequest request = SpeechRequest.builder()
            .withModel("tts-1-hd")
            .withVoice("nova")
            .withSpeed(1.0f)
            .build();

        return speechApi.textToSpeech(text, request);
    }
}
```

### Image Generation

```java
@Service
public class ImageGenerationService {
    @Autowired
    private ImageModel imageModel;

    public byte[] generateImage(String prompt) {
        ImagePrompt imagePrompt = new ImagePrompt(prompt,
            ImageOptions.builder()
                .withModel("dall-e-3")
                .withN(1)
                .withHeight(1024)
                .withWidth(1024)
                .withQuality("hd")
                .build());

        ImageResponse response = imageModel.call(imagePrompt);

        return downloadImage(
            response.getResult().getOutput().getB64Json()
        );
    }
}
```

### Image Understanding (Vision)

```java
@Service
public class ImageUnderstandingService {
    @Autowired
    private ChatClient chatClient;

    public String describeImage(String imageUrl) {
        return chatClient.prompt()
            .user("Describe this image",
                new ImageContent("image/url", imageUrl))
            .call()
            .content();
    }

    public String analyzeDocument(File documentImage) {
        byte[] imageBytes = readFile(documentImage);
        String base64 = Base64.encoder.encodeToString(imageBytes);

        return chatClient.prompt()
            .user("Extract all text from this document",
                new ImageContent("image/jpeg",
                    "data:image/jpeg;base64," + base64))
            .call()
            .content();
    }
}
```

### Multimodal Chat (Text + Image)

```java
@Service
public class MultimodalService {
    @Autowired
    private ChatClient chatClient;

    public String analyzeBoth(String question, String imageUrl) {
        return chatClient.prompt()
            .user(question)
            .user("And here is the image:",
                new ImageContent("image/url", imageUrl))
            .call()
            .content();
    }

    public ChatResponse multimodalAnalysis(
            String textContext,
            String imageUrl,
            String question) {

        return chatClient.prompt()
            .system("You are a multimodal analyzer")
            .user("Context: " + textContext)
            .user("Image:", new ImageContent("image/url", imageUrl))
            .user("Question: " + question)
            .call()
            .chatResponse();
    }
}
```

### Streaming Audio

```java
@Service
public class StreamingAudioService {
    @Autowired
    private SpeechApi speechApi;

    public Flux<byte[]> streamSpeech(String text) {
        SpeechRequest request = SpeechRequest.builder()
            .withModel("tts-1")
            .withVoice("echo")
            .withResponseFormat("mp3")
            .build();

        return Flux.create(sink -> {
            try {
                Flux<byte[]> audioStream =
                    speechApi.textToSpeechStream(text, request);

                audioStream.subscribe(
                    chunk -> sink.next(chunk),
                    error -> sink.error(error),
                    () -> sink.complete()
                );
            } catch (Exception e) {
                sink.error(e);
            }
        });
    }
}
```

### Batch Image Generation

```java
@Service
public class BatchImageService {
    @Autowired
    private ImageModel imageModel;

    public List<byte[]> generateImageVariations(
            String prompt,
            int count) {

        ImagePrompt imagePrompt = new ImagePrompt(prompt,
            ImageOptions.builder()
                .withModel("dall-e-3")
                .withN(count)
                .withSize("1024x1024")
                .build());

        ImageResponse response = imageModel.call(imagePrompt);

        return response.getResults().stream()
            .map(result -> downloadImage(result.getOutput().getB64Json()))
            .toList();
    }
}
```

### Multimodal with Structured Output

```java
@Service
public class StructuredMultimodalService {
    @Autowired
    private ChatClient chatClient;

    public ImageAnalysisResult analyzeAndStructure(
            String imageUrl) {

        return chatClient.prompt()
            .user("Analyze this image",
                new ImageContent("image/url", imageUrl))
            .call()
            .entity(ImageAnalysisResult.class);
    }
}

record ImageAnalysisResult(
    String description,
    List<String> objects,
    String dominantColor,
    float confidenceScore
) {}
```

## Configuration

### Audio Setup

```java
@Configuration
public class AudioConfiguration {
    @Bean
    public OpenAiSpeechClient speechClient(
            OpenAiApi openAiApi) {
        return new OpenAiSpeechClient(openAiApi);
    }
}
```

### Image Setup

```java
@Configuration
public class ImageConfiguration {
    @Bean
    public ImageModel imageModel(OpenAiApi openAiApi) {
        return new OpenAiImageModel(openAiApi,
            OpenAiImageOptions.builder()
                .withModel("dall-e-3")
                .withQuality("standard")
                .build());
    }
}
```

### Properties

```properties
# Audio
spring.ai.openai.speech.model=tts-1-hd
spring.ai.openai.speech.voice=nova
spring.ai.openai.speech.speed=1.0

# Images
spring.ai.openai.image.model=dall-e-3
spring.ai.openai.image.quality=hd
spring.ai.openai.image.size=1024x1024

# Streaming
spring.ai.streaming.enabled=true
spring.ai.streaming.chunk-size=1024
```

## Best Practices

- Cache generated audio/images
- Handle streaming backpressure
- Implement appropriate timeouts
- Monitor API usage and costs
- Validate image URLs before processing
- Use appropriate models for tasks
- Implement error recovery
- Test with various media formats

## Related Skills

- `chat-models/SKILL.md` - Chat integration
- `structured-output/SKILL.md` - Type-safe responses
- `providers/SKILL.md` - Provider setup

## References

- API: `/pages/api/audio.adoc`, `/pages/api/image.adoc`
- Multimodal: `/pages/api/multimodality.adoc`
- Speech: `/pages/api/speech.adoc`, `/pages/api/transcriptions.adoc`