Multimodal Prompts

Multimodal Prompts

Send images, audio, video, and files to LLMs within user messages. Koog supports two approaches: auto-configured attachments and custom-configured attachments.

Supported Media Types

Note: Multimodal content support varies by LLM provider. Check individual provider documentation.

Auto-Configured Attachments

Pass a URL or file path, and Koog automatically constructs the attachment parameters based on the file extension.

Kotlin — Images

import ai.koog.prompt.dsl.prompt

// From file path
val imagePrompt = prompt("image-analysis") {
    system("You are an image analysis assistant.")
    user {
        text("What's in this image?")
        image(Path("/path/to/photo.jpg"))
    }
}

// From URL
val urlPrompt = prompt("url-image") {
    user {
        text("What's in this image?")
        image("https://example.com/photo.jpg")
    }
}

Kotlin — Audio

val audioPrompt = prompt("audio-transcription") {
    system("Transcribe the following audio.")
    user {
        text("Please transcribe this:")
        audio(Path("/path/to/recording.mp3"))
    }
}

Kotlin — Video

val videoPrompt = prompt("video-analysis") {
    system("Analyze the content of this video.")
    user {
        text("What happens in this video?")
        video(Path("/path/to/clip.mp4"))
    }
}

Kotlin — Files

val filePrompt = prompt("document-analysis") {
    user {
        text("Summarize this document:")
        binaryFile(Path("/path/to/report.pdf"), mimeType = "application/pdf")
    }
}

val textFilePrompt = prompt("code-review") {
    user {
        text("Review this code:")
        textFile(Path("/path/to/code.kt"))
    }
}

Java

import ai.koog.prompt.dsl.Prompt;
import ai.koog.prompt.dsl.ContentPartsBuilder;

var imagePrompt = Prompt.builder("image-analysis")
    .system("You are an image analysis assistant.")
    .user(ContentPartsBuilder.create()
        .text("What's in this image?")
        .image(Path.of("/path/to/photo.jpg"))
        .build())
    .build();

Custom-Configured Attachments

The ContentPart interface allows fine-grained control over attachments:

ContentPart Types

Attachment Parameters

AttachmentContent Sources

Example — Custom Image Attachment

import ai.koog.prompt.message.ContentPart
import ai.koog.prompt.message.AttachmentContent

val customImagePrompt = prompt("custom-image") {
    user {
        text("Analyze this image:")
        attachment(
            ContentPart.Image(
                content = AttachmentContent.Binary.Bytes(imageBytes),
                format = "png",
                fileName = "screenshot.png"
            )
        )
    }
}

Example — Base64 Image

val base64Image = java.util.Base64.getEncoder().encodeToString(imageBytes)

val base64Prompt = prompt("base64-image") {
    user {
        text("Describe this image:")
        attachment(
            ContentPart.Image(
                content = AttachmentContent.Binary.Base64(base64Image),
                format = "png"
            )
        )
    }
}

Mixed Attachments

Combine multiple attachment types in a single user message:

val mixedPrompt = prompt("mixed-content") {
    system("You are a document analysis assistant.")
    user {
        text("Analyze the following:")
        text("1. This screenshot:")
        image(Path("/path/to/screenshot.png"))
        text("2. And this PDF report:")
        binaryFile(Path("/path/to/report.pdf"), mimeType = "application/pdf")
    }
}

Java — Mixed Attachments

var mixedPrompt = Prompt.builder("mixed-content")
    .system("You are a document analysis assistant.")
    .user(List.of(
        new ContentPart.Text("Analyze the following:"),
        new ContentPart.Text("1. This screenshot:"),
        new ContentPart.Image(
            new AttachmentContent.Binary.Bytes(imageBytes),
            "png", null, "screenshot.png"
        ),
        new ContentPart.Text("2. And this PDF report:"),
        new ContentPart.File(
            new AttachmentContent.Binary.Bytes(pdfBytes),
            "pdf", "application/pdf", "report.pdf"
        )
    ))
    .build();

Multiple Images

val comparePrompt = prompt("compare-images") {
    system("Compare the following images and describe the differences.")
    user {
        text("Image 1:")
        image(Path("/path/to/before.png"))
        text("Image 2:")
        image(Path("/path/to/after.png"))
        text("What changed between these images?")
    }
}

Provider-Specific Notes

Best Practices

1. Image size: Resize large images to reduce token usage
2. Format: Use JPEG for photos, PNG for screenshots/diagrams
3. Resolution: Higher resolution for OCR tasks, lower for general description
4. Batch: Combine related images in one prompt for comparison
5. Error handling: Handle unsupported formats gracefully
6. MIME types: Always specify MIME types for ContentPart.File attachments

Troubleshooting