Running Prompts

Running Prompts

Execute prompts against LLM providers using Koog's two levels of abstraction: LLM Clients (low-level) and Prompt Executors (high-level).

Execution Flow

Prompt → LLM Client or Executor → LLM Provider → Response → Application

LLM Clients (Low-Level)

Direct interfaces for interacting with specific LLM providers. Best when working with a single provider without needing advanced lifecycle management.

Prompt Executors (High-Level)

Manage the lifecycle of one or multiple LLM clients with a unified API. Supports dynamic switching between providers and fallbacks.

Executor Types

Factory Functions

All providers offer a simple factory function that returns SingleLLMPromptExecutor:

Kotlin

import ai.koog.agents.ext.simple.simpleOpenAIExecutor
import ai.koog.agents.ext.llm.OpenAIModels
import ai.koog.prompt.dsl.prompt

// Create executor
val executor = simpleOpenAIExecutor(System.getenv("OPENAI_API_KEY"))

// Build prompt
val myPrompt = prompt("greeting") {
    system("You are a friendly assistant.")
    user("Hello!")
}

// Execute
val response = executor.execute(
    prompt = myPrompt,
    model = OpenAIModels.Chat.GPT4o
)

println(response.content)

Java

import ai.koog.agents.ext.simple.SimpleOpenAIExecutorKt;
import ai.koog.agents.ext.llm.OpenAIModels;
import ai.koog.prompt.dsl.Prompt;

var executor = SimpleOpenAIExecutorKt.simpleOpenAIExecutor(
    System.getenv("OPENAI_API_KEY")
);

var myPrompt = Prompt.builder("greeting")
    .system("You are a friendly assistant.")
    .user("Hello!")
    .build();

var response = executor.execute(myPrompt, OpenAIModels.Chat.GPT4o);
System.out.println(response.getContent());

Multi-Provider Executor

Switch between providers without changing code:

import ai.koog.agents.ext.prompt.executor.MultiLLMPromptExecutor
import ai.koog.agents.ext.llm.LLMProvider

val executor = MultiLLMPromptExecutor(
    LLMProvider.OpenAI to openAIClient,
    LLMProvider.Anthropic to anthropicClient,
    LLMProvider.Google to googleClient
)

// Route to OpenAI
val openAIResult = executor.execute(prompt, OpenAIModels.Chat.GPT4o)

// Route to Anthropic (same executor, different model)
val anthropicResult = executor.execute(prompt, AnthropicModels.Sonnet_4_5)

Configuring Fallbacks

Fallbacks apply when a requested client isn't available:

val executor = MultiLLMPromptExecutor(
    LLMProvider.OpenAI to openAIClient,
    LLMProvider.Ollama to ollamaClient,
    fallback = MultiLLMPromptExecutor.FallbackPromptExecutorSettings(
        fallbackProvider = LLMProvider.Ollama,
        fallbackModel = OllamaModels.Meta.LLAMA_3_2
    )
)

Routing Executor (Experimental)

Distribute requests across multiple client instances:

import ai.koog.agents.ext.prompt.executor.RoutingLLMPromptExecutor
import ai.koog.agents.ext.prompt.executor.RoundRobinRouter

val openAI1 = OpenAILLMClient(apiKey = "key-1")
val openAI2 = OpenAILLMClient(apiKey = "key-2")
val anthropic = AnthropicLLMClient(apiKey = "anthropic-key")

val router = RoundRobinRouter(openAI1, openAI2, anthropic)
val routingExecutor = RoutingLLMPromptExecutor(router)

Note: Requires @OptIn(ExperimentalRoutingApi::class).

Running a Prompt

val response = executor.execute(
    prompt = prompt("demo") {
        system("You are a helpful assistant.")
        user("Summarize this document.")
    },
    model = OpenAIModels.Chat.GPT4o
)

The executor supports streaming, multiple choice generation, and content moderation — matching whatever the underlying client supports.

Using with Agents

// Simple agent
val agent = AIAgent(
    promptExecutor = simpleOpenAIExecutor(apiKey),
    llmModel = OpenAIModels.Chat.GPT4o,
    systemPrompt = "You are a helpful assistant."
)

// Multi-provider agent
val multiAgent = AIAgent(
    promptExecutor = MultiLLMPromptExecutor(
        LLMProvider.OpenAI to openAIClient,
        LLMProvider.Anthropic to anthropicClient
    ),
    llmModel = OpenAIModels.Chat.GPT4o, // Default model
    systemPrompt = "You are a helpful assistant."
)

Troubleshooting