Golem Add LLM Rust

Adding LLM and AI Capabilities (Rust)

Overview

Golem provides the golem-ai library collection — a set of Rust crates from golemcloud/golem-ai that provide unified, provider-agnostic APIs for AI capabilities. Each domain has a core crate (shared types and traits) plus provider crates (concrete backends). You add them as regular Cargo dependencies and call them directly from your agent code.

These crates are not on crates.io yet. Use git dependencies pointing to the v0.5.1 tag.

Available Libraries

LLM (Chat Completions)

Core crate: golem-ai-llm — unified chat completion API with blocking and streaming responses, multi-turn conversation, tool calling, and multimodal image inputs.

Provider crates (pick one):

Embeddings & Reranking

Core crate: golem-ai-embed — generate vector embeddings from text/images and rerank documents by relevance.

Provider crates:

Web Search

Core crate: golem-ai-web-search — unified web search with one-shot and paginated session modes.

Provider crates:

Document Search

Core crate: golem-ai-search — full-text/document search with index management, document CRUD, faceted search.

Provider crates:

Graph Databases

Core crate: golem-ai-graph — vertex/edge CRUD, traversal, path-finding, transactions, schema management.

Provider crates:

Vector Databases

Core crate: golem-ai-vector — collection management, vector upsert/search, ANN queries, namespaces.

Provider crates:

Video Generation

Core crate: golem-ai-video — text-to-video, image-to-video, async job polling.

Provider crates:

Speech-to-Text

Core crate: golem-ai-stt — audio transcription with speaker diarization, word-level timing.

Provider crates:

Text-to-Speech

Core crate: golem-ai-tts — voice discovery, batch/streaming synthesis, SSML support.

Provider crates:

Adding Dependencies

Add the core crate plus your chosen provider to the component's Cargo.toml:

[dependencies]
# LLM — core + provider
golem-ai-llm = "0.5.1"
golem-ai-llm-openai = "0.5.1"

Store the required API key as a secret using Golem's typed config system. Load the golem-add-secret-rust skill for full details. In brief:

use golem_rust::ConfigSchema;
use golem_rust::agentic::{Config, Secret};

#[derive(ConfigSchema)]
pub struct MyAgentConfig {
    #[config_schema(secret)]
    pub api_key: Secret<String>,
}

Then manage the secret via the CLI:

golem secret create api_key --secret-type String --secret-value "sk-..."

Usage: LLM Chat Completion

use golem_ai_llm::model::*;
use golem_ai_llm::LlmProvider;

// Pick a provider — type alias makes it easy to swap later
type Provider = golem_ai_llm_openai::DurableOpenAI;

let config = Config {
    model: "gpt-4o".to_string(),
    temperature: None,
    max_tokens: None,
    stop_sequences: None,
    tools: None,
    tool_choice: None,
    provider_options: None,
};

let events = vec![Event::Message(Message {
    role: Role::User,
    name: None,
    content: vec![ContentPart::Text("Hello!".to_string())],
})];

// Blocking request
let response = Provider::send(events, config).expect("LLM call failed");

// Extract text from response
let text: String = response
    .content
    .iter()
    .filter_map(|part| match part {
        ContentPart::Text(txt) => Some(txt.clone()),
        _ => None,
    })
    .collect::<Vec<_>>()
    .join("\n");

Usage: Multi-turn Conversation (Session)

use golem_ai_llm::model::*;
use golem_ai_llm::LlmProvider;

type Provider = golem_ai_llm_openai::DurableOpenAI;

// Keep events as agent state for multi-turn conversation
let mut events: Vec<Event> = vec![];

// Add system message
events.push(Event::Message(Message {
    role: Role::System,
    name: None,
    content: vec![ContentPart::Text("You are a helpful assistant.".to_string())],
}));

// Add user message
events.push(Event::Message(Message {
    role: Role::User,
    name: None,
    content: vec![ContentPart::Text("What is Golem?".to_string())],
}));

let config = Config {
    model: "gpt-4o".to_string(),
    temperature: None,
    max_tokens: None,
    stop_sequences: None,
    tools: None,
    tool_choice: None,
    provider_options: None,
};

// Send and record the response for next turn
let response = Provider::send(events.clone(), config.clone()).expect("LLM call failed");
events.push(Event::Response(response));

Usage: Web Search

use golem_ai_web_search::model::types;
use golem_ai_web_search::model::web_search;

type SearchProvider = golem_ai_web_search_google::DurableGoogleCustomSearch;

let session = SearchProvider::start_search(&web_search::SearchParams {
    query: "Golem distributed computing".to_string(),
    language: Some("lang_en".to_string()),
    safe_search: Some(types::SafeSearchLevel::Off),
    max_results: Some(10),
    time_range: None,
    include_domains: None,
    exclude_domains: None,
    include_images: None,
    include_html: None,
    advanced_answer: Some(true),
    region: None,
}).expect("Failed to start search");

let results = session.next_page().expect("Failed to get results");
for result in results {
    println!("{}: {}", result.title, result.url);
}

Switching Providers

To switch providers, change the type alias and dependency. The core API stays the same:

// Switch from OpenAI to Anthropic:
// 1. In Cargo.toml: replace golem-ai-llm-openai with golem-ai-llm-anthropic
// 2. In code:
type Provider = golem_ai_llm_anthropic::DurableAnthropic;
// All other code stays the same

Complete Agent Example

use golem_ai_llm::model::*;
use golem_ai_llm::LlmProvider;
use golem_rust::{agent_definition, agent_implementation, endpoint};

type Provider = golem_ai_llm_openai::DurableOpenAI;

#[agent_definition(mount = "/chats/{chat_name}")]
pub trait ChatAgent {
    fn new(chat_name: String) -> Self;

    #[endpoint(post = "/ask")]
    async fn ask(&mut self, question: String) -> String;
}

struct ChatAgentImpl {
    chat_name: String,
    events: Vec<Event>,
    config: Config,
}

#[agent_implementation]
impl ChatAgent for ChatAgentImpl {
    fn new(chat_name: String) -> Self {
        let config = Config {
            model: std::env::var("LLM_MODEL").unwrap_or_else(|_| "gpt-4o".to_string()),
            temperature: None,
            max_tokens: None,
            stop_sequences: None,
            tools: None,
            tool_choice: None,
            provider_options: None,
        };
        let events = vec![Event::Message(Message {
            role: Role::System,
            name: None,
            content: vec![ContentPart::Text(format!(
                "You are a helpful assistant for chat '{}'",
                chat_name
            ))],
        })];
        Self { chat_name, events, config }
    }

    async fn ask(&mut self, question: String) -> String {
        self.events.push(Event::Message(Message {
            role: Role::User,
            name: None,
            content: vec![ContentPart::Text(question)],
        }));

        let response = Provider::send(self.events.clone(), self.config.clone())
            .expect("LLM call failed");
        self.events.push(Event::Response(response.clone()));

        response
            .content
            .iter()
            .filter_map(|part| match part {
                ContentPart::Text(txt) => Some(txt.clone()),
                _ => None,
            })
            .collect::<Vec<_>>()
            .join("\n")
    }
}

Key Constraints

• All golem-ai crates should use version "0.5.1" from crates.io
• Always add both the core crate and a provider crate (e.g., golem-ai-llm + golem-ai-llm-openai)
• Provider API keys should be stored as secrets using Golem's typed config system (load the golem-add-secret-rust skill)
• The Durable* provider types (e.g., DurableOpenAI) automatically integrate with Golem's durable execution — responses are recorded in the oplog and replayed on recovery
• To switch providers, change the type alias and Cargo dependency — the rest of the code stays the same
• These crates target wasm32-wasip1 and work correctly in Golem's WebAssembly environment