LLM Service Integration

LLM Service Integration for DEVS

When working with LLM functionality in the DEVS platform, always use the abstracted LLM service layer. Never call provider APIs directly.

Core Principle

DEVS is provider-agnostic. The LLMService in src/lib/llm/ abstracts multiple providers:

• OpenAI
• Anthropic (Claude)
• Google Gemini
• Mistral
• Ollama (local)
• Custom endpoints

Basic Usage

import { LLMService } from '@/lib/llm'
import type { Message } from '@/types'

async function generateResponse(userPrompt: string): Promise<string> {
  const messages: Message[] = [
    { role: 'system', content: 'You are a helpful assistant.' },
    { role: 'user', content: userPrompt },
  ]

  const response = await LLMService.chat(messages, {
    temperature: 0.7,
    maxTokens: 2000,
  })

  return response.content
}

Message Types

interface Message {
  role: 'system' | 'user' | 'assistant'
  content: string
  name?: string // For multi-agent conversations
}

Configuration Options

interface LLMConfig {
  temperature?: number // 0-1, default 0.7
  maxTokens?: number // Max response tokens
  model?: string // Override default model
  stream?: boolean // Enable streaming
  topP?: number // Nucleus sampling
  frequencyPenalty?: number // Reduce repetition
  presencePenalty?: number // Encourage new topics
}

Streaming Responses

For real-time UI updates:

import { LLMService } from '@/lib/llm'

async function streamResponse(
  messages: Message[],
  onChunk: (text: string) => void,
): Promise<string> {
  let fullResponse = ''

  await LLMService.streamChat(messages, {
    onChunk: (chunk) => {
      fullResponse += chunk
      onChunk(chunk)
    },
    temperature: 0.7,
  })

  return fullResponse
}

Error Handling

Always wrap LLM calls in try/catch:

import { LLMService } from '@/lib/llm'
import { toast } from '@/lib/toast'

async function safeGenerate(prompt: string): Promise<string | null> {
  try {
    const response = await LLMService.chat([{ role: 'user', content: prompt }])
    return response.content
  } catch (error) {
    console.error('LLM call failed:', error)
    toast.error('Failed to generate response. Please try again.')
    return null
  }
}

Agent Context Integration

When generating responses for agents, include their instructions:

import { LLMService } from '@/lib/llm'
import { getAgentById } from '@/stores/agentStore'
import type { Agent, Message } from '@/types'

async function generateAgentResponse(
  agentId: string,
  conversationHistory: Message[],
  userMessage: string,
): Promise<string> {
  const agent = getAgentById(agentId)
  if (!agent) throw new Error('Agent not found')

  const messages: Message[] = [
    {
      role: 'system',
      content: buildAgentSystemPrompt(agent),
    },
    ...conversationHistory,
    { role: 'user', content: userMessage },
  ]

  const response = await LLMService.chat(messages, {
    temperature: agent.temperature ?? 0.7,
  })

  return response.content
}

function buildAgentSystemPrompt(agent: Agent): string {
  return `You are ${agent.name}, ${agent.role}.

${agent.instructions}

Always stay in character and respond according to your role and expertise.`
}

Structured Output (JSON)

For extracting structured data:

import { LLMService } from '@/lib/llm'

interface ExtractedData {
  title: string
  summary: string
  keywords: string[]
}

async function extractStructuredData(text: string): Promise<ExtractedData> {
  const response = await LLMService.chat(
    [
      {
        role: 'system',
        content: `Extract information from the text and return as JSON:
{
  "title": "string",
  "summary": "string",
  "keywords": ["string"]
}
Return ONLY valid JSON, no other text.`,
      },
      { role: 'user', content: text },
    ],
    {
      temperature: 0.3, // Lower for more deterministic output
    },
  )

  // Parse with error handling
  try {
    return JSON.parse(response.content)
  } catch {
    // Attempt to extract JSON from response
    const jsonMatch = response.content.match(/\{[\s\S]*\}/)
    if (jsonMatch) {
      return JSON.parse(jsonMatch[0])
    }
    throw new Error('Failed to parse LLM response as JSON')
  }
}

Cost & Usage Tracking

The platform tracks LLM usage via the traces feature. Include metadata when relevant:

import { LLMService } from '@/lib/llm'

const response = await LLMService.chat(messages, {
  metadata: {
    feature: 'task-analysis',
    agentId: agent.id,
    taskId: task.id,
  },
})

Testing LLM Integration

Mock the LLM service in tests:

import { describe, it, expect, vi, beforeEach } from 'vitest'
import { LLMService } from '@/lib/llm'

vi.mock('@/lib/llm', () => ({
  LLMService: {
    chat: vi.fn(),
    streamChat: vi.fn(),
  },
}))

describe('MyFeature', () => {
  beforeEach(() => {
    vi.clearAllMocks()
  })

  it('should process LLM response correctly', async () => {
    vi.mocked(LLMService.chat).mockResolvedValue({
      content: '{"result": "success"}',
      usage: { promptTokens: 100, completionTokens: 50 },
    })

    const result = await myFunction()
    expect(result).toEqual({ result: 'success' })
  })
})

Common Patterns

Task Analysis

See src/lib/task-analyzer.ts for breaking down complex prompts.

Conversation Title Generation

See src/lib/conversation-title-generator.ts for generating chat titles.

Memory Learning

See src/lib/memory-learning-service.ts for extracting learnable facts from conversations.

Requirement Validation

See src/lib/requirement-validator.ts for validating task deliverables.