Environment Variables

OpenAI-Compatible

Overview

The OpenAI-Compatible provider enables you to use any service that implements the OpenAI API format. This includes local deployments, custom endpoints, and third-party services, giving you maximum flexibility while maintaining Esperanto's unified interface.

Supported Capabilities:

Official Documentation: Varies by implementation

Prerequisites

Supported Endpoints

Local Deployments:

• LM Studio - User-friendly local model server with GUI
• Ollama - Simple local model deployment (via OpenAI-compatible mode)
• vLLM - High-performance inference server
• LocalAI - Self-hosted OpenAI alternative
• text-generation-webui - Popular WebUI for local LLMs

Speech Services:

• Speaches - OpenAI-compatible server for faster-whisper (STT) and Piper/Kokoro (TTS)
• Shabdabhav - OpenAI-compatible server for Piper and Parler TTS models

Cloud Services:

• Custom OpenAI-format APIs
• Self-hosted embedding services
• Edge computing deployments

Installation Requirements

Requirements vary by endpoint. For LM Studio:

1. Download from https://lmstudio.ai
2. Load a model (LLM or embedding)
3. Start the local server (default: http://localhost:1234)

Environment Variables

# Generic (works for all OpenAI-compatible providers)
OPENAI_COMPATIBLE_BASE_URL="http://localhost:1234/v1"
OPENAI_COMPATIBLE_API_KEY="your-api-key"  # Optional for local endpoints

# Provider-specific (takes precedence, new in v2.7.0)
# Use these when you want different endpoints for different AI capabilities
OPENAI_COMPATIBLE_BASE_URL_LLM="http://localhost:1234/v1"
OPENAI_COMPATIBLE_API_KEY_LLM="your-key"

OPENAI_COMPATIBLE_BASE_URL_EMBEDDING="http://localhost:8080/v1"
OPENAI_COMPATIBLE_API_KEY_EMBEDDING="your-key"

OPENAI_COMPATIBLE_BASE_URL_STT="http://localhost:9000/v1"
OPENAI_COMPATIBLE_API_KEY_STT="your-key"

OPENAI_COMPATIBLE_BASE_URL_TTS="http://localhost:7000/v1"
OPENAI_COMPATIBLE_API_KEY_TTS="your-key"

Configuration Precedence (highest to lowest):

1. Direct parameters in config dictionary (base_url=, api_key=)
2. Provider-specific environment variables (OPENAI_COMPATIBLE_BASE_URL_LLM, etc.)
3. Generic environment variables (OPENAI_COMPATIBLE_BASE_URL, etc.)
4. Default values

This allows you to use different OpenAI-compatible endpoints for different AI capabilities (LLM, Embedding, STT, TTS) without code changes.

Quick Start

Via Factory (Recommended)

from esperanto.factory import AIFactory

# Language model
model = AIFactory.create_language(
    "openai-compatible",
    "your-model-name",
    config={"base_url": "http://localhost:1234/v1"}
)

# Embedding model
embedder = AIFactory.create_embedding(
    "openai-compatible",
    "nomic-embed-text",
    config={"base_url": "http://localhost:1234/v1"}
)

# Speech-to-text
transcriber = AIFactory.create_speech_to_text(
    "openai-compatible",
    "faster-whisper-large-v3",
    config={"base_url": "http://localhost:8000"}
)

# Text-to-speech
speaker = AIFactory.create_text_to_speech(
    "openai-compatible",
    "piper-tts",
    config={"base_url": "http://localhost:8000"}
)

Direct Instantiation

from esperanto.providers.llm.openai_compatible import OpenAICompatibleLanguageModel
from esperanto.providers.embedding.openai_compatible import OpenAICompatibleEmbeddingModel
from esperanto.providers.speech_to_text.openai_compatible import OpenAICompatibleSpeechToText
from esperanto.providers.text_to_speech.openai_compatible import OpenAICompatibleTextToSpeech

# Language model
llm = OpenAICompatibleLanguageModel(
    base_url="http://localhost:1234/v1",
    api_key="not-required",  # Often not needed
    model_name="your-model-name"
)

# Embedding model
embedder = OpenAICompatibleEmbeddingModel(
    base_url="http://localhost:1234/v1",
    model_name="nomic-embed-text"
)

# Speech-to-text
stt = OpenAICompatibleSpeechToText(
    base_url="http://localhost:8000",
    model_name="faster-whisper-large-v3"
)

# Text-to-speech
tts = OpenAICompatibleTextToSpeech(
    base_url="http://localhost:8000",
    model_name="piper-tts"
)

Capabilities

Language Models (LLM)

Common Endpoints:

• LM Studio: http://localhost:1234/v1
• Ollama: http://localhost:11434/v1
• vLLM: http://localhost:8000/v1
• Custom: https://your-endpoint.com/v1

Configuration:

from esperanto.factory import AIFactory

model = AIFactory.create_language(
    "openai-compatible",
    "lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF",
    config={
        "base_url": "http://localhost:1234/v1",  # Required
        "api_key": "lm-studio",                   # Optional
        "temperature": 0.7,                       # Standard OpenAI params
        "max_tokens": 1000,
        "streaming": True
    }
)

Example - LM Studio:

from esperanto.factory import AIFactory

# Connect to LM Studio running locally
model = AIFactory.create_language(
    "openai-compatible",
    "lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF",
    config={
        "base_url": "http://localhost:1234/v1",
        "api_key": "lm-studio"
    }
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain quantum computing in simple terms."}
]

# Regular completion
response = model.chat_complete(messages)
print(response.choices[0].message.content)

# Streaming completion
for chunk in model.chat_complete(messages, stream=True):
    print(chunk.choices[0].delta.content, end="", flush=True)

Example - vLLM:

# Connect to vLLM server
model = AIFactory.create_language(
    "openai-compatible",
    "facebook/opt-125m",
    config={
        "base_url": "http://localhost:8000/v1",
        "temperature": 0.8
    }
)

response = model.chat_complete(messages)

Example - Custom Endpoint:

# Use any OpenAI-compatible endpoint
model = AIFactory.create_language(
    "openai-compatible",
    "custom-model",
    config={
        "base_url": "https://api.yourcompany.com/v1",
        "api_key": "your-service-token"
    }
)

response = model.chat_complete(messages)

Example - JSON Mode:

# Note: JSON mode support depends on endpoint implementation
model = AIFactory.create_language(
    "openai-compatible",
    "your-model",
    config={
        "base_url": "http://localhost:1234/v1",
        "structured": {"type": "json"}
    }
)

messages = [{
    "role": "user",
    "content": "List three programming languages as JSON"
}]

response = model.chat_complete(messages)

Embeddings

Common Models (via LM Studio/LocalAI):

• nomic-embed-text (768 dimensions)
• all-MiniLM-L6-v2 (384 dimensions)
• e5-large-v2 (1024 dimensions)
• bge-large-en-v1.5 (1024 dimensions)

Configuration:

from esperanto.factory import AIFactory
from esperanto.common_types.task_type import EmbeddingTaskType

embedder = AIFactory.create_embedding(
    "openai-compatible",
    "nomic-embed-text",
    config={
        "base_url": "http://localhost:1234/v1",
        "api_key": "not-required",              # Often not needed
        "timeout": 120,                          # Default: 120 seconds
        "task_type": EmbeddingTaskType.RETRIEVAL_DOCUMENT  # Task optimization via prefixes
    }
)

Example - Basic Embeddings:

from esperanto.factory import AIFactory

# Create embedding model
embedder = AIFactory.create_embedding(
    "openai-compatible",
    "nomic-embed-text",
    config={"base_url": "http://localhost:1234/v1"}
)

# Generate embeddings
texts = ["Hello, world!", "Local embeddings are great"]
response = embedder.embed(texts)

# Access embeddings
for i, embedding_obj in enumerate(response.data):
    print(f"Text {i}: {len(embedding_obj.embedding)} dimensions")

Example - Large Batch Processing:

# Increase timeout for large batches
embedder = AIFactory.create_embedding(
    "openai-compatible",
    "nomic-embed-text",
    config={
        "base_url": "http://localhost:1234/v1",
        "timeout": 300  # 5 minutes for large batches
    }
)

# Process many documents
large_batch = [f"Document {i}" for i in range(1000)]
response = embedder.embed(large_batch)

Example - Task-Optimized:

from esperanto.common_types.task_type import EmbeddingTaskType

# Query embeddings
query_model = AIFactory.create_embedding(
    "openai-compatible",
    "nomic-embed-text",
    config={
        "base_url": "http://localhost:1234/v1",
        "task_type": EmbeddingTaskType.RETRIEVAL_QUERY
    }
)

# Document embeddings
doc_model = AIFactory.create_embedding(
    "openai-compatible",
    "nomic-embed-text",
    config={
        "base_url": "http://localhost:1234/v1",
        "task_type": EmbeddingTaskType.RETRIEVAL_DOCUMENT
    }
)

Speech-to-Text

Supported Implementations:

• Speaches - faster-whisper models
• Custom faster-whisper deployments
• OpenAI Whisper servers
• Custom STT endpoints

Configuration:

from esperanto.factory import AIFactory

transcriber = AIFactory.create_speech_to_text(
    "openai-compatible",
    "faster-whisper-large-v3",
    config={
        "base_url": "http://localhost:8000",
        "api_key": "your-api-key",  # Optional
        "timeout": 600               # 10 minutes for large files
    }
)

Example - Basic Transcription:

from esperanto.factory import AIFactory

# Create STT model
stt = AIFactory.create_speech_to_text(
    "openai-compatible",
    "faster-whisper-large-v3",
    config={"base_url": "http://localhost:8000"}
)

# Transcribe audio
response = stt.transcribe("meeting.mp3")
print(response.text)

Example - With Language and Context:

# Improve accuracy with language and context
response = stt.transcribe(
    "podcast.wav",
    language="en",
    prompt="This is a technical discussion about AI and machine learning"
)
print(f"Transcription: {response.text}")
print(f"Language: {response.language}")

Example - Async Transcription:

async def transcribe_batch():
    stt = AIFactory.create_speech_to_text(
        "openai-compatible",
        "faster-whisper-large-v3",
        config={"base_url": "http://localhost:8000"}
    )

    files = ["audio1.mp3", "audio2.wav", "audio3.m4a"]
    for audio_file in files:
        response = await stt.atranscribe(audio_file)
        print(f"{audio_file}: {response.text[:100]}...")

# Run async
# await transcribe_batch()

Text-to-Speech

Supported Implementations:

• Speaches - Piper, Kokoro models
• Shabdabhav - Piper and Parler models
• Custom Piper-TTS deployments
• Custom TTS endpoints

Configuration:

from esperanto.factory import AIFactory

speaker = AIFactory.create_text_to_speech(
    "openai-compatible",
    "piper-tts",
    config={
        "base_url": "http://localhost:8000",  # Required
        "api_key": "your-api-key"             # Optional
    }
)

Example - Basic TTS:

from esperanto.factory import AIFactory

# Create TTS model
tts = AIFactory.create_text_to_speech(
    "openai-compatible",
    "speaches-ai/Kokoro-82M-v1.0-ONNX",
    config={"base_url": "http://localhost:8000"}
)

# Generate speech
response = tts.generate_speech(
    text="Hello from OpenAI-compatible TTS!",
    voice="af_heart",  # Use voice supported by your endpoint
    output_file="output.mp3"
)

print(f"Generated {len(response.audio_data)} bytes of audio")

Example - Async TTS:

async def generate_speech_async():
    tts = AIFactory.create_text_to_speech(
        "openai-compatible",
        "piper-tts",
        config={"base_url": "http://localhost:8000"}
    )

    response = await tts.agenerate_speech(
        text="Async speech generation example",
        voice="en_US-amy-medium"
    )
    return response.audio_data

# Run async
# audio = await generate_speech_async()

Example - Custom Parameters:

# Pass additional parameters supported by your endpoint
response = tts.generate_speech(
    text="Custom speech generation",
    voice="en_US-amy-medium",
    speed=1.2,      # Custom parameter
    format="wav",   # Custom parameter
    quality="high"  # Custom parameter
)

Advanced Features

Multi-Endpoint Configuration

Use different endpoints for different capabilities:

# Set up environment for multiple endpoints
export OPENAI_COMPATIBLE_BASE_URL_LLM="http://localhost:1234/v1"
export OPENAI_COMPATIBLE_BASE_URL_EMBEDDING="http://localhost:8080/v1"
export OPENAI_COMPATIBLE_BASE_URL_STT="http://192.168.1.100:9000"
export OPENAI_COMPATIBLE_BASE_URL_TTS="http://192.168.1.100:7000"

# Each capability automatically uses its specific endpoint
llm = AIFactory.create_language("openai-compatible", "llama-3")
embedder = AIFactory.create_embedding("openai-compatible", "nomic-embed")
stt = AIFactory.create_speech_to_text("openai-compatible", "whisper")
tts = AIFactory.create_text_to_speech("openai-compatible", "piper")

Model Discovery

Some endpoints support listing available models:

# If endpoint supports /v1/models
model = AIFactory.create_language(
    "openai-compatible",
    "any-model",
    config={"base_url": "http://localhost:1234/v1"}
)

# The provider will query available models automatically

Graceful Degradation

The provider handles varying feature support:

try:
    # Attempt JSON mode
    model = AIFactory.create_language(
        "openai-compatible",
        "model-name",
        config={
            "base_url": "http://localhost:1234/v1",
            "structured": {"type": "json"}
        }
    )
    response = model.chat_complete(messages)
except Exception as e:
    # Falls back gracefully if not supported
    print(f"JSON mode not supported: {e}")

LangChain Integration

from esperanto.factory import AIFactory

model = AIFactory.create_language(
    "openai-compatible",
    "your-model",
    config={"base_url": "http://localhost:1234/v1"}
)

langchain_model = model.to_langchain()

# Use with LangChain
from langchain.chains import ConversationChain
chain = ConversationChain(llm=langchain_model)

Troubleshooting

Common Errors

Connection Error:

Error: Failed to connect to endpoint

Solution:

1. Ensure your endpoint is running and accessible
2. Verify the base URL is correct (include /v1 if required)
3. Check firewall/network settings

Authentication Error:

Error: Unauthorized

Solution:

1. Check if your endpoint requires an API key
2. Verify the API key is correct
3. Some local endpoints don't require authentication - try removing the api_key

Model Not Found:

Error: Model 'xxx' not found

Solution:

1. Verify your model name matches what's loaded in your endpoint
2. Check the endpoint's model list (if available)
3. For LM Studio: ensure the model is loaded in the GUI

Timeout Error:

Error: Request timed out

Solution: Increase timeout for large requests:

config={
    "base_url": "http://localhost:1234/v1",
    "timeout": 600  # 10 minutes
}

LM Studio Setup

1. Download and install LM Studio
2. Download a model from the "Discover" tab
3. Load the model in the "Chat" tab
4. Click the "Local Server" tab and start the server
5. Use base URL: http://localhost:1234/v1

LM Studio Example:

model = AIFactory.create_language(
    "openai-compatible",
    "lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF",
    config={
        "base_url": "http://localhost:1234/v1",
        "api_key": "not-required"
    }
)

Endpoint Compatibility

Required Endpoints:

• LLM: POST /v1/chat/completions
• Embedding: POST /v1/embeddings
• STT: POST /audio/transcriptions
• TTS: POST /audio/speech

Optional Endpoints:

• GET /v1/models - List available models
• GET /audio/voices - List available voices (TTS)

Use Cases

When to Choose OpenAI-Compatible

Perfect for:

• Privacy-sensitive applications (local processing)
• Local/on-premise deployments
• Custom or specialized models
• Cost optimization with high volume
• Development and testing environments
• Air-gapped/offline environments
• Integration with existing local infrastructure

Consider alternatives if:

• Want zero-setup cloud solution
• Need guaranteed enterprise SLA
• Prefer not to manage infrastructure
• Want cutting-edge cloud features immediately

Deployment Scenarios

1. Local Development:

# LM Studio for easy local testing
model = AIFactory.create_language(
    "openai-compatible",
    "local-model",
    config={"base_url": "http://localhost:1234/v1"}
)

2. Production Self-Hosted:

# vLLM for high-performance production
model = AIFactory.create_language(
    "openai-compatible",
    "production-model",
    config={
        "base_url": "https://llm.yourcompany.com/v1",
        "api_key": "your-service-token"
    }
)

3. Edge Computing:

# Local inference on edge devices
model = AIFactory.create_language(
    "openai-compatible",
    "edge-model",
    config={"base_url": "http://edge-device.local:8000/v1"}
)

See Also

• Language Models Guide
• Embeddings Guide
• Speech-to-Text Guide
• Text-to-Speech Guide
• Ollama Provider
• OpenAI Provider