Quickstart

litellm Plugin

Overview

The litellm plugin wraps the LiteLLM CLI. LiteLLM is a lightweight LLM API proxy that provides an OpenAI-format API for calling 100+ LLM providers including OpenAI, Anthropic, Cohere, Together AI, Replicate, Hugging Face, and many more.

What is litellm?

LiteLLM is a proxy server that translates your requests from OpenAI format to any LLM provider's format. It handles authentication, rate limiting, failover, and cost tracking across multiple providers.

Quick Start

1. Install litellm

pip install litellm

2. Start the proxy server

litellm --model gpt-3.5-turbo --port 8000

3. Use with OpenAI-compatible clients

# The proxy exposes an OpenAI-compatible API at http://localhost:8000
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "Hello"}]}'

Common Use Cases

Proxy with multiple models

litellm --model gpt-4 --model claude-3-opus --port 8000

Use with environment variables

export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
litellm --model gpt-4 --port 8000

Set model alias

litellm --model huggingface/codellama/CodeLlama-7b-Instruct-hf --alias codellama

Run with debug logging

litellm --model gpt-4 --debug

Custom API base

litellm --model gpt-4 --api_base https://my-custom-endpoint.com

Key Flags

• --host - Host to bind the server to (default: 0.0.0.0)
• --port - Port to bind (default: 8000)
• --model, -m - Model name to pass to litellm
• --alias - User-friendly alias for the model
• --api_base - Custom API base URL
• --api_version - API version (for Azure)
• --num_workers - Number of workers
• --add_key - Additional API key
• --headers - Custom headers for API calls
• --save - Save model-specific config
• --debug - Enable debug logging

Useful Commands

• sc litellm _ _ --model gpt-4 --port 8000 - Start proxy server
• sc litellm _ _ --model gpt-4 --debug - Start proxy with debug logging

Tips

• Set API keys as environment variables: OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.
• Use --alias to give models friendly names
• The proxy supports load balancing across multiple instances of the same model

Requirements

• Python 3.8+

Resources

• GitHub: https://github.com/BerriAI/litellm
• Docs: https://docs.litellm.ai