Together AI Inference

Together AI — Serverless Inference & Fine-Tuning

Together AI is an AI cloud platform providing serverless inference on 200+ open-source models through an OpenAI-compatible API. It supports chat completions, embeddings, fine-tuning, image generation, and batch processing at https://api.together.xyz/v1.

When to Use Together AI

Use Together AI when:

• You need fast serverless inference on open-source models (Llama, DeepSeek, Qwen, Mistral)
• You want an OpenAI-compatible API so you can swap providers with a single line change
• You need to fine-tune open-source models without managing GPU infrastructure
• You want cost-effective inference with pay-per-token pricing
• You need function calling, JSON mode, or structured outputs from open-source models
• You want batch processing at 50% lower cost for non-urgent workloads
• You need embeddings or image generation alongside chat completions

Use alternatives instead:

Credential Setup

Credentials are auto-injected by openscience when connected via the dashboard.

# Verify credentials
[ -n "$TOGETHER_API_KEY" ] && echo "TOGETHER_API_KEY set" || echo "NOT SET"

If not set: add your Together AI key in Customize → Models or export TOGETHER_API_KEY locally.

Quick Start

Install

pip install together openai

Set API Key

import os
os.environ["TOGETHER_API_KEY"] = "your-api-key"

# Or export in shell:
# export TOGETHER_API_KEY="your-api-key"

Get your API key from https://api.together.xyz/settings/api-keys

Basic Chat Completion

from together import Together

client = Together()

response = client.chat.completions.create(
    model="meta-llama/Llama-3.3-70B-Instruct-Reference",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain gradient descent in one paragraph."},
    ],
    max_tokens=256,
    temperature=0.7,
)

print(response.choices[0].message.content)

Inference

Chat Completions

The primary endpoint for conversational AI. Supports system/user/assistant messages, temperature, top_p, top_k, repetition_penalty, and stop sequences.

from together import Together

client = Together()

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3",
    messages=[
        {"role": "system", "content": "You are an expert ML researcher."},
        {"role": "user", "content": "Compare LoRA vs full fine-tuning."},
    ],
    max_tokens=512,
    temperature=0.7,
    top_p=0.9,
    top_k=50,
    repetition_penalty=1.1,
    stop=["</s>"],
)

print(response.choices[0].message.content)
print(f"Tokens used: {response.usage.total_tokens}")

Streaming

from together import Together

client = Together()

stream = client.chat.completions.create(
    model="meta-llama/Llama-3.3-70B-Instruct-Reference",
    messages=[{"role": "user", "content": "Write a haiku about transformers."}],
    max_tokens=128,
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)
print()

Function Calling

Supported on select models including Llama, DeepSeek, Qwen, and Mistral variants.

from together import Together
import json

client = Together()

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "City and state, e.g. San Francisco, CA",
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                    },
                },
                "required": ["location"],
            },
        },
    }
]

response = client.chat.completions.create(
    model="meta-llama/Llama-3.3-70B-Instruct-Reference",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What's the weather in San Francisco?"},
    ],
    tools=tools,
    tool_choice="auto",
)

tool_calls = response.choices[0].message.tool_calls
if tool_calls:
    call = tool_calls[0]
    print(f"Function: {call.function.name}")
    print(f"Arguments: {call.function.arguments}")

JSON Mode (Structured Outputs)

Force the model to return valid JSON conforming to a schema.

import json
from together import Together

client = Together()

schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "age": {"type": "integer"},
        "skills": {"type": "array", "items": {"type": "string"}},
    },
    "required": ["name", "age", "skills"],
}

response = client.chat.completions.create(
    model="meta-llama/Llama-3.3-70B-Instruct-Reference",
    messages=[
        {
            "role": "system",
            "content": f"Respond only in JSON matching this schema: {json.dumps(schema)}",
        },
        {"role": "user", "content": "Create a profile for a senior ML engineer."},
    ],
    response_format={
        "type": "json_object",
        "schema": schema,
    },
)

data = json.loads(response.choices[0].message.content)
print(json.dumps(data, indent=2))

Vision Models

from together import Together

client = Together()

response = client.chat.completions.create(
    model="meta-llama/Llama-4-Scout-17B-16E-Instruct-VLM",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this image in detail."},
                {
                    "type": "image_url",
                    "image_url": {"url": "https://example.com/image.jpg"},
                },
            ],
        }
    ],
    max_tokens=512,
)

print(response.choices[0].message.content)

Fine-Tuning

Together AI supports both LoRA and full fine-tuning on a wide range of open-source models.

Data Format

Training data must be in JSONL format with chat-style messages:

{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the capital of France?"}, {"role": "assistant", "content": "The capital of France is Paris."}]}
{"messages": [{"role": "user", "content": "Explain photosynthesis."}, {"role": "assistant", "content": "Photosynthesis is the process by which plants convert sunlight, water, and carbon dioxide into glucose and oxygen."}]}

Each line is one training example. The system role is optional. The model learns to generate the assistant responses.

Upload Training Data

from together import Together

client = Together()

# Upload training file
file = client.files.upload(file="training_data.jsonl")
print(f"File ID: {file.id}")

Create Fine-Tuning Job

from together import Together

client = Together()

# Create LoRA fine-tuning job
job = client.fine_tuning.create(
    training_file="file-abc123",
    model="meta-llama/Llama-3.3-70B-Instruct-Reference",
    n_epochs=3,
    learning_rate=1e-5,
    batch_size=4,
    lora=True,
    lora_r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    suffix="my-custom-model",
)

print(f"Job ID: {job.id}")
print(f"Status: {job.status}")

Monitor Fine-Tuning

from together import Together

client = Together()

# Check job status
job = client.fine_tuning.retrieve(id="ft-abc123")
print(f"Status: {job.status}")

# List all jobs
jobs = client.fine_tuning.list()
for j in jobs:
    print(f"{j.id}: {j.status}")

# List events (training logs)
events = client.fine_tuning.list_events(id="ft-abc123")
for event in events:
    print(event)

# Cancel a job
client.fine_tuning.cancel(id="ft-abc123")

CLI Fine-Tuning

# Install CLI
pip install --upgrade together

# Upload data
together files upload training_data.jsonl

# Create fine-tuning job
together fine-tuning create \
    --training-file file-abc123 \
    -m meta-llama/Meta-Llama-3.1-8B-Instruct-Reference

# Check status
together fine-tuning status ft-abc123

# List checkpoints
together fine-tuning list-checkpoints ft-abc123

# Download fine-tuned model
together fine-tuning download --ft-id ft-abc123

Supported Fine-Tuning Models (Selection)

See the full list at https://docs.together.ai/docs/fine-tuning-models

Embeddings

Generate Embeddings

from together import Together

client = Together()

response = client.embeddings.create(
    model="BAAI/bge-large-en-v1.5",
    input="What is the meaning of life?",
)

embedding = response.data[0].embedding
print(f"Dimensions: {len(embedding)}")
print(f"First 5 values: {embedding[:5]}")

Batch Embeddings

from together import Together

client = Together()

texts = [
    "Machine learning is a subset of AI.",
    "Deep learning uses neural networks.",
    "Transformers revolutionized NLP.",
]

response = client.embeddings.create(
    model="BAAI/bge-large-en-v1.5",
    input=texts,
)

for i, item in enumerate(response.data):
    print(f"Text {i}: {len(item.embedding)} dimensions")

Available Embedding Models

Image Generation

Together AI hosts FLUX models from Black Forest Labs for high-quality image generation.

Generate Images

from together import Together

client = Together()

response = client.images.generate(
    model="black-forest-labs/FLUX.1-schnell",
    prompt="A photorealistic mountain landscape at sunset with a lake reflection",
    steps=4,
    n=1,
    width=1024,
    height=1024,
)

# Response contains URL or base64 image data
print(response.data[0].url)

Available Image Models

Image with OpenAI SDK

from openai import OpenAI

client = OpenAI(
    api_key=os.environ["TOGETHER_API_KEY"],
    base_url="https://api.together.xyz/v1",
)

response = client.images.generate(
    model="black-forest-labs/FLUX.1-schnell",
    prompt="A cyberpunk cityscape at night",
    n=1,
)

print(response.data[0].url)

Batch Inference

Batch API processes large volumes of requests asynchronously at 50% lower cost with a 24-hour turnaround.

Input File Format

Create a JSONL file where each line is a request:

{"custom_id": "request-1", "body": {"model": "deepseek-ai/DeepSeek-V3", "messages": [{"role": "user", "content": "What is machine learning?"}], "max_tokens": 200}}
{"custom_id": "request-2", "body": {"model": "deepseek-ai/DeepSeek-V3", "messages": [{"role": "user", "content": "Explain gradient descent."}], "max_tokens": 200}}
{"custom_id": "request-3", "body": {"model": "deepseek-ai/DeepSeek-V3", "messages": [{"role": "user", "content": "What are transformers?"}], "max_tokens": 200}}

Submit Batch Job

from together import Together

client = Together()

# 1. Upload the batch file
batch_file = client.files.upload(
    file="batch_requests.jsonl",
    purpose="batch-api",
)
print(f"File ID: {batch_file.id}")

# 2. Create the batch job
batch = client.batches.create_batch(
    file_id=batch_file.id,
    endpoint="/v1/chat/completions",
)
print(f"Batch ID: {batch.id}")

# 3. Monitor status
status = client.batches.get_batch(batch.id)
print(f"Status: {status.status}")

# 4. Download results when completed
if status.status == "COMPLETED":
    results = client.files.retrieve_content(
        id=status.output_file_id,
    )
    print(results)

List Batches

from together import Together

client = Together()

batches = client.batches.list_batches()
for b in batches:
    print(f"{b.id}: {b.status}")

Model Selection

Chat / Instruct Models

Reasoning Models

Code Models

Pricing is approximate and subject to change. Check https://www.together.ai/pricing for current rates.

OpenAI Compatibility

Together AI is fully compatible with the OpenAI Python SDK. Change two lines to switch from OpenAI to Together AI.

Using OpenAI SDK

from openai import OpenAI
import os

# Just change the API key and base_url
client = OpenAI(
    api_key=os.environ["TOGETHER_API_KEY"],
    base_url="https://api.together.xyz/v1",
)

# Everything else is identical to OpenAI usage
response = client.chat.completions.create(
    model="meta-llama/Llama-3.3-70B-Instruct-Reference",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hello!"},
    ],
    max_tokens=256,
)

print(response.choices[0].message.content)

What Works with OpenAI SDK

LangChain Integration

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="meta-llama/Llama-3.3-70B-Instruct-Reference",
    openai_api_key=os.environ["TOGETHER_API_KEY"],
    openai_api_base="https://api.together.xyz/v1",
)

response = llm.invoke("What is the meaning of life?")
print(response.content)

CLI Reference

The together CLI provides direct access from the terminal.

# Install
pip install --upgrade together

# Set API key
export TOGETHER_API_KEY="your-api-key"

# Chat completion
together chat.completions \
    --message "system" "You are helpful." \
    --message "user" "What is PyTorch?" \
    --model meta-llama/Llama-3.3-70B-Instruct-Reference

# List models
together models list

# Image generation
together images generate \
    "A futuristic city at sunset" \
    --model black-forest-labs/FLUX.1-schnell \
    --n 1

# File operations
together files upload training_data.jsonl
together files list
together files retrieve file-abc123
together files delete file-abc123

# Fine-tuning
together fine-tuning create \
    --training-file file-abc123 \
    -m meta-llama/Meta-Llama-3.1-8B-Instruct-Reference
together fine-tuning list
together fine-tuning status ft-abc123
together fine-tuning list-events ft-abc123
together fine-tuning list-checkpoints ft-abc123
together fine-tuning download --ft-id ft-abc123
together fine-tuning cancel ft-abc123

Cost Optimization

1. Use MoE Models

Mixture-of-Experts models activate only a fraction of parameters per token, offering better price-to-performance:

2. Use Batch API

Submit non-urgent workloads via the batch API for 50% cost reduction. Ideal for evaluations, dataset generation, and bulk classification.

3. Use JSON Mode for Structured Output

Avoid post-processing errors and retry loops by constraining output format upfront.

4. Right-Size Your Model

• Simple classification/extraction: 8B models ($0.18/M)
• General chat/instruction: 70B models ($0.88/M)
• Complex reasoning/code: DeepSeek-V3 ($1.25/M) or R1 ($3.00/M)
• Cost-sensitive high volume: MoE models like Qwen3-Next ($0.15/M)

5. Minimize Max Tokens

Set max_tokens to the minimum needed. You pay for output tokens generated, not the max_tokens budget.

6. Use Streaming for Long Outputs

Streaming does not cost more but gives faster time-to-first-token and lets you abort early if the output goes off track.

Common Issues

Resources

• Documentation: https://docs.together.ai
• API Reference: https://docs.together.ai/reference
• Pricing: https://www.together.ai/pricing
• Dashboard: https://api.together.xyz
• Model Catalog: https://docs.together.ai/docs/serverless-models
• Fine-Tuning Guide: https://docs.together.ai/docs/fine-tuning-quickstart
• Python SDK: https://github.com/togethercomputer/together-python
• TypeScript SDK: https://github.com/togethercomputer/together-typescript
• Cookbook (Examples): https://github.com/togethercomputer/together-cookbook
• Status Page: https://status.together.ai