Together Fireworks

together-fireworks

You run open-weight LLMs on two neutral, OpenAI-compatible hosts: Together AI and Fireworks AI. Neither trains the flagship models — they host open weights (Llama, DeepSeek, Qwen, GPT-OSS, Kimi, Mistral) behind a billed-per-token endpoint. No GPU to provision, no server to babysit. You pay for tokens.

Both speak the OpenAI wire protocol. So the entire mental model is: same SDK, change three things — base_url, api_key, and the model id. That is why these two providers live in one skill: the working knowledge (connect, pick a model, batch for 50% off, do the cost math) is ~90% shared. The only real differences are the base URL string and the model-id naming scheme. Learn both at once.

This skill is where the tokens come from. Designing the prompt is ../prompt-engineering/SKILL.md; chaining calls into a workflow is ../llm-pipeline/SKILL.md; tracking spend across many providers as a discipline is ../cost-tracking/SKILL.md. Here you only do the per-model math and the endpoint plumbing.

The two endpoints (memorize these)

The example ids show the shape of a valid id, not a guaranteed-live id — the catalog churns and casing matters. Before you ship any id, confirm the exact string on the provider's own catalog (docs.together.ai/docs/serverless-models, docs.fireworks.ai/serverless/pricing). A plausible-looking id that is not in the catalog 404s exactly like a typo, so an unconfirmed id is a bug, not a default.

The #1 failure is a bare model name. model="llama-3.3-70b" returns 404 model not found on both — the id MUST carry its namespace prefix.

Connect in 30 seconds

Use the official openai SDK. Do not install a provider-specific client unless you need a provider-only feature (Together's native Batch API, below).

import os
from openai import OpenAI

# Together
together = OpenAI(
    base_url="https://api.together.ai/v1",
    api_key=os.environ["TOGETHER_API_KEY"],   # never a string literal
)
r = together.chat.completions.create(
    model="openai/gpt-oss-20b",               # namespaced — vendor/model
    messages=[{"role": "user", "content": "Classify: spam or ham?"}],
)

# Fireworks — same SDK, three things change
fireworks = OpenAI(
    base_url="https://api.fireworks.ai/inference/v1",
    api_key=os.environ["FIREWORKS_API_KEY"],
)
r = fireworks.chat.completions.create(
    model="accounts/fireworks/models/gpt-oss-120b",  # accounts/fireworks/models/<name>
    messages=[{"role": "user", "content": "Explain this stack trace."}],
)

import OpenAI from "openai";

const fireworks = new OpenAI({
  baseURL: "https://api.fireworks.ai/inference/v1",
  apiKey: process.env.FIREWORKS_API_KEY,
});

const r = await fireworks.chat.completions.create({
  model: "accounts/fireworks/models/llama-v3p1-8b-instruct", // namespaced
  messages: [{ role: "user", content: "Summarize in one line." }],
});

• Bad → Good (model id): model="gpt-oss-120b" → model="openai/gpt-oss-120b" (Together) or model="accounts/fireworks/models/gpt-oss-120b" (Fireworks). Why: the compat layer routes by the full namespaced id; a bare name has no route → 404.
• Bad → Good (base_url): base_url="https://api.openai.com/v1" with a Together key → auth error / wrong models. Why: a Together/Fireworks key only authenticates against its own host.
• Bad → Good (key): api_key="sk-..." literal → api_key=os.environ["TOGETHER_API_KEY"]. Why: a committed key is a leaked key.

client.embeddings.create() works identically on both for embedding models — same surface, just swap the model id.

Pick the model

Match the task to the smallest model that clears the quality bar. Every $/1M figure below is per 1M tokens, input/output, USD, read directly off each provider's own pricing page on 2026-06-02 (Together: together.ai/pricing; Fireworks: docs.fireworks.ai/serverless/pricing) — never a tracker. Confirm the exact id string and rate at that page before you ship; casing and suffixes (-Turbo, -Lite) are load-bearing, and a plausible id absent from the catalog 404s like a typo.

Rows tagged (projected) are current-generation flagships whose id and price move fast and that you may not recognize from older training data — the number is the page's figure on 2026-06-02, but re-read the page before you quote one. Untagged rows are long-lived ids with stable pricing; the worked examples and defaults below lean on these on purpose.

Rules:

• Output tokens cost 3–7× input. A reasoning model that thinks for 2k tokens before answering is the expensive part — budget on output, not input. Qwen3.6-Plus is the trap: $0.50 in but $3.00 out.
• Fireworks size-tiered fallback (published on docs.fireworks.ai/serverless/pricing, applies to any model with no named price): <4B $0.10, 4B–16B $0.20, >16B (dense) $0.90, MoE ≤56B $0.50, MoE 56.1B–176B $1.20 per 1M. Use the matching tier to estimate a model not in the named list — these are the provider's own published numbers, so they hold up in a quote (still confirm the specific model's tier).
• Default to the cheap small model and only escalate when an eval shows it fails. Most "we need DeepSeek V4-Pro" is a gpt-oss-20b job in disguise.

Fuller catalog and embedding/fine-tuning numbers: references/models-and-pricing.md.

Serverless vs Batch vs Dedicated

Decision: real-time → serverless. Big offline job (eval, synthetic data, bulk classify) → batch. Sustained heavy traffic with an SLA → dedicated.

Gotcha — Together batch is NOT the OpenAI Batch endpoint. Together's OpenAI-compat layer does not expose /v1/batches. Use Together's native Batch API: upload a JSONL file, default 24h window, up to 50,000 requests/file, up to 50% off, separate rate-limit pool. Pointing the OpenAI Batch client at Together fails. Fireworks instead exposes Batch as a serving path through the one API (Serverless 2.0: Standard / Priority / Batch), batch = 50% of serverless. JSONL shape, the upload/poll/download flow, limits, and dedicated-deployment notes: references/batch-and-tuning.md.

Two more Fireworks multipliers worth knowing:

• Cached input tokens default to 50% of input price (text/vision models) — repeated prefixes get cheaper automatically.
• Priority serving ≈ 1.5× Standard. Priority is opt-in, not the default; do not budget at Priority rates unless you set it.

Cost & latency math

Per request:

cost = (in_tokens * in_price_per_1M + out_tokens * out_price_per_1M) / 1_000_000

Apply the multipliers: cached input → in_price * 0.5 on the cached portion; Priority → * 1.5; Batch → total * 0.5.

Worked example — 1,000,000 input + 500,000 output tokens, one shot. Both anchors are stable ids (gpt-oss, Llama 3.3 70B), so the arithmetic stays checkable even after the flagship rows churn:

• GPT-OSS 20B on Together, serverless: (1M*0.05 + 0.5M*0.20)/1e6 = $0.05 + $0.10 = $0.15.
• GPT-OSS 120B on Together, serverless: (1M*0.15 + 0.5M*0.60)/1e6 = $0.15 + $0.30 = $0.45 — 3× the 20B for the bigger open GPT-OSS.
• Llama 3.3 70B on Together, serverless: (1M*1.04 + 0.5M*1.04)/1e6 = $1.04 + $0.52 = $1.56 — ~10× the 20B for general chat.
• Same Llama 3.3 70B job as batch: $1.56 * 0.5 = $0.78.
• Escalating to a (projected) reasoning flagship (e.g. DeepSeek V4-Pro at $2.10/$4.40 on the 2026-06-02 page) lands near $4.30 serverless / $2.15 batch — ~29× the 20B — but re-read the page before you commit to that number.

So for a large offline run the lever is model choice first (up to ~29×), batch second (2×). Pick the smallest model that passes the eval AND batch it. Realtime is only worth its premium when a human is waiting.

Write provider-agnostic code

Drive base_url and model from env so you can switch providers (or arbitrage price) without touching code. Why: these are commodity endpoints — portability is leverage.

import os
from openai import OpenAI

client = OpenAI(
    base_url=os.environ["LLM_BASE_URL"],   # together or fireworks URL
    api_key=os.environ["LLM_API_KEY"],
)
model = os.environ["LLM_MODEL"]            # the namespaced id for that provider

Equivalent ids for the same underlying model:

Keep the id mapping in config, not in if provider == ... branches scattered through the code.

Anti-patterns

Validate any inference snippet/config with scripts/verify.sh <file-or-dir> — a static, no-network lint for the right base URLs, namespaced model ids, and env-var keys.