Groq Inference

Groq — Ultra-Fast LLM Inference

Groq runs open-weight LLMs on custom LPU (Language Processing Unit) hardware, delivering the lowest inference latency in the industry — up to 1000+ tokens/sec. OpenAI-compatible API, free tier, pay-per-token pricing. Inference only, no training.

When to Use Groq

Use when:

• You need the fastest possible inference latency (real-time chat, agents, interactive apps)
• You want open-weight models (Llama, Qwen, DeepSeek) without managing GPUs
• You need an OpenAI-compatible drop-in replacement with open models
• You want free-tier access for prototyping
• You need fast Whisper transcription or vision with Llama 4

Don't use when:

• You need to fine-tune or train models (inference-only)
• You need proprietary models (GPT-4o, Claude, Gemini)
• You need embeddings or image generation APIs (not offered)

Credential Setup

Credentials are auto-injected by openscience when connected via the dashboard.

# Verify credentials
[ -n "$GROQ_API_KEY" ] && echo "GROQ_API_KEY set" || echo "NOT SET"

If not set: add your Groq key in Customize → Models or export GROQ_API_KEY locally.

Quick Start

pip install groq
export GROQ_API_KEY="gsk_..."   # https://console.groq.com/keys

from groq import Groq

client = Groq()
response = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain quantum computing in one paragraph."}
    ],
    temperature=0.7,
    max_completion_tokens=512,
)
print(response.choices[0].message.content)

Chat Completions

Streaming

from groq import Groq

client = Groq()
stream = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "Explain fast inference"}],
    stream=True,
)
for chunk in stream:
    content = chunk.choices[0].delta.content
    if content:
        print(content, end="")

JSON Mode

from groq import Groq
import json

client = Groq()
response = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[
        {"role": "system", "content": "Output valid JSON only."},
        {"role": "user", "content": "List 3 programming languages with their paradigms."}
    ],
    response_format={"type": "json_object"},
    temperature=0,
)
data = json.loads(response.choices[0].message.content)

Tool / Function Calling

from groq import Groq
import json

client = Groq()
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get weather for a location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string", "description": "City name"},
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
            },
            "required": ["location"],
            "additionalProperties": False
        }
    }
}]

response = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "Weather in San Francisco?"}],
    tools=tools,
    tool_choice="auto",
)

message = response.choices[0].message
if message.tool_calls:
    call = message.tool_calls[0]
    args = json.loads(call.function.arguments)
    # Execute function, then send result back as role="tool" message
    result = {"temperature": 62, "condition": "foggy"}
    followup = client.chat.completions.create(
        model="llama-3.3-70b-versatile",
        messages=[
            {"role": "user", "content": "Weather in San Francisco?"},
            message,
            {"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)}
        ],
    )
    print(followup.choices[0].message.content)

Vision

Llama 4 models support image input via URL or base64. Max image size: 20 MB.

from groq import Groq

client = Groq()
response = client.chat.completions.create(
    model="meta-llama/llama-4-scout-17b-16e-instruct",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this image."},
            {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}}
        ]
    }],
    max_completion_tokens=1024,
)
print(response.choices[0].message.content)

Vision models: meta-llama/llama-4-scout-17b-16e-instruct, meta-llama/llama-4-maverick-17b-128e-instruct.

Audio / Speech-to-Text

Whisper on LPU for near-instant transcription. Formats: flac, mp3, mp4, mpeg, mpga, m4a, ogg, wav, webm. Max: 25 MB free / 100 MB developer tier.

from groq import Groq

client = Groq()

# Transcription
with open("audio.mp3", "rb") as f:
    result = client.audio.transcriptions.create(
        file=("audio.mp3", f),
        model="whisper-large-v3-turbo",
        language="en",
        response_format="verbose_json",
    )
print(result.text)

# Translation (any language -> English)
with open("french.mp3", "rb") as f:
    result = client.audio.translations.create(
        file=("french.mp3", f), model="whisper-large-v3",
    )
print(result.text)

Text-to-Speech

from groq import Groq
from pathlib import Path

client = Groq()
speech = client.audio.speech.create(
    model="playai-tts",
    voice="Arista-PlayAI",
    input="Hello from Groq!",
)
Path("output.mp3").write_bytes(speech.content)

Model Selection

Production Models

Preview Models

Systems & TTS

Selection Guide

• Fastest + cheapest: llama-3.1-8b-instant — classification, extraction, simple tasks
• Best quality: llama-3.3-70b-versatile — complex reasoning, coding, chat
• Best value: openai/gpt-oss-120b — strong quality, 500 t/s, $0.15 input
• Vision: meta-llama/llama-4-scout-17b-16e-instruct — 750 t/s, cheapest multimodal
• Long context: moonshotai/kimi-k2-instruct-0905 — 262K context window
• Safety: meta-llama/llama-guard-4-12b or openai/gpt-oss-safeguard-20b

OpenAI Compatibility

Drop-in replacement — change base_url and api_key:

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.groq.com/openai/v1",
    api_key=os.environ.get("GROQ_API_KEY"),
)
response = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "Hello from OpenAI SDK!"}],
)
print(response.choices[0].message.content)

Unsupported fields (400 error): logprobs, logit_bias, top_logprobs, messages[].name, n (must be 1).

Rate Limits

Rate limit headers returned with every response: x-ratelimit-limit-requests, x-ratelimit-remaining-requests, x-ratelimit-limit-tokens, x-ratelimit-remaining-tokens.

import time
from groq import Groq, RateLimitError

client = Groq()

def call_with_retry(messages, model="llama-3.3-70b-versatile", max_retries=3):
    for attempt in range(max_retries):
        try:
            return client.chat.completions.create(model=model, messages=messages)
        except RateLimitError:
            if attempt == max_retries - 1:
                raise
            time.sleep(2 ** attempt)

Cost Optimization

1. Use the smallest model that works. llama-3.1-8b-instant is 10x cheaper than the 70B for simple tasks.
2. Cap output with max_completion_tokens. Output tokens cost more than input.
3. Try openai/gpt-oss-20b — best price/performance at $0.075 input and 1000 t/s.
4. Cache at temperature=0. Deterministic output means identical prompts yield identical results.
5. Use Whisper Turbo. whisper-large-v3-turbo is 64% cheaper than v3 with minimal quality loss.
6. Preprocess audio to 16KHz mono WAV to reduce upload size.

Common Issues

Resources

• API Docs: https://console.groq.com/docs/overview
• Models: https://console.groq.com/docs/models
• API Keys: https://console.groq.com/keys
• Pricing: https://groq.com/pricing
• Rate Limits: https://console.groq.com/docs/rate-limits
• OpenAI Compat: https://console.groq.com/docs/openai
• Tool Use: https://console.groq.com/docs/tool-use/local-tool-calling
• Vision: https://console.groq.com/docs/vision
• Speech-to-Text: https://console.groq.com/docs/speech-to-text
• Structured Outputs: https://console.groq.com/docs/structured-outputs
• Python SDK: https://github.com/groq/groq-python (pip install groq)
• Status: https://status.groq.com
• LPU Architecture: https://groq.com/blog/inside-the-lpu-deconstructing-groq-speed