Case 05011

Test Suite

This skill includes a comprehensive test suite. Test fixtures use mock credentials that are NOT real secrets:

# All tokens in tests/ are fake placeholders:
MOCK_API_KEY = 'sk-test-1234567890abcdef1234567890abcdef'
MOCK_AWS_KEY = 'AKIAIOSFODNN7EXAMPLE'

See tests/conftest.py for the full mock configuration.

Doubleword Batch Inference

Process multiple AI inference requests asynchronously using the Doubleword batch API with high throughput and low cost.

Prerequisites

Before submitting batches, you need:

1. Doubleword Account - Sign up at https://app.doubleword.ai/
2. API Key - Create one in the API Keys section of your dashboard
3. Account Credits - Add credits to process requests (see pricing below)

When to Use Batches

Batches are ideal for:

• Multiple independent requests that can run simultaneously
• Workloads that don't require immediate responses
• Large volumes that would exceed rate limits if sent individually
• Cost-sensitive workloads (24h window = 50-60% cheaper than realtime)
• Tool calling and structured output generation at scale

Available Models & Pricing

Pricing is per 1 million tokens (input / output):

Qwen3-VL-30B-A3B-Instruct-FP8 (mid-size):

• Realtime SLA: $0.16 / $0.80
• 1-hour SLA: $0.07 / $0.30 (56% cheaper)
• 24-hour SLA: $0.05 / $0.20 (69% cheaper)

Qwen3-VL-235B-A22B-Instruct-FP8 (flagship):

• Realtime SLA: $0.60 / $1.20
• 1-hour SLA: $0.15 / $0.55 (75% cheaper)
• 24-hour SLA: $0.10 / $0.40 (83% cheaper)
• Supports up to 262K total tokens, 16K new tokens per request

Cost estimation: Upload files to the Doubleword Console to preview expenses before submitting.

Quick Start

Two ways to submit batches:

Via API:

1. Create JSONL file with requests
2. Upload file to get file ID
3. Create batch using file ID
4. Poll status until complete
5. Download results from output_file_id

Via Web Console:

1. Navigate to Batches section at https://app.doubleword.ai/
2. Upload JSONL file
3. Configure batch settings (model, completion window)
4. Monitor progress in real-time dashboard
5. Download results when ready

Workflow

Step 1: Create Batch Request File

Create a .jsonl file where each line contains a complete, valid JSON object with no line breaks within the object:

{"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "anthropic/claude-3-5-sonnet", "messages": [{"role": "user", "content": "What is 2+2?"}]}}
{"custom_id": "req-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "anthropic/claude-3-5-sonnet", "messages": [{"role": "user", "content": "What is the capital of France?"}]}}

Required fields per line:

• custom_id: Unique identifier (max 64 chars) - use descriptive IDs like "user-123-question-5" for easier result mapping
• method: Always "POST"
• url: API endpoint - "/v1/chat/completions" or "/v1/embeddings"
• body: Standard API request with model and messages

Optional body parameters:

• temperature: 0-2 (default: 1.0)
• max_tokens: Maximum response tokens
• top_p: Nucleus sampling parameter
• stop: Stop sequences
• tools: Tool definitions for tool calling (see Tool Calling section)
• response_format: JSON schema for structured outputs (see Structured Outputs section)

File requirements:

• Max size: 200MB
• Format: JSONL only (JSON Lines - newline-delimited JSON)
• Each line must be valid JSON with no internal line breaks
• No duplicate custom_id values
• Split large batches into multiple files if needed

Common pitfalls:

• Line breaks within JSON objects (will cause parsing errors)
• Invalid JSON syntax
• Duplicate custom_id values

Helper script: Use scripts/create_batch_file.py to generate JSONL files programmatically:

python scripts/create_batch_file.py output.jsonl

Modify the script's requests list to generate your specific batch requests.

Step 2: Upload File

Via API:

curl https://api.doubleword.ai/v1/files \
  -H "Authorization: Bearer $DOUBLEWORD_API_KEY" \
  -F purpose="batch" \
  -F file="@batch_requests.jsonl"

Via Console: Upload through the Batches section at https://app.doubleword.ai/

Response contains id field - save this file ID for next step.

Step 3: Create Batch

Via API:

curl https://api.doubleword.ai/v1/batches \
  -H "Authorization: Bearer $DOUBLEWORD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input_file_id": "file-abc123",
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h"
  }'

Via Console: Configure batch settings in the web interface.

Parameters:

• input_file_id: File ID from upload step
• endpoint: API endpoint ("/v1/chat/completions" or "/v1/embeddings")
• completion_window: Choose based on urgency and budget:
  • "24h": Best pricing, results within 24 hours (typically faster)
  • "1h": 50% price premium, results within 1 hour (typically faster)
  • Realtime: Limited capacity, highest cost (batch service optimized for async)

Response contains batch id - save this for status polling.

Before submitting, verify:

• You have access to the specified model
• Your API key is active
• You have sufficient account credits

Step 4: Poll Status

Via API:

curl https://api.doubleword.ai/v1/batches/batch-xyz789 \
  -H "Authorization: Bearer $DOUBLEWORD_API_KEY"

Via Console: Monitor real-time progress in the Batches dashboard.

Status progression:

1. validating - Checking input file format
2. in_progress - Processing requests
3. completed - All requests finished

Other statuses:

• failed - Batch failed (check error_file_id)
• expired - Batch timed out
• cancelling/cancelled - Batch cancelled

Response includes:

• output_file_id - Download results here
• error_file_id - Failed requests (if any)
• request_counts - Total/completed/failed counts

Polling frequency: Check every 30-60 seconds during processing.

Early access: Results available via output_file_id before batch fully completes - check X-Incomplete header.

Step 5: Download Results

Via API:

curl https://api.doubleword.ai/v1/files/file-output123/content \
  -H "Authorization: Bearer $DOUBLEWORD_API_KEY" \
  > results.jsonl

Via Console: Download results directly from the Batches dashboard.

Response headers:

• X-Incomplete: true - Batch still processing, more results coming
• X-Last-Line: 45 - Resume point for partial downloads

Output format (each line):

{
  "id": "batch-req-abc",
  "custom_id": "request-1",
  "response": {
    "status_code": 200,
    "body": {
      "id": "chatcmpl-xyz",
      "choices": [{
        "message": {
          "role": "assistant",
          "content": "The answer is 4."
        }
      }]
    }
  }
}

Download errors (if any):

curl https://api.doubleword.ai/v1/files/file-error123/content \
  -H "Authorization: Bearer $DOUBLEWORD_API_KEY" \
  > errors.jsonl

Error format (each line):

{
  "id": "batch-req-def",
  "custom_id": "request-2",
  "error": {
    "code": "invalid_request",
    "message": "Missing required parameter"
  }
}

Tool Calling in Batches

Tool calling (function calling) enables models to intelligently select and use external tools. Doubleword maintains full OpenAI compatibility.

Example batch request with tools:

{
  "custom_id": "tool-req-1",
  "method": "POST",
  "url": "/v1/chat/completions",
  "body": {
    "model": "anthropic/claude-3-5-sonnet",
    "messages": [{"role": "user", "content": "What's the weather in Paris?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get current weather for a location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {"type": "string"}
          },
          "required": ["location"]
        }
      }
    }]
  }
}

Use cases:

• Agents that interact with APIs at scale
• Fetching real-time information for multiple queries
• Executing actions through standardized tool definitions

Structured Outputs in Batches

Structured outputs guarantee that model responses conform to your JSON Schema, eliminating issues with missing fields or invalid enum values.

Example batch request with structured output:

{
  "custom_id": "structured-req-1",
  "method": "POST",
  "url": "/v1/chat/completions",
  "body": {
    "model": "anthropic/claude-3-5-sonnet",
    "messages": [{"role": "user", "content": "Extract key info from: John Doe, 30 years old, lives in NYC"}],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "person_info",
        "schema": {
          "type": "object",
          "properties": {
            "name": {"type": "string"},
            "age": {"type": "integer"},
            "city": {"type": "string"}
          },
          "required": ["name", "age", "city"]
        }
      }
    }
  }
}

Benefits:

• Guaranteed schema compliance
• No missing required keys
• No hallucinated enum values
• Seamless OpenAI compatibility

autobatcher: Automatic Batching

autobatcher is a Python client that automatically converts individual API calls into batched requests, reducing costs without code changes.

Installation:

pip install autobatcher

How it works:

1. Collection Phase: Requests accumulate during a time window (default: 1 second) or until batch size threshold
2. Batch Submission: Collected requests are submitted together
3. Result Polling: System monitors for completed responses
4. Transparent Response: Your code receives standard ChatCompletion responses

Key benefit: Significant cost reduction through automatic batching while writing normal async code using the familiar OpenAI interface.

Documentation: https://github.com/doublewordai/autobatcher

Additional Operations

List All Batches

Via API:

curl https://api.doubleword.ai/v1/batches?limit=10 \
  -H "Authorization: Bearer $DOUBLEWORD_API_KEY"

Via Console: View all batches in the dashboard.

Cancel Batch

Via API:

curl https://api.doubleword.ai/v1/batches/batch-xyz789/cancel \
  -X POST \
  -H "Authorization: Bearer $DOUBLEWORD_API_KEY"

Via Console: Click cancel in the batch details view.

Notes:

• Unprocessed requests are cancelled
• Already-processed results remain downloadable
• Only charged for completed work
• Cannot cancel completed batches

Common Patterns

Processing Results

Parse JSONL output line-by-line:

import json

with open('results.jsonl') as f:
    for line in f:
        result = json.loads(line)
        custom_id = result['custom_id']
        content = result['response']['body']['choices'][0]['message']['content']
        print(f"{custom_id}: {content}")

Handling Partial Results

Check for incomplete batches and resume:

import requests

response = requests.get(
    'https://api.doubleword.ai/v1/files/file-output123/content',
    headers={'Authorization': f'Bearer {api_key}'}
)

if response.headers.get('X-Incomplete') == 'true':
    last_line = int(response.headers.get('X-Last-Line', 0))
    print(f"Batch incomplete. Processed {last_line} requests so far.")
    # Continue polling and download again later

Retry Failed Requests

Extract failed requests from error file and resubmit:

import json

failed_ids = []
with open('errors.jsonl') as f:
    for line in f:
        error = json.loads(line)
        failed_ids.append(error['custom_id'])

print(f"Failed requests: {failed_ids}")
# Create new batch with only failed requests

Processing Tool Calls

Handle tool call responses:

import json

with open('results.jsonl') as f:
    for line in f:
        result = json.loads(line)
        message = result['response']['body']['choices'][0]['message']

        if message.get('tool_calls'):
            for tool_call in message['tool_calls']:
                print(f"Tool: {tool_call['function']['name']}")
                print(f"Args: {tool_call['function']['arguments']}")

Best Practices

1. Descriptive custom_ids: Include context in IDs for easier result mapping

   • Good: "user-123-question-5", "dataset-A-row-42"
   • Bad: "1", "req1"

2. Validate JSONL locally: Ensure each line is valid JSON with no internal line breaks before upload

3. No duplicate IDs: Each custom_id must be unique within the batch

4. Split large files: Keep under 200MB limit by splitting into multiple batches

5. Choose appropriate window: Use 24h for cost savings (50-83% cheaper), 1h only when time-sensitive

6. Handle errors gracefully: Always check error_file_id and retry failed requests

7. Monitor request_counts: Track progress via completed/total ratio

8. Save file IDs: Store batch_id, input_file_id, output_file_id for later retrieval

9. Use cost estimator: Preview expenses in console before submitting large batches

10. Consider autobatcher: For ongoing workloads, use autobatcher to automatically batch individual API calls

Reference Documentation

For complete API details, see:

• API Reference: references/api_reference.md - Full endpoint documentation and schemas
• Getting Started Guide: references/getting_started.md - Detailed setup and account management
• Pricing Details: references/pricing.md - Model costs and SLA comparison