Rnow Train Jsonl

train.jsonl Format

One JSON object per line. Each entry is a training example.

Fields

Message Roles

Basic Examples

RL Entry

{"messages": [{"role": "user", "content": "What is 2+2?"}], "rewards": ["accuracy"], "metadata": {"answer": "4"}}

SFT Entry

{"messages": [{"role": "user", "content": "Hello"}, {"role": "assistant", "content": "Hi there!"}]}

SFT with Tool Calls (Agentic Distillation)

SFT supports training on conversations with tool calls (e.g., from teacher model distillation):

{
  "messages": [
    {"role": "user", "content": "Find the weather in Paris"},
    {"role": "assistant", "content": "", "tool_calls": [{"id": "call_1", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"}}]},
    {"role": "tool", "tool_call_id": "call_1", "content": "72°F, sunny"},
    {"role": "assistant", "content": "The weather in Paris is 72°F and sunny."}
  ]
}

Tool call format (OpenAI-compatible):

{
  "id": "call_xxx",
  "type": "function",
  "function": {
    "name": "tool_name",
    "arguments": "{\"arg\": \"value\"}"
  }
}

Notes:

• arguments must be a JSON string, not an object
• content can be empty string "" when assistant makes tool calls
• Tool results use role: "tool" with matching tool_call_id
• Works with all model renderers (Qwen3, DeepSeek, Kimi, etc.)

With System Prompt

{"messages": [{"role": "system", "content": "You are a math tutor"}, {"role": "user", "content": "Explain fractions"}], "rewards": ["quality"]}

Using Tools

Filter which tools are available for a specific entry with the tools field:

{"messages": [{"role": "user", "content": "Search for AI news"}], "rewards": ["relevance"], "tools": ["web_search"]}

If tools is omitted, ALL defined tools in tools.py are available.

For writing tool functions, see the rnow-tools skill.

Sandbox Entries

For entries that need isolated execution (code execution, file operations), use the docker field. This spawns a Modal sandbox where state persists between tool calls within the same rollout.

Required when: Any reward or tool uses sandbox=True.

Basic Sandbox

{
  "messages": [{"role": "user", "content": "Write and run a Python script"}],
  "rewards": ["code_runs", "output_correct"],
  "tools": ["execute_python"],
  "docker": "python:3.11-slim"
}

Custom Docker Image

{
  "messages": [{"role": "user", "content": "Analyze the data"}],
  "rewards": ["accuracy"],
  "docker": "myorg/custom-image:latest",
  "docker_env": {"DEBUG": "true", "DATA_PATH": "/data"}
}

Building Custom Images

CRITICAL: Docker images must be built for linux/amd64:

# Correct - Modal compatible
docker build --platform linux/amd64 -t myorg/image:latest .
docker push myorg/image:latest

# Wrong - will fail on x86_64 servers
docker build -t myorg/image:latest .

Modal runs on x86_64 Linux servers. Images built on ARM Macs without --platform linux/amd64 will fail.

Multi-Turn Context

Provide conversation history for multi-turn training:

{
  "messages": [
    {"role": "user", "content": "What's the capital of France?"},
    {"role": "assistant", "content": "Paris"},
    {"role": "user", "content": "What's its population?"}
  ],
  "rewards": ["accuracy"],
  "metadata": {"answer": "2.1 million"}
}

VLM Images (Vision-Language Models)

For vision-language model training, images must be included as base64 inline data URIs. File references are not supported.

Image Format

Images use multimodal content with type: "image":

{
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "image", "image": "data:image/png;base64,iVBORw0KGgo..."},
        {"type": "text", "text": "What's in this image?"}
      ]
    }
  ],
  "rewards": ["accuracy"]
}

Converting Images to Base64

When preparing datasets, convert PIL images to base64:

import base64
import io

def image_to_base64(img) -> str:
    """Convert PIL Image to base64 data URI."""
    buffer = io.BytesIO()
    img.save(buffer, format="PNG")
    b64 = base64.b64encode(buffer.getvalue()).decode("utf-8")
    return f"data:image/png;base64,{b64}"

# Usage with HuggingFace datasets
from datasets import load_dataset
ds = load_dataset("dataset_name", split="train")

for row in ds:
    img = row["image"]  # PIL Image
    image_b64 = image_to_base64(img)

    entry = {
        "messages": [
            {
                "role": "user",
                "content": [
                    {"type": "image", "image": image_b64},
                    {"type": "text", "text": "Describe this image"}
                ]
            }
        ],
        "rewards": ["quality"]
    }

Supported Formats

• data:image/png;base64,...
• data:image/jpeg;base64,...
• data:image/webp;base64,...
• data:image/gif;base64,...

Validation Rules

1. Rewards must exist - Names in rewards must match @reward functions in rewards.py
2. Tools must exist - Names in tools must match @tool functions in tools.py
3. sandbox=True requires docker - If any reward/tool uses sandbox=True, the entry needs a docker field
4. Messages format - Must have at least one user message; system must be first if present

Related Skills

• rnow-tools - Writing tool functions (@tool decorator)
• rnow-rewards - Writing reward functions (@reward decorator)
• rnow-config - config.yml settings and HuggingFace dataset conversion

Source: ReinforceNow/reinforcenow-cli — distributed by TomeVault.