imbue-ai You can create and access Parquet files produced by evals/eda/eval_to_parquet_cli.py.
These files contain flattened results from Keystone eval runs.
How the Parquet is produced
# S3 → local
uv run python evals/eda/eval_to_parquet_cli.py \
s3://int8-datasets/keystone/evals/<run_name> \
./results.parquet
# Local → local
uv run python evals/eda/eval_to_parquet_cli.py \
/path/to/eval_output /tmp/results.parquet
Schema
Each row is one trial (one agent run on one repo). Columns:
| Column | Type | Description |
|---|---|---|
source_path |
str | fsspec path to the original eval_result.json this row was read from |
raw_json |
str | Full KeystoneRepoResult serialized as JSON — deserialize with KeystoneRepoResult.model_validate_json(row.raw_json) to access any field |
config_name |
str | null | EvalConfig.name — the primary identifier for which eval configuration was used (e.g. model name or experiment label) |
repo_id |
str | Unique repo identifier (e.g. "requests", "flask") |
trial_index |
int | null | Trial number (0-indexed) when trials_per_repo > 1 |
success |
bool | Whether the keystone run succeeded |
error_message |
str | null | Error message if success is False |
agent_exit_code |
int | null | Agent process exit code |
agent_walltime_seconds |
float | null | Wall-clock time for the agent |
agent_timed_out |
bool | null | Whether the agent hit its time limit |
cost_usd |
float | null | Inference cost in USD |
input_tokens |
int | null | LLM input tokens consumed |
output_tokens |
int | null | LLM output tokens consumed |
image_build_seconds |
float | null | Docker image build time |
test_execution_seconds |
float | null | Test suite execution time |
tests_passed |
int | null | Number of unique tests passed (deduplicated by test name; logical OR across duplicate runs) |
tests_failed |
int | null | Number of unique tests failed (deduplicated; tests where no execution passed) |
tests_discovered |
int | null | Total unique test names discovered (= tests_passed + tests_failed) |
summary |
str | null | agent.summary.message — the agent's final summary of what it did |
status_messages |
str (JSON) | JSON array of {"timestamp": "...", "message": "..."} objects — the agent's step-by-step progress |
Primary key
Rows are uniquely identified by (config_name, repo_id, trial_index).
Example queries
When the user asks you to analyze eval results, load the parquet with pandas and answer their questions. Here are common patterns:
import pandas as pd
df = pd.read_parquet("results.parquet")
Success rate by config
df.groupby("config_name")["success"].mean().sort_values(ascending=False)
Cost summary by config
df.groupby("config_name").agg(
mean_cost=("cost_usd", "mean"),
median_cost=("cost_usd", "median"),
total_cost=("cost_usd", "sum"),
)
Slowest repos
df.nlargest(10, "agent_walltime_seconds")[["config_name", "repo_id", "agent_walltime_seconds", "success"]]
Failures with error messages
df[~df["success"]][["config_name", "repo_id", "error_message", "summary"]]
Compare two configs head-to-head on the same repos
a, b = "claude-opus", "codex"
merged = df[df.config_name == a].merge(
df[df.config_name == b],
on=["repo_id", "trial_index"],
suffixes=("_a", "_b"),
)
merged["both_pass"] = merged.success_a & merged.success_b
merged["a_only"] = merged.success_a & ~merged.success_b
merged["b_only"] = ~merged.success_a & merged.success_b
merged[["both_pass", "a_only", "b_only"]].sum()
Parse status_messages for a specific run
import json
row = df[(df.config_name == "claude-opus") & (df.repo_id == "requests")].iloc[0]
messages = json.loads(row.status_messages)
for m in messages:
print(f"[{m['timestamp']}] {m['message']}")
Reconstruct full KeystoneRepoResult from raw_json
from eval_schema import KeystoneRepoResult
result = KeystoneRepoResult.model_validate_json(row.raw_json)
# Now access any nested field, e.g.:
result.bootstrap_result.verification.test_results
Test pass rate distribution
df["test_pass_rate"] = df.tests_passed / (df.tests_passed + df.tests_failed)
df.groupby("config_name")["test_pass_rate"].describe()
Tips
raw_jsonpreserves everything — if a column doesn't exist for something you need, deserializeraw_jsonback intoKeystoneRepoResult.status_messagesis a JSON string, not a list — usejson.loads()to parse it.source_pathlets you go back to the original file on S3 or local disk for debugging.- Null values in agent/verification columns mean the run failed before reaching that stage.