Hf Dataset Publishing

Publishing Analysis Results to Hugging Face

Turn the results of any analysis/computation (from any repo in this ecosystem) into a queryable, auto-rendered Hugging Face dataset. HF's datasets-server auto-indexes parquet tables and exposes them both in the in-browser viewer and via a REST API — so a published dataset becomes a live, linkable, machine-queryable projection of your results.

This skill encodes a workflow that was executed end-to-end on 2026-07-11. Follow THIS, not generic HF docs.

When to use

• You have analysis output (JSON, dicts, dataframes, a results folder) and want to save it somewhere queryable and viz-renderable, not just a file on disk.
• You want a shareable, indexable surface for a set of computed results (per field / per well / per country / per run-summary, etc.).
• Someone says "publish this to Hugging Face" / "save these results as an HF dataset".

Do NOT use this to record algorithm-run identity/replay ledgers — that is the separate aceengineer/<repo>-runs contract (workspace-hub#3433). See Conventions below.

Prerequisites (environment-backed auth)

• CLI is hf (huggingface-cli is DEPRECATED — do not use it).
• Token lives at ~/.cache/huggingface/token. Verify:

  hf auth whoami          # expect: user=<u> and orgs include aceengineer
  

• Python libs available: huggingface_hub (1.21+), pandas, pyarrow.
• Never print, echo, log, or commit the token. Secrets stay in env / token-file only.

The 4-step workflow

Step 1 — Reshape nested results → flat tabular tables (one per entity)

Load the analysis output and build one pandas.DataFrame per entity type (e.g. fields, wells, countries). For each table select SCALAR columns only:

• Flatten a few useful nested scalars (e.g. econ.breakeven -> econ_breakeven).
• SKIP list/dict-valued columns (they don't render in the tabular viewer).
• Sanitize NaN/inf. The source JSON may carry literal NaN/Infinity tokens that break strict JSON consumers. Parquet stores genuine nulls natively, so coerce non-finite floats to null and write parquet — which is exactly what the HF datasets-server auto-renders.

import math, pandas as pd, numpy as np

def clean(df: pd.DataFrame) -> pd.DataFrame:
    # non-finite floats -> genuine null (parquet-native)
    return df.replace([np.inf, -np.inf], np.nan)

df_fields = clean(pd.DataFrame(field_rows))         # one row per field
df_fields.to_parquet("out/fields.parquet", index=False)

CSV is an acceptable fallback if parquet is unavailable, but parquet is preferred (native nulls

• auto-render).

Step 2 — Dataset card README.md with YAML frontmatter

The frontmatter MUST include license: (see routing rule), pretty_name, tags, and a configs: block listing each parquet table as its own viewer config:

---
license: cc-by-4.0
pretty_name: World Energy Data — Explorer Projection
tags:
  - energy
  - oil-and-gas
  - analysis-results
configs:
  - config_name: fields
    data_files: fields.parquet
  - config_name: wells
    data_files: wells.parquet
  - config_name: countries
    data_files: countries.parquet
---

The card body must cover:

• What it is — one-paragraph description of the projection and its source analysis.
• A table of configs + row counts (e.g. fields — 56 rows).
• Provenance — source file sha256 hashes + a schema_version. This is how a consumer proves which analysis run produced the data.
• Data basis — what the numbers mean, units, and the note that nulls are genuine (not missing-by-accident) because non-finite values were coerced to null.
• Any withheld columns (see the data-quality gate) with a link to the filed issue.

Step 3 — Publish

from huggingface_hub import HfApi
api = HfApi()
repo_id = "aceengineer/<repo>-<projection>"          # e.g. aceengineer/worldenergydata-explorer
api.create_repo(repo_id=repo_id, repo_type="dataset",
                private=<per routing>, exist_ok=True)
api.upload_folder(folder_path="out", repo_id=repo_id, repo_type="dataset",
                  commit_message="Publish <projection> analysis projection")

The bundled publish_analysis_to_hf.py does exactly this (plus auth check + verify) — prefer it over hand-rolling.

Step 4 — Verify (the render-via-API surface)

api.list_repo_files(repo_id=repo_id, repo_type="dataset")   # confirm parquet + README landed

Then hit the datasets-server API (the surface that powers the in-browser viewer):

https://datasets-server.huggingface.co/is-valid?dataset=<repo_id>
https://datasets-server.huggingface.co/rows?dataset=<repo_id>&config=<table>&split=train&offset=0&length=10

Indexing lag is normal. A brand-new dataset returns "server is busier than usual / retry later" for a few minutes before /rows and the in-browser viewer come up. Poll — do not assume failure. Once /is-valid reports valid and /rows returns data, the viewer is live.

Conventions the skill MUST encode

• Naming: aceengineer/<repo>-<projection> — e.g. worldenergydata-explorer. Datasets on the aceengineer org are free.
• Keep SEPARATE from the run-dataset contract. aceengineer/<repo>-runs is workspace-hub#3433's contract-managed algorithm-run ledger (run-identity + replay). Analysis-result projections are a different artifact and get their own dataset. Do not dump projections into -runs.
• License / public-vs-private routing (cite .claude/rules/codes-standards-data-routing.md):
  • Public-domain federal data (BSEE / NOAA / USGS) → public, license: cc-by-4.0.
  • Vendor-licensed / private / client data → must NOT be published to a public HF dataset. A private repo is OK only with explicit owner sign-off.
  • Synthetic / own-analysis output → publisher's choice. Calculations derived from codes and standards are own analysis, not republished standard text — publishable.
  • If unsure of the source's provenance, default to private and ask the owner.
  • Visibility is enforced on every publish, not just at creation (workspace-hub#3483). Re-publishing with --public now flips an existing private dataset, and the tool reads the visibility back off the remote and exits non-zero if it disagrees with what you asked for. The manual update_repo_settings(...) workaround is no longer needed.

DATA-QUALITY GATE (load-bearing — do not skip)

Faithful to source != correct.

Row counts matching the source proves nothing about whether the numbers are right. Before publishing, run a domain plausibility check on the actual values — publishing to a new, labeled, queryable surface is a natural audit trigger, so audit here.

• If values are implausible, withhold those columns, file a cat:data / bug issue, and note the withholding in the card (with the issue link).
• Real case (2026-07-11): economics columns showed $380 breakevens on producing fields — clearly wrong. Those columns were withheld pending worldenergydata#971 rather than published as fact.
• Honesty principle: missing/withheld data → visible provenance/placeholder, never silent omission.

Sandbox gotchas (each cost real cycles)

• python3 -c "..." is DENIED → use a heredoc: python3 - <<'EOF' ... EOF.
• base64 -d is DENIED → fetch repo file content raw instead of decoding the contents API:

  gh api -H "Accept: application/vnd.github.raw" \
    "repos/<owner>/<repo>/contents/<path>?ref=main"
  

• Post-publish, raw.githubusercontent.com and the HF CDN can lag → verify via gh api / HfApi / the datasets-server API, not the raw CDN.
• Org Gradio/Docker Spaces need a paid HF plan (datasets are free). Relevant only if this data later gets a viz Space — the dataset itself costs nothing.

Rebuild-on-publish (C5, workspace-hub#3488)

Publishing a registered dataset (one that has a config/capabilities.yaml entry in aceengineer-website) should refresh the live site automatically — "live" data without a live runtime dependency.

• save_results_to_hf.py POSTs a Vercel Deploy Hook after a verified publish, so the site rebuilds and picks up the new rows (the build re-fetches HF and re-bakes snapshots).
• The hook URL is a secret, read from $VERCEL_DEPLOY_HOOK_URL — never committed. Setting that env var on a publishing host is the opt-in ("publishes from here refresh the site"). Unset → the publish still succeeds, it just doesn't trigger a rebuild.
• Suppress per-run with --no-deploy-hook. A failed/absent hook never fails the publish.
• Backstop: aceengineer-website/.github/workflows/nightly-rebuild.yml triggers a nightly rebuild (covers datasets updated out-of-band), reading the same URL from a GH Actions secret.
• Owner setup (once): Vercel → project Settings → Git → Deploy Hooks → create a main hook → store the URL as $VERCEL_DEPLOY_HOOK_URL (publishing hosts) and as the VERCEL_DEPLOY_HOOK_URL GitHub Actions secret in aceengineer-website.

Bundled resources

• publish_analysis_to_hf.py — reusable helper: verifies hf auth whoami, creates the dataset repo, uploads a prepared folder (parquet + README), then polls the datasets-server /is-valid to report readiness. Parameterized --repo-id / --folder / --private, with a --dry-run. Run python3 publish_analysis_to_hf.py --help.
• PROMPT.md — a copy-paste, parameterized prompt to hand any agent so it runs this whole flow (reshape → card+provenance → sanity-check → publish → verify) while honoring license routing, the withhold-and-file-issue data-quality gate, and the keep-separate-from--runs convention.