# Audit Skill Md > # Audit `skills/datafusion_python/SKILL.md` You are auditing the user-facing skill at [`skills/datafusion_python/SKILL.md`](../../skills/datafusion_python/SKILL.md) against the current state of the Python API. The skill is the source of truth for how AI coding assistants are taught to write `datafusion-python` code, so it must match what the project actually ships. This skill identifies gaps caused by upstream syncs, refactors, or renames, and (if asked) applies the edits directly to `SKILL.md`. The skill is most usefully run **after** the `check-upstream` step of an upstream sync (see `dev/release/upstream-sync.md`) — once any new APIs are exposed, this skill makes sure they get documented. ## What the skill covers The user-facing `SKILL.md` documents these public surfaces. This list is not exhaustive — if a new top-level area is added (e.g., a new `Catalog` API exposed at the package root), include it. | Surface | Module | Sections in SKILL.md | |---|---|---| | `SessionContext` | `python/datafusion/context.py` | "Data Loading" | | `DataFrame` | `python/datafusion/dataframe.py` | "DataFrame Operations Quick Reference", "Executing and Collecting Results", "Idiomatic Patterns" | | `Expr` | `python/datafusion/expr.py` | "Expression Building", "Common Pitfalls" | | `functions` | `python/datafusion/functions/__init__.py` | "Available Functions (Categorized)", scattered uses throughout | | `functions.spark` | `python/datafusion/functions/spark.py` | "Available Functions (Categorized)" → "Spark-Compatible Functions" subsection | | Top-level helpers (`col`, `lit`, `WindowFrame`, ...) | `python/datafusion/__init__.py` | "Import Conventions", "Core Abstractions" | ## Scope argument The user may specify a scope via `$ARGUMENTS` to limit the audit. If no scope is given or `all` is specified, audit every area. | Scope | Audit target | |---|---| | `session-context` | `SessionContext` methods and the "Data Loading" section | | `dataframe` | `DataFrame` methods and the operations / executing / patterns sections | | `expr` | `Expr` methods/operators and the "Expression Building" section | | `functions` | `functions/__init__.py` `__all__` and the "Available Functions (Categorized)" section | | `spark-functions` | `functions/spark.py` `__all__`, the "Spark-Compatible Functions" subsection, and the divergent-semantics table | | `patterns` | "Idiomatic Patterns" section — confirm patterns still match recommended style | | `pitfalls` | "Common Pitfalls" — confirm each pitfall still reproduces, drop ones fixed upstream | | `version-notes` | Cross-check version annotations (see below) | | `all` | Everything above | ## Inputs to read Before producing the report: 1. `skills/datafusion_python/SKILL.md` — the document being audited. 2. The relevant Python module(s) for the chosen scope. Public surface is the `__all__` list (where defined) plus `class` and `def` symbols not prefixed with `_`. 3. `Cargo.toml` (root) for the current `datafusion-python` version — read the `version` field under `[workspace.package]` (format `NN.0.0`). The major version always matches the upstream `datafusion` crate, so a single `datafusion-python` version expresses both. `python/datafusion/__init__.py`'s `__version__` is the same value exposed at runtime. 4. Recent commits touching the relevant module(s) for context on what changed since the last sync: ```bash git log --oneline -- python/datafusion/dataframe.py | head -20 ``` ## What to look for Walk through each scoped area and flag four kinds of issues. ### 1. New APIs not mentioned For each public symbol in the module's `__all__` (or each public class method), check whether it appears anywhere in `SKILL.md`. A symbol is "covered" if it shows up in: - A code block (the strongest signal — it's demonstrated). - The "Available Functions (Categorized)" list. - The SQL-to-DataFrame Reference table. **Decide whether each missing symbol deserves an entry.** Not every public symbol belongs in `SKILL.md` — the skill is curated for the patterns users hit daily, not exhaustive API reference. Use these heuristics: - **Add it** if it replaces or supersedes something already in the skill (e.g., a new operation that is the idiomatic alternative to a documented workaround). - **Add it** if it fits a category already present (a new aggregate function goes in the aggregate list; a new join type goes in the joining section). - **Add it** if it changes how a documented pattern should be written. - **Skip it** if it is genuinely niche / advanced / experimental. - **Skip it** if it is internal plumbing exposed for FFI but not user-facing. When you flag a missing symbol, include a one-line proposed insertion point (which section / which table row) so a reviewer can decide quickly. ### 2. Stale mentions For each function name, method name, or import shown in `SKILL.md`, verify it still exists in the current API: - Function names mentioned in prose or in the categorized list should appear in `python/datafusion/functions/__init__.py`'s `__all__`. - Spark function names mentioned in the "Spark-Compatible Functions" subsection should appear in `python/datafusion/functions/spark.py`'s `__all__`. Also confirm the divergent-semantics table still matches the current spark vs. main signatures. - Method calls in code blocks should resolve against the current class. - Imports (`from datafusion import ...`) should succeed against the current `__init__.py`. A quick way to check imports without running them: ```bash python -c "from datafusion import SessionContext, col, lit; from datafusion import functions as F; print('ok')" ``` For each stale mention, propose either: - a rename to the current name, or - removal if the API is gone with no replacement. ### 3. Examples that drifted from idiomatic style The skill teaches a Pythonic style: prefer plain strings to `col(...)` when a column reference is all you need; prefer raw Python values to `lit(...)` where auto-wrapping applies. Recent refactors (see the `make-pythonic` skill) keep moving more functions toward accepting native types. For each code example in `SKILL.md`, check: - Does it use `lit(value)` where a raw value would work? Comparison RHS, arithmetic with a column, etc. all auto-wrap. (Reserve `lit()` for the cases listed in pitfall #2.) - Does it use `col("name")` where a plain string would work? `select(...)`, `aggregate([keys], ...)`, `sort(...)`, `sort_by(...)` all accept plain name strings. - Do `functions.py` calls match the current pythonic signature for that function? If `make-pythonic` recently changed a signature (e.g., `repeat(string, n: Expr | int)`), the example should pass `3` rather than `lit(3)`. - Does any example use a deprecated or removed parameter name? For drift, propose the updated snippet. If the change is purely stylistic and the older form still works, mark the suggestion as **non-blocking**. ### 4. Missing or stale version notes When an API depends on a specific version, the skill should say so — otherwise an agent referencing the skill in an older project will write code that fails at import or at runtime. `datafusion-python` shares its major version number with the upstream `datafusion` crate (e.g., `datafusion-python 53.x` tracks upstream `datafusion 53`). Always express version requirements in terms of `datafusion-python` only — there is no need to call out upstream and package versions separately. Add a version note when: - A method or function shown in the skill was added in a specific release (e.g., a new `DataFrame` method that didn't exist before 53). - A breaking change altered behavior in a specific release (signature change, default-value change, new required argument). - A pitfall was fixed in a specific release. Either annotate the pitfall block with "fixed in datafusion-python NN, kept here for users on older versions" or remove it once the supported floor moves past that version. Format for version notes (inline, italicized): ```markdown *Requires datafusion-python 53 or newer.* ``` For each missing/stale version note, propose the exact line and where it belongs. ## How to discover changes since the last audit If the user supplies a previous version or commit SHA where the audit was last run, diff against it: ```bash # Public-API-relevant changes since SHA git log --oneline ..HEAD -- python/datafusion/ # Whose signatures actually moved git diff ..HEAD -- python/datafusion/functions.py | grep '^[+-]def ' ``` If no prior audit point is given, fall back to "since the last upstream sync" by inspecting commits that touch `Cargo.toml`'s `datafusion` pin: ```bash git log --oneline -- Cargo.toml | grep -i datafusion | head -5 ``` ## Output Format Produce a report grouped by scope. Each finding is one bullet with a proposed action, so a maintainer can review the list quickly and apply edits in order. ``` ## SKILL.md Audit (scope: ) Audited against: - skills/datafusion_python/SKILL.md @ - datafusion-python ### New APIs to cover - `DataFrame.foo()` — added in datafusion-python 53. Insert in "DataFrame Operations Quick Reference" under . Proposed snippet: ```python df.foo(...) ``` ### Stale mentions - "old_function_name" referenced in the categorized list (line N) — renamed to "new_function_name". Replace. ### Drifted examples - "Filtering" section, `df.filter(col("a") > lit(10))` — drop `lit(10)`, auto-wrap applies. (non-blocking) - "Aggregation" section, `df.aggregate([col("region")], ...)` — pass `"region"` as a plain string per "Projection" guidance. ### Version notes - `DataFrame.foo()` block needs *Requires datafusion-python 53 or newer.* - "Common Pitfalls" #N — fixed in datafusion-python 53; remove the pitfall and update the SQL-to-DataFrame row to no longer flag the workaround. ### No-change confirmed - `SessionContext` data-loading section — all entries match current API. ``` If asked to apply the changes, edit `skills/datafusion_python/SKILL.md` directly with `Edit` tool calls, one finding at a time, and re-run the relevant doctest sanity check at the end: ```bash pytest --doctest-modules python/datafusion -q ``` ## What NOT to flag - **Internal helpers / underscored names.** Private symbols are not part of the user-facing surface. - **Functions intentionally omitted.** Niche / advanced APIs (custom catalogs, raw FFI plumbing, low-level execution plan accessors) live in the API reference, not the skill. If an omission was deliberate and a comment / commit explains why, leave it out. - **Style nits inside explanatory prose.** The skill mixes example code and prose; only enforce the pythonic style on actual code blocks. - **Function-by-function coverage of every `functions.py` symbol.** The "Available Functions (Categorized)" list is curated by category, not exhaustive. Adding a single new aggregate to the aggregate list is enough — the user follows the pointer to the API reference for the rest. ## Coordination with other skills - Run `/check-upstream` first to expose any missing upstream APIs into the Python layer. Without that, this skill cannot recommend documenting something that is not yet exposed. - Run `/make-pythonic` before this skill if a Pythonic-signature pass is planned for a release — that way this skill can update examples to the final signature in one shot rather than churning them twice. - The order during an upstream sync (PR 3 of `dev/release/upstream-sync.md`) is therefore: `/check-upstream` → `/make-pythonic` (optional) → `/audit-skill-md`.