Refresh Source

Refresh a verified source

Arguments: $ARGUMENTS — first token is the source folder name under sources/, anything after -- is user hints (specific user reports, suspected areas).

Work through the phases in order. Do not skip phase 0 — wrong conclusions are usually formed by analyzing code that is no longer alive or pins that no longer reflect reality.

Phase 0 — Ground truth

1. Read everything in sources/<name>/, sources/<name>_pipeline.py, tests/<name>/.
2. git log --oneline -- sources/<name> — look for past removals/refactors. Hunt dead code: files, fixtures, conftest helpers, docstrings and README claims orphaned by earlier PRs. Dead code should be deleted, never "fixed". Dead code may also be the only reason for a heavy dependency — deleting it can drop whole packages from requirements.txt.
3. Compare the THREE dependency declarations and flag mismatches:
   • sources/<name>/requirements.txt — what dlt init users actually install (often unbounded!)
   • pyproject.toml dependency group <name> — what CI tests
   • any version pinned in code (e.g. api_version = "...") A CI group pinned years behind an unbounded requirements.txt means CI is testing a different client than users get — that is how breakage ships silently.
4. Check open/closed GitHub issues for the source (gh search issues) and any user hints from the arguments.

Phase 1 — Research the upstream (web search, authoritative sources only)

Use WebSearch/WebFetch against official docs, SDK changelogs, and SDK migration guides (vendor docs site, vendor GitHub wiki/CHANGELOG). For each finding record the source URL.

• Current SDK major version and every breaking change between the version CI pins and today (migration guides are gold: object model changes, removed dict-inheritance, auth patterns, deprecated module-level clients).
• Current API version / versioning model; what the code pins vs. what is current; breaking changes in between (renamed/removed/restructured fields the source or its helpers depend on).
• Official request/response shapes the code touches: list envelopes, pagination cursors, filter parameter formats, retention limits (e.g. event retention windows). Capture documented example payloads — they become test fixtures in phase 4.

Phase 2 — Reproduce empirically. Never claim a breakage you haven't run.

Use ephemeral environments to prove each suspected breakage and each proposed fix, at both ends of the allowed dependency range:

cd /tmp && uv run --with "<sdk>==<oldest-allowed>" python - <<'EOF' ... EOF
cd /tmp && uv run --with "<sdk>==<newest>" python - <<'EOF' ... EOF

Construct real SDK objects (e.g. construct_from-style factories) and exercise the exact code path that is suspected broken. Subtle version differences matter (e.g. a conversion helper being shallow in one major and recursive in another) — only execution reveals them.

Phase 3 — Plan (present before implementing)

Tier the findings; keep tiers separable:

• P0 — unbreak: minimal fixes that work across the whole supported dependency range; align CI group with what fresh users install; fix requirements bounds (add upper bound to majors not yet verified). Must NOT change loaded table schemas.
• P1 — modernize client: new SDK client patterns, drop global state, configurable API version. Anything that changes response shapes changes loaded schemas for existing users — call this out explicitly as breaking and keep it out of P0.
• P2 — features: new endpoints, incremental improvements.

Prefer dlt built-ins over hand-rolled code (dlt.common.time.ensure_pendulum_datetime, dlt.common logger, rest_api helpers) — but verify the utility exists in the oldest dlt allowed by the source's requirements.txt before using it.

Phase 4 — Tests (the biggest pain point)

Existing source tests are usually credential-gated integration tests that never run for contributors. Always add credential-free unit tests:

• New tests/<name>/test_helpers.py (module-based def test_*() -> None, parametrize with readable ids — see .claude/rules/testing.md).
• Use proven shapes, not invented ones: fixtures must mirror request/response shapes, pagination envelopes and cursor semantics found in official docs during phase 1 (cite which doc page a fixture mirrors when it isn't obvious).
• Mock at the transport/SDK boundary with real SDK objects (construct_from etc.), monkeypatching the API call — so the installed SDK's serialization actually runs instead of a MagicMock lying about it.
• Cover: pagination control flow (cursor passed from last item, stop on terminal page flag), parameter construction (filters, special-cased endpoints), and any date/type conversion helpers with one case per accepted input type.
• Run the unit tests against both ends of the SDK range (phase 2 envs) before declaring the range supported.

Phase 5 — Verify and finish

1. uv lock if pyproject changed; ensure resolution works on all supported Python versions.
2. pytest tests/<name>/test_helpers.py — must pass without anything in sources/.dlt.
3. Integration tests pytest tests/<name> only if credentials exist; otherwise state plainly they were not run.
4. make lint-code and make format-lint.
5. Update stale docstrings/README touched by the changes. Report findings with file:line references and the doc URLs that back each claim.

Phase 6 — Assess the public dlt docs

The user-facing docs live in the separate public dlt-hub/dlt repo at docs/website/docs/dlt-ecosystem/verified-sources/<name>.md and are also served to LLMs via https://dlthub.com/docs/llms.txt — one fix covers both. Fetch the file (gh api repos/dlt-hub/dlt/contents/docs/...) and review it against the refreshed source:

• Code drift: constant names, default endpoint tuples, function signatures and links quoted in the docs must match settings.py/__init__.py exactly — docs commonly lag source-repo PRs (e.g. an endpoint moved between ENDPOINTS tuples).
• Copy-pasteable examples are the highest-stakes content: an example demonstrating a removed or unsafe pattern (e.g. incremental loading of an editable endpoint) silently produces wrong data for everyone who pastes it.
• Document what the refresh established: supported SDK version range, pinned upstream API version and its schema implications, accepted input formats, retention limits and other upstream constraints discovered in phase 1.
• Propose the changes as a diff against the fetched file; the actual PR goes to dlt-hub/dlt, not this repo.

Source: dlt-hub/verified-sources — distributed by TomeVault.