Add an AI-Q Data Source
Use this skill when a developer wants to add a retrieval or search source to
AI-Q and expose it as a toggleable source in the UI. A data source is a NeMo
Agent Toolkit (NAT) function package under sources/, registered in the
data_source_registry.
Start Here
- Confirm this is a new retrieval/search source (not a UI, auth, or prompt
change). For a general utility function, use
aiq-add-tool instead.
- Read the authoritative files below before editing.
- Copy the closest existing source package rather than inventing a new shape.
- Never print or commit API keys; resolve secrets at runtime via
SecretStr.
Authoritative References
docs/source/extending/adding-a-data-source.md: canonical package and
registration walkthrough (the steps below mirror it).
sources/google_scholar_paper_search/: complete example package with a
client, a config + registration, a graceful missing-secret stub, and tests.
sources/tavily_web_search/: minimal source package for comparison.
src/aiq_agent/common/data_source_registry.py: the data_source_registry
config (name="data_source_registry") that drives GET /v1/data_sources.
docs/source/customization/tools-and-sources.md: how the registry maps to UI
toggles and per-request filtering.
frontends/ui/src/features/layout/data-sources.ts: the UI DataSource type;
sources are fetched dynamically, so usually no UI code change is needed.
Longer procedures live in this bundle:
- references/package-layout.md: package files,
pyproject.toml, config class, @register_function, and the missing-secret stub.
- references/registry-and-ui.md: registering in
YAML and how the registry surfaces toggles and filtering.
- references/validation.md: install, test, and lint
commands with expected results.
Workflow
- Pick the closest existing package under
sources/ and inspect its layout.
- Create
sources/<my_data_source>/ with src/register.py, the client
module, pyproject.toml, and tests/ (see package-layout reference).
- Define a
FunctionBaseConfig subclass with a stable name= and resolve any
API key via SecretStr; register it with @register_function.
- Yield a graceful stub when the required secret is missing.
- Install the package editable and add it to the
data_source_registry in the
relevant config under configs/.
- Add focused tests; run the validation commands below.
- Summarize changed files and paste the test/lint evidence.
Validation
Run the narrowest commands first; broaden only if the change touches shared code.
uv pip install -e ./sources/my_data_source
uv run pytest sources/my_data_source/tests
uv run ruff check sources/my_data_source
uv run ruff format --check sources/my_data_source
Expected: the package installs, its tests pass, and Ruff reports no lint or
format failures for the new source package.
Common Mistakes
- Forgetting to add the source to the
data_source_registry, so the UI cannot
toggle it and agents do not inherit the tool.
- Omitting the
[project.entry-points."nat.plugins"] entry in pyproject.toml,
so NAT never discovers the registration.
- Crashing on a missing API key instead of yielding a stub that returns a clear
error message.
- Returning unstructured or citation-poor output, which weakens report grounding.
- Printing API keys or embedding secrets in YAML instead of using environment
variables or
SecretStr.
Related Skills
aiq-configure-workflow
aiq-add-tool
aiq-release-qa
aiq-prepare-pr
1---2name: aiq-add-data-source3description: Use when adding or changing an AI-Q data source under sources/, registering it as a NeMo Agent Toolkit function, wiring it into the data_source_registry for UI toggles, or validating retrieval behavior with tests.4license: Apache-2.05compatibility: Claude Code, Codex, Cursor, OpenCode, and Agent Skills-compatible tools.6metadata:7 source: https://skillmd.com/skills/nvidia-ai-blueprints-aiq/aiq-add-data-source8 version: "0.1.0"9 source-repo: "NVIDIA-AI-Blueprints/aiq"10 tags: "aiq nemo-agent-toolkit data-source sources registry"11allowed-tools: Read Bash Edit12---1314# Add an AI-Q Data Source1516Use this skill when a developer wants to add a retrieval or search source to17AI-Q and expose it as a toggleable source in the UI. A data source is a NeMo18Agent Toolkit (NAT) function package under `sources/`, registered in the19`data_source_registry`.2021## Start Here2223- Confirm this is a new retrieval/search source (not a UI, auth, or prompt24 change). For a general utility function, use `aiq-add-tool` instead.25- Read the authoritative files below before editing.26- Copy the closest existing source package rather than inventing a new shape.27- Never print or commit API keys; resolve secrets at runtime via `SecretStr`.2829## Authoritative References3031- `docs/source/extending/adding-a-data-source.md`: canonical package and32 registration walkthrough (the steps below mirror it).33- `sources/google_scholar_paper_search/`: complete example package with a34 client, a config + registration, a graceful missing-secret stub, and tests.35- `sources/tavily_web_search/`: minimal source package for comparison.36- `src/aiq_agent/common/data_source_registry.py`: the `data_source_registry`37 config (`name="data_source_registry"`) that drives `GET /v1/data_sources`.38- `docs/source/customization/tools-and-sources.md`: how the registry maps to UI39 toggles and per-request filtering.40- `frontends/ui/src/features/layout/data-sources.ts`: the UI `DataSource` type;41 sources are fetched dynamically, so usually no UI code change is needed.4243Longer procedures live in this bundle:4445- [references/package-layout.md](references/package-layout.md): package files,46 `pyproject.toml`, config class, `@register_function`, and the missing-secret stub.47- [references/registry-and-ui.md](references/registry-and-ui.md): registering in48 YAML and how the registry surfaces toggles and filtering.49- [references/validation.md](references/validation.md): install, test, and lint50 commands with expected results.5152## Workflow53541. Pick the closest existing package under `sources/` and inspect its layout.552. Create `sources/<my_data_source>/` with `src/register.py`, the client56 module, `pyproject.toml`, and `tests/` (see package-layout reference).573. Define a `FunctionBaseConfig` subclass with a stable `name=` and resolve any58 API key via `SecretStr`; register it with `@register_function`.594. Yield a graceful stub when the required secret is missing.605. Install the package editable and add it to the `data_source_registry` in the61 relevant config under `configs/`.626. Add focused tests; run the validation commands below.637. Summarize changed files and paste the test/lint evidence.6465## Validation6667Run the narrowest commands first; broaden only if the change touches shared code.6869```bash70uv pip install -e ./sources/my_data_source71uv run pytest sources/my_data_source/tests72uv run ruff check sources/my_data_source73uv run ruff format --check sources/my_data_source74```7576Expected: the package installs, its tests pass, and Ruff reports no lint or77format failures for the new source package.7879## Common Mistakes8081- Forgetting to add the source to the `data_source_registry`, so the UI cannot82 toggle it and agents do not inherit the tool.83- Omitting the `[project.entry-points."nat.plugins"]` entry in `pyproject.toml`,84 so NAT never discovers the registration.85- Crashing on a missing API key instead of yielding a stub that returns a clear86 error message.87- Returning unstructured or citation-poor output, which weakens report grounding.88- Printing API keys or embedding secrets in YAML instead of using environment89 variables or `SecretStr`.9091## Related Skills9293- `aiq-configure-workflow`94- `aiq-add-tool`95- `aiq-release-qa`96- `aiq-prepare-pr`