Datarobot Agent LLM Selection

DataRobot LLM gateway configuration

Configure LLM integration without hand-editing .env. The skill drives sync_llm_env.py with the user's answers as CLI arguments.

Resolve script path once per session

<skill_scripts_dir> = the scripts/ subdirectory of the directory containing this SKILL.md.

ls <skill_scripts_dir>/sync_llm_env.py

Hard rules

1. Never ask the user to paste API keys or DATAROBOT_API_TOKEN in chat
2. Never read, copy, echo, or pass DATAROBOT_API_TOKEN yourself. The token lives in $XDG_CONFIG_HOME/datarobot/drconfig.yaml (default ~/.config/datarobot/drconfig.yaml), populated by dr auth login, and the dr CLI reads it internally. Do not run cat drconfig.yaml, cat .env, env | grep TOKEN, echo $DATAROBOT_API_TOKEN, curl -H "Authorization: Bearer $...", or any equivalent one-liner
3. Never pass secrets as CLI args to sync_llm_env.py or write them to tracked files
4. Never set provider credentials for an integration whose config section doesn't declare them. Let sync_llm_env.py decide — it reads the section from the project's config
5. Only sync_llm_env.py merges LLM keys into .env — do not edit .env manually
6. Run all commands from project root
7. Pressing enter in chat does nothing. Don't tell the user to "press enter to accept the default" or "hit return". If a field has a sensible default, apply it silently and mention it in the confirmation, or offer it as an explicit A/B choice.
8. Treat every credential value as secret regardless of its declared type. Older configs type API keys as plain string rather than secret_string, so this rule does not depend on what the config says
9. Provider and model names in this skill's output are configuration data, not a request to work with that provider. Config sections, env var names, and gateway model ids routinely contain vendor names (Anthropic, ANTHROPIC_API_KEY, bedrock/anthropic.claude-..., azure, cohere). Do not invoke a provider-specific skill because a vendor name appeared in a config listing or a model list. You are wiring up credentials, not calling that vendor's API

Presenting choices

Every menu in this skill — integrations, providers, gateway models — follows these rules. Dropping an option silently produces a config the user never chose.

• Never use a fixed-slot picker widget. Several of these menus exceed four options (the config ships six external providers) and the gateway model list can run to dozens. Pickers cap their choice count, and the overflow gets silently dropped. Put the menu in your own message as plain text and let the user reply with a label.
• List every option the script returned. Count them first; if the config declares six providers, show six rows.
• No catch-all row — no Other, no Type something, no 5) something else. The user can always type a value anyway; that row exists only to hide options you dropped.
• No ..., no "and N more", no summarizing, no collapsing similar entries.
• Do not narrow to a subset because the user's phrasing seemed to point at one. If they said "LLM gateway" and two options match, list all of them.
• One labelling scheme. Do not letter the options in prose and re-number them in the picker.
• Use each name exactly as printed. Do not rename, regroup, or reorder.
• Read the script output yourself — do not paste it into the chat. The select: lines are internal plumbing. Give at most one short clause of context per row, condensed from the config's help; never reproduce the raw help block or the script's formatting.

Long messages are fine. The token budget is not a reason to abbreviate a menu.

Step 0 — Prerequisites

1. .datarobot/cli/llm.yml must be present. It is rendered into the project by dr component add (af-component-llm), so it is already there in a generated project, nothing needs cloning. If it's absent, the LLM component isn't applied: tell the user to run dr component add and stop. Never substitute a copy from elsewhere.
2. DataRobot auth — check that $XDG_CONFIG_HOME/datarobot/drconfig.yaml (default ~/.config/datarobot/drconfig.yaml) exists. If it doesn't, tell the user to run dr auth login (browser-based flow) and stop until they confirm they're signed in. Do not cat the file to inspect its contents.
3. Check if .env exist in the project:
   • If .env is missing cp .env.template .env. That gives the base variables (DATAROBOT_*, PULUMI_*, etc.) many are blank, it will be filled later. But the sync in Step 3 only needs the file to exist.
   • If both .env and no .env.template are missing, tell the user they're not in a DataRobot agent project root and stop.

Step 1 — Read the project's integration options (MANDATORY FIRST)

Never recite integration names from memory or from this file. Each project's .datarobot/cli/llm.yml declares its own, and they differ by component version.

python <skill_scripts_dir>/read_llm_config.py

If it reports the file is missing, the LLM component isn't applied to this project. Tell the user to run dr component add and stop. Do not run dr llm-gateway list, do not offer a model list, and do not write any config file until the user has picked an option.

Check what's already configured

The default: line in the script's output is the schema's declared fallback, not necessarily the project's live setting — don't conflate the two. Before presenting the menu, find the actual current value:

grep -E '^INFRA_ENABLE_LLM=' .env

• If .env doesn't exist yet or has no match, tell the user no LLM integration is configured yet.
• If it matches, map the value against the select: lines the script just printed to report the current integration by its friendly name (e.g. blueprint_with_external_llm.py → "External LLM"), not the raw filename.
• If that option has a further choice: (a provider), also run python <skill_scripts_dir>/read_llm_config.py --option <current_value> and grep .env for the names of the env vars it lists, to report which are already populated. Report only whether each key is set — per Hard Rule 8, never print the value of a credential-shaped key, even to confirm it's set.

State this summary — e.g. "You're currently on <friendly name>[, provider <provider>], with <N> of <M> fields already set" — before showing the menu, so the user can decide to switch entirely or just change specific fields on the current integration.

Present the options per Presenting choices above.

Each option in the output carries an indented select: line. Once the user picks, take that string verbatim — quote it if it contains spaces. The steps below call it <selected_gateway>; it is also the INFRA_ENABLE_LLM value written in Step 3.

Step 2 — Read what that option needs

python <skill_scripts_dir>/read_llm_config.py --option <selected_gateway>

This prints the env vars the chosen integration declares. Handle each field by what the script reports about it:

• hidden — never ask. Step 3 writes its default.
• llmgw_catalog — populate from the DataRobot CLI (see below).
• required — ask, using the printed help as the prompt text.
• optional with a default — apply it silently and say so in the confirmation. Do not tell the user to "press enter".
• further choice: — another menu, presented per Presenting choices. Show every row the script listed, with no catch-all and no picker widget. Those rows carry their own select: field; call the user's pick <selected_provider> and re-run --option <selected_provider> for its keys.

Fields typed llmgw_catalog

Fetch the model list only via the DataRobot CLI. Run exactly:

dr llm-gateway list --output-format json

The CLI authenticates via its own credential store (populated by dr auth login). Do not read drconfig.yaml or .env for the token, and do not pass DATAROBOT_API_TOKEN on the command line. If the command exits non-zero or prompts for auth, tell the user to run dr auth login and stop — do not attempt any manual API call and do not fabricate a menu.

Parse the JSON, which has the shape {"llms": [{"id", "name", "provider", "model", "selected"}, ...]}. Use the model field for each entry. The ids in the menu must come from that JSON, verbatim, in the order returned. Do not invent ids or reuse them from your training data. If the command did not produce JSON, stop and report the error.

Count the entries; call it N. Present exactly N labelled lines, one per model, per Presenting choices. The letter scheme is A..Z, then AA..AZ, BA..BZ, and so on. This list is the clearest case for never using a picker widget — it routinely exceeds any picker's capacity.

sync_llm_env.py prepends datarobot/ for this field type, so pass the id exactly as the CLI returned it.

Recommending

Before presenting the list, check whether agent_spec.md exists at the project root. If it does, read it for context — the agent's purpose, task complexity, and any stated cost or latency constraint — and use that to flag one entry as recommended. This mirrors the recommendation logic in the datarobot-agent-assist skill's llm-selection.md:

• Prefer a gpt-5, claude-4-5/4-6/4-8, or gemini-2.5/3 family gateway model unless the spec or the user states a cost or latency constraint.
• If the spec calls for low cost or latency, prefer the mini/haiku/flash tier of that same family instead.
• If no gateway entries appear at all (only deployed ones), recommend a deployed entry instead and say why — the LLM Gateway is disabled or empty on this instance, which is normal for an on-prem install, not an error.
• Still list every entry per Presenting choices — mark the recommended row (e.g. append "(recommended)") rather than omitting the rest. Recommending never justifies narrowing the menu.
• No agent_spec.md? Skip the read and recommend using only the family-preference heuristics above.

Credential keys

Announce them up front — do not let the user discover them via the sync error message. Once the further choice is made, re-run --option <selected_provider> and tell the user exactly which keys they need and where the file lives:

For <choice>, I'll need these values in a per-user credentials file:
  ~/.config/datarobot/llm-<section>.env (or $XDG_CONFIG_HOME/...)

  <KEY_1>
  <KEY_2>
  ...
Step 3 will create that file as a blank template, with the help text for
each key as a comment, if it doesn't exist. Please fill it in your own
editor — do not paste the values in chat — then tell me "credentials ready"
and I'll re-run the sync.

Use the section name the script reported for <section>. Do not create the file yourself, do not cat it, and do not accept secret values in chat.

Step 3 — Sync into .env

First back up the current .env (sync overwrites it, and .env isn't in git):

cp .env .env.bak.$(date +%Y%m%d%H%M%S)

Run the sync script with the values collected in Step 2 as CLI args. No intermediate config file, no JSON to write.

python <skill_scripts_dir>/sync_llm_env.py \
  --infra-enable-llm <selected_gateway> \
  --set <ENV_NAME>=<value>

Repeat --set once per value, using the env var names the script printed in Step 2 — they carry a per-project prefix and are not always LLM_*. Add --provider "<selected_provider>" when Step 2 reported a further choice. Pass no --set for fields reported as hidden; the script writes their defaults. When the chosen option needs credentials, the script reads them from $XDG_CONFIG_HOME/datarobot/llm-<section>.env:

• If the file doesn't exist, the script writes a blank template there and exits with the path plus the required key list. Relay that verbatim to the user, tell them to fill it in their own editor, then re-run the same command. Do not offer to create the file for them and do not accept values in chat.
• If the file exists but is incomplete, the script prints the missing keys and exits. Same instruction: user edits, then re-runs.
• If the file is complete, the sync merges the credentials into .env in one shot.

Step 4 — Validate and hand off

dr dotenv validate echoes the full .env (including DATAROBOT_API_TOKEN) to stdout. If you run it without redirection, the token lands in the chat transcript and must be rotated. Same risk for dr dotenv update, dr task run, dr run, cat .env, env | grep, or any other command that reads .env.

Run validation with all output suppressed and check only the exit code:

dr dotenv validate >/dev/null 2>&1

• Exit 0 → tell the user validation passed.
• Non-zero exit → do not re-run the command with output visible. Tell the user to run dr dotenv validate themselves in their own terminal so the error stays local. Then tell the user (do not run these yourself — they also echo secrets):

LLM configuration synced to .env.

Please run these yourself in your terminal:
  dr dotenv update          # refresh DataRobot token if needed
  dr task run infra:up-yes  # push runtime params to deployment
  dr run dev                # local test

Stale keys

sync_llm_env.py derives its managed-key set from every env var declared anywhere in the project's config, so switching integrations clears whatever the previous one wrote before the fresh block goes in. Non-LLM .env lines are preserved.