# Kiln Check Finetune Deprecation > Check Kiln's fine-tunable model list for deprecated or unsupported base models. Use when the user wants to audit fine-tuning support, check if fine-tune base models are still valid, or mentions fine-tune model deprecation. - Skill: `gabrielmoreira/kiln-check-finetune-deprecation` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add gabrielmoreira/kiln-check-finetune-deprecation` - Raw SKILL.md: https://api.skillmd.com/api/skills/gabrielmoreira/kiln-check-finetune-deprecation/raw - Safety review: pending (external: skill-scanner PASS, skillspector CAUTION) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: gabrielmoreira (https://skillmd.com/u/gabrielmoreira) - Updated: 2026-09-09 - Page: https://skillmd.com/skills/gabrielmoreira/kiln-check-finetune-deprecation --- # Check Fine-Tune Model Deprecation in Kiln Audit the fine-tunable models listed in Kiln to find base models that are no longer supported for fine-tuning by their providers. This is separate from general model deprecation — a model can still be available for inference but no longer supported as a fine-tuning base. --- ## Global Rules - **Sandbox:** All `curl`, `uv run`, and `python3` commands that hit the network MUST use `required_permissions: ["all"]`. The sandbox blocks network access. - **Env vars:** Source `.env` before running check scripts: `export $(grep -v '^#' .env | xargs)` - **Vertex AI auth:** The Vertex check requires `gcloud` CLI authentication. Before running, ask the user to run `gcloud auth login` if they haven't recently. If `gcloud auth print-access-token` fails, prompt the user to authenticate. - **Non-destructive:** This skill only reports findings. It does not automatically remove or deprecate models. --- ## Supported Providers | Provider | Env Var | Auth | |----------|---------|------| | OpenAI | `OPENAI_API_KEY` | Bearer | | Together AI | `TOGETHER_API_KEY` | Bearer | | Vertex AI | `VERTEX_PROJECT_ID` + `gcloud` CLI auth | OAuth (gcloud) | | Fireworks AI | `FIREWORKS_API_KEY` | Bearer | --- ## Background Kiln has two types of fine-tunable model entries: 1. **Static entries** — Models in `libs/core/kiln_ai/adapters/ml_model_list.py` with `provider_finetune_id` set. Currently used by Together AI and Vertex AI. 2. **Dynamic entries (Fireworks)** — Fetched at runtime from `api.fireworks.ai/v1/accounts/fireworks/models` filtering by `tunable=True`. This list is managed by Fireworks and may include stale/unsupported models. The API endpoint that serves the fine-tune dropdown is `GET /api/finetune_providers` in `app/desktop/studio_server/finetune_api.py`. --- ## Phase 1 – Check Static Fine-Tune IDs Run the check script from the repo root: ```bash uv run python3 .agents/skills/kiln-check-finetune-deprecation/scripts/check_finetune.py static > /tmp/kiln_finetune_static.json ``` This extracts all `provider_finetune_id` entries from `ml_model_list.py` and checks each provider's docs/API to see if the model is still available for fine-tuning: - **Together AI:** Scrapes the Together AI docs page for the list of supported fine-tune models. Together uses full HuggingFace-style IDs (e.g. `meta-llama/Meta-Llama-3.1-8B-Instruct-Reference`). - **Vertex AI:** Checks the Vertex AI publisher models API. Vertex fine-tuning uses specific versioned model IDs (e.g. `gemini-2.0-flash-001`). **Output:** JSON to stdout with per-provider results, human summary to stderr. --- ## Phase 2 – Check Fireworks Dynamic Models Fireworks fine-tunable models are fetched dynamically from their API, not stored in `ml_model_list.py`. The script checks the API's `supervisedLoraTunable` and `supervisedFullParameterTunable` fields (not the old `tunable` field, which is stale) and cross-references against our allowlist. ```bash uv run python3 .agents/skills/kiln-check-finetune-deprecation/scripts/check_finetune.py fireworks > /tmp/kiln_finetune_fireworks.json ``` This script: 1. Fetches all models with `supervisedLoraTunable=True` or `supervisedFullParameterTunable=True` from `api.fireworks.ai/v1/accounts/fireworks/models` 2. Cross-references against `FIREWORKS_SUPPORTED_FINETUNE_MODELS` from `libs/core/kiln_ai/adapters/fine_tune/fireworks_finetune.py` — the same allowlist that filters the runtime fine-tune dropdown 3. Checks both directions: - **Stale allowlist entries**: models in our allowlist that the API no longer marks as supervised-tunable — these may need to be removed - **Missing from allowlist**: models the API marks as supervised-tunable but aren't in our allowlist — these are candidates to add The canonical allowlist lives in `fireworks_finetune.py` and is shared between the runtime dropdown filter and this audit skill. There is a single place to update when Fireworks changes their supported models. **Interpreting results:** The supervised API fields return ~43 models, but our allowlist intentionally filters to ~20 that match Fireworks' documented supported models. Seeing ~23 "models in API but NOT in allowlist" is normal — these are models Fireworks marks as technically tunable but doesn't officially document or support. Only flag these if a model appears in the Fireworks docs but is missing from our allowlist. The important direction is **stale allowlist entries** — models we ship that the API no longer marks as supervised-tunable. **Output:** JSON to stdout, human summary to stderr. --- ## Phase 3 – Report Findings Read the JSON outputs and present findings in a clear table: ```text Fine-Tune Deprecation Audit Results ==================================== STATIC PROVIDERS (provider_finetune_id in ml_model_list.py): ✅ OpenAI: 5/5 found gpt-4.1-2025-04-14, gpt-4.1-mini-2025-04-14, gpt-4.1-nano-2025-04-14, gpt-4o-2024-08-06, gpt-4o-mini-2024-07-18 ✅ Vertex AI: 2/2 found gemini-2.0-flash-001, gemini-2.0-flash-lite-001 ❌ Together AI: 2/4 missing ✅ Qwen/Qwen2.5-72B-Instruct — found ✅ Qwen/Qwen2.5-14B-Instruct — found ❌ meta-llama/Meta-Llama-3.1-8B-Instruct-Reference — NOT FOUND ❌ meta-llama/Meta-Llama-3.1-70B-Instruct-Reference — NOT FOUND ⏭️ SKIPPED (no credentials): - vertex (VERTEX_PROJECT_ID not set) DYNAMIC PROVIDER (Fireworks): ⚠️ 167 models marked tunable in API ⚠️ 15 models in known-good docs list ✅ 11 models in both API and docs ❌ 156 models in API but NOT in docs (likely stale) - accounts/fireworks/models/qwen2-72b-instruct - accounts/fireworks/models/code-llama-7b - ... ⚠️ 4 models in docs but NOT in API - accounts/fireworks/models/gemma-2-9b-it - ... ``` Note: The example above is illustrative. Actual model counts and entries will vary. --- ## Phase 4 – Remediation Based on findings, recommend specific actions: ### For static entries (`provider_finetune_id`): - If a model's fine-tune ID is no longer valid, either: - **Update** the `provider_finetune_id` to a newer version if one exists - **Remove** the `provider_finetune_id` field to stop offering fine-tuning for that model on that provider - **Set `deprecated=True`** on the provider entry if the model is fully gone ### For Fireworks dynamic entries: - The stale models come from Fireworks' API, not our code. Options: - **Filter in our code** — update `fetch_fireworks_finetune_models()` in `finetune_api.py` to filter against a known-good list - **Report to Fireworks** — flag the stale models to Fireworks support - **Both** — filter now, report later **Always ask the user to confirm** before making any code changes. --- ## Phase 5 – Verify After any changes, run the relevant tests: ```bash uv run python3 -m pytest app/desktop/studio_server/test_finetune_api.py -q ``` --- ## Checklist - [ ] Env vars sourced from `.env` - [ ] Vertex auth confirmed (`gcloud auth print-access-token` works, or skip Vertex) - [ ] Static fine-tune IDs checked against provider APIs (Together, Vertex) - [ ] Fireworks dynamic models cross-referenced against known-good list - [ ] Findings reported to user with clear table - [ ] User confirmed remediation approach - [ ] Code changes made (if any) - [ ] Tests pass after changes - [ ] FIREWORKS_SUPPORTED_FINETUNE_MODELS in fireworks_finetune.py updated if docs have changed