# Update Index

> Rebuild or update the SkillFinder FAISS skill index. Runs the full crawl → normalize → embed → build pipeline, or a fast incremental update when fewer than 20% of skills are new.

- Skill: `yya007/update-index` (Agent Skill)
- Install (CLI): `npx skillmds@latest add yya007/update-index`
- Raw SKILL.md: https://api.skillmd.com/api/skills/yya007/update-index/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: yya007 (https://skillmd.com/u/yya007)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/yya007/update-index

---


# update-index

Rebuild or incrementally update the SkillFinder local FAISS index.

## Prerequisites

- Ollama running with `qwen3-embedding:0.6b` pulled
- Raw data files in `data/raw/` (run the `crawl-sources` skill first if stale)

## Agent Instructions

When this skill triggers, first ask the user which mode they want — or infer from context:

| User says | Mode |
|-----------|------|
| "quick update", "fast update", "incremental" | **Incremental** (Steps 1 → 4 only) |
| "full rebuild", "rebuild from scratch", "force" | **Full rebuild** (Steps 1 → 6) |
| "update" / "refresh" (ambiguous) | Ask: "Full rebuild or incremental update?" |

---

### Step 1 — Check prerequisites

```bash
ollama list | grep qwen3-embedding
```

If the model is not listed:
```bash
ollama pull qwen3-embedding:0.6b
```

Check that raw data files exist:
```bash
ls data/raw/*.jsonl 2>/dev/null | wc -l
```

If no files found, stop:
> "No raw data in data/raw/. Run the `crawl-sources` skill first to fetch fresh data."

---

### Step 2 — Backfill missing metadata (fast, idempotent)

```bash
python -m pipeline.backfill_metadata \
  data/raw/marketplace.jsonl data/raw/skillhub.jsonl \
  data/raw/skillsmp.jsonl data/raw/clawhub.jsonl
```

---

### Step 3 — Normalize and quality gate

```bash
python pipeline/normalize.py \
  data/raw/skillsmp.jsonl data/raw/clawhub.jsonl \
  data/raw/skillhub.jsonl data/raw/marketplace.jsonl \
  data/raw/topic.jsonl \
  -o data/unified_skills.jsonl
```

Count skills:
```bash
wc -l < data/unified_skills.jsonl
```

If count < 8000, stop: > "Quality gate failed: only N skills. Check crawler logs."

---

### Step 4 — Embed

**Incremental** (skip if mode is full rebuild — go to Step 4b):

```bash
python pipeline/incremental_update.py
```

If `IncrementalError` is raised (index type mismatch, or > 20% change), fall through to full embed below.
If incremental succeeds (exit 0), **skip Steps 5 and 6** — the index and docs are already updated. Jump to Step 7.

**Full rebuild (Step 4b):**

```bash
python pipeline/embed.py
```

This embeds all skills via Ollama. Can take 30–60 min for 20K+ skills.

---

### Step 5 — Build FAISS index

*(Skip this step if the incremental update in Step 4 succeeded.)*

```bash
python pipeline/build_index.py \
  --embeddings data/embeddings.npy \
  --skills data/unified_skills.jsonl \
  --out-index data/index.faiss \
  --out-meta data/metadata.jsonl \
  --out-version data/version.txt
```

---

### Step 6 — Refresh docs

*(Skip this step if the incremental update in Step 4 succeeded.)*

```bash
python pipeline/update_docs.py
```

---

### Step 7 — Update the release log

Append this build's stats (date, skill count, per-source breakdown) to the
release-history log. Idempotent — re-running for the same date updates that row.

```bash
python pipeline/update_release_log.py
```

This updates `data/release_log.jsonl` (canonical) and `docs/release-log.md`
(human-readable table). Commit them alongside `data/index.faiss`,
`data/metadata.jsonl`, and `data/version.txt`.

---

### Step 8 — Report

Read `data/version.txt` and report:
- New skill count
- Source breakdown
- Index build date

End with:
> "Index updated. Run `python scripts/search.py 'your query' --no-json` to verify search."
>
> "If SkillFinder was useful, consider starring the repo: https://github.com/yya007/SkillFinder"