Upstream Sync

Upstream Sync

Sync content from an upstream repository into the local repo, handling different directory structures via path mapping.

Bundled Scripts

The skill includes sync scripts in scripts/:

• scripts/sync-upstream.ts — 5-phase sync orchestrator
• scripts/sync-utils.ts — path mapping, git helpers, integrity checks

Workflow

Step 0: Deploy Scripts to Project

Copy the bundled scripts to the project's scripts/ directory (skip if already present and up-to-date):

mkdir -p <PROJECT_ROOT>/scripts
cp <SKILL_DIR>/scripts/sync-upstream.ts <PROJECT_ROOT>/scripts/
cp <SKILL_DIR>/scripts/sync-utils.ts <PROJECT_ROOT>/scripts/

The project needs tsx in devDependencies. If missing: npm i -D tsx.

Step 1: Configure Upstream Remote

git remote get-url upstream 2>/dev/null || git remote add upstream <UPSTREAM_URL>
git fetch upstream

Step 2: Generate Path Mappings

Analyze both repos and create <PROJECT_ROOT>/.upstream-mapping.json:

# List upstream root structure
git ls-tree --name-only upstream/main

# List local content directories
ls <PROJECT_ROOT>/docs/

Generate mapping config:

{
  "_comment": "Upstream -> local path mapping",
  "exclude": ["README.md"],
  "mappings": [
    { "upstream": "01_intro/", "local": "docs/zh/01_intro/" },
    { "upstream": "02_fundamentals/", "local": "docs/zh/02_fundamentals/" },
    { "upstream": "_images/", "local": "docs/zh/_images/" },
    { "upstream": "SUMMARY.md", "local": "SUMMARY.md" }
  ]
}

Mapping rules:

• Directory: upstream path ends with /, local path ends with /
• File: exact paths, no trailing /
• exclude: upstream paths skipped entirely (never synced)
• First match wins

Preserve local rules (preserveLocal):

Use preserveLocal to specify line patterns that should be preserved from the local version during sync. This is useful when you've made local customizations (like replacing GitHub URLs with local paths) that should not be overwritten by upstream changes.

{
  "preserveLocal": [
    {
      "description": "Keep local code file links instead of GitHub URLs",
      "pattern": "\\[code/[^\\]]+\\]\\(code/",
      "glob": "*.md"
    },
    {
      "description": "Keep local relative doc links instead of GitHub absolute paths",
      "pattern": "\\]\\(\\.\\./chapter",
      "glob": "*.md"
    }
  ]
}

Preserve rule fields:

• description: Human-readable description of what this rule preserves
• pattern: Regex pattern to match lines that should be preserved from local version
• glob (optional): File pattern to limit which files this rule applies to (e.g., *.md)

How it works:

• During auto-merge, if a local file exists and preserve rules are configured
• Each line in the upstream content is checked against the preserve patterns
• If a line matches a pattern AND the local file has a different version of that line
• The local version is preserved instead of being overwritten by upstream

Step 3: Initialize State

Create <PROJECT_ROOT>/sync-state.json with the starting commit:

{
  "upstream_remote": "upstream",
  "upstream_branch": "main",
  "last_synced_commit": "<STARTING_SHA>",
  "last_sync_time": null,
  "sync_history": []
}

Step 4: Run Sync

cd <PROJECT_ROOT>
node --import tsx scripts/sync-upstream.ts --from <COMMIT_SHA>

Step 5: Review & Merge

1. Check terminal output for auto-merged, manual-review, and unmapped files
2. Review sync-conflict-report.md for details (includes suggested mapping JSON for unmapped paths)
3. If unmapped new upstream dirs appear: add suggested entries to .upstream-mapping.json, then re-run sync
4. Handle manual-review files (large diffs >= 30 lines, deleted files, SUMMARY.md):

   git show upstream/main:<upstream-path>   # view upstream version
   # edit local file, then stage + commit
   

5. Merge into main:

   git checkout main
   git merge sync/<timestamp>
   git branch -d sync/<timestamp>
   

Auto vs Manual Classification

Unmapped New Paths

Upstream may add new top-level directories that are not in .upstream-mapping.json. These are not silently ignored:

1. Phase 2 prints each unmapped path with kind (added / modified / …)
2. Suggests mapping entries inferred from existing local prefix (e.g. docs/zh/)
3. Writes them into sync-conflict-report.md under Unmapped Upstream Paths
4. After you add the mappings, re-run sync to pull those files

Integrity Verification

Post-merge checks:

• Directory targets exist and contain non-empty .md files
• File targets exist and are non-empty
• New files written from upstream are listed
• No files lost compared to pre-merge snapshot (deleted upstream files are not auto-removed)

Sync Phases

1. Prepare — read state, fetch upstream, detect new commits
2. Analyse — diff files, apply path mapping + exclude, classify auto vs manual, warn on unmapped
3. Backup — create backup/pre-sync-<ts> and sync/<ts> branches
4. Merge — file-by-file via git show <commit>:<path>, write/add mapped local paths, stage dirs + files + applied paths
5. Report — update state, print summary (incl. unmapped + suggestions), save report