# Manual Sync > Fetch new LeetCode comp posts and manually parse/normalize them (replacing the LLM pipeline). Invoke with /manual-sync. Use when this capability is needed. - Skill: `tomevault-io/manual-sync` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add tomevault-io/manual-sync` - Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/manual-sync/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: tomevault-io (https://skillmd.com/u/tomevault-io) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/tomevault-io/manual-sync --- # LeetComp Manual Sync You replace the LLM steps of `leetcomp-sync`. The fetch script grabs raw posts, then YOU parse, normalize, and rebuild final data. ## Step 1: Fetch Raw Posts ```bash cd {{workspace}} && uv run python -c " import asyncio from leetcomp.utils import last_fetched_info from leetcomp.fetch import fetch_posts_in_bulk from leetcomp import POSTS_FILE till_id, till_ts = last_fetched_info(POSTS_FILE) print(f'OLD_FIRST_ID={till_id}') asyncio.run(fetch_posts_in_bulk(till_id=till_id, till_timestamp=till_ts)) print('FETCH_DONE') " ``` Note the `OLD_FIRST_ID` printed. All lines in `data/posts.jsonl` before that id's line are new posts. If no new posts were fetched, stop and report that. ## Step 2: Parse New Posts Read the new posts from `data/posts.jsonl`. For each post: - **Skip** if downvotes > upvotes → `{"id": ..., "created_at": ..., "skip": true}` - **Skip** if not a compensation post → `{"id": ..., "created_at": ..., "skip": true}` - **Parse** India-based offers only (Lakhs/LPA/INR/₹ = India signal) Parsed record JSON line: ```json {"id": , "created_at": "", "skip": false, "compensation-post": true, "offer-type": "full-time", "company": "", "company-normalized": "", "role": "", "role-normalized": "", "yoe": , "base": , "total": , "total-calculated": , "location": "", "currency": "INR"} ``` Key rules: - LPA values: "30 lacs"→30, "₹10,00,000"→10, "1.5 Cr"→150 - `total` = user-stated total; `total-calculated` = annual compensation computed from recurring annual components only (base + annual bonus/target bonus + annualized stocks/equity), and only if no explicit total is stated - Exclude one-time or non-recurring items from `total-calculated`: joining/sign-on bonus, relocation bonus, retention bonus, PF, gratuity, retirals, benefits, reimbursements, clawback notes, etc. - RSU/stock grants: annualize over 4 years (industry standard) even if the vesting period isn't explicitly stated, as long as the grant value is given in currency (INR/$). If RSUs are in units only, look up the current stock price and USD/INR rate to convert to INR, then annualize over 4 years. - Always compute `total-calculated` when there are enough components to exceed just `base` alone, and no explicit `total` is stated. Do not leave it out just because one component (e.g. vesting period) uses a standard default. - Mark duplicate/reposted compensation posts as skip (same author, same comp details, different post ID) - If the user explicitly states a first-year/total compensation number, prefer `total` exactly as stated instead of recomputing - Omit missing fields entirely (never N/A) - One post can yield multiple parsed records - company-normalized: lowercase normalized ("Samsung R&D" → "samsung") - role-normalized: lowercase normalized key matching existing `data/role_map.json` conventions when possible; prefer specific normalized titles over overly generic ones when the role is explicitly stated (e.g. `application engineer l3` is better than `l3`) **Critical accuracy rules:** - Never infer unstated fields. Only include fields explicitly stated in the post or directly derivable from explicit compensation text - If role is not stated, omit `role` and `role-normalized` - If location for the offer is not stated, omit `location` even if the author mentions current city, preferred city, NCR, office preference, or a company office elsewhere - Do not invent a generic role like `sde`/`sse` just because the YOE/company suggests it - For multi-offer posts, keep each offer isolated; do not leak role/location/YOE/comp components from one offer into another unless clearly shared - Preserve raw `role` text from the post when present, and normalize separately in `role-normalized` **Mandatory verification before writing:** - Re-read every newly parsed record against the raw post text - Check that every non-derived field (`company`, `role`, `location`, `yoe`, explicit `total`) appears in the post text - Check that every derived field is traceable and formula-correct - Check `role-normalized` against existing `data/role_map.json` patterns before creating a new style of normalization - Check that `total-calculated` excludes one-time components unless the post explicitly gives a total, in which case use `total` - Only after this verification, prepend all new parsed lines to `data/parsed_posts.jsonl` (newest first) ## Step 3: Update Entity Maps For each of `company_map.json`, `role_map.json`, `location_map.json`: 1. Load existing map (keys are lowercase) 2. Collect new entity values from your parsed records (`company-normalized`, `role-normalized`, `location`) 3. For any value not already in the map, add: `"lowercase_variant": "Canonical Name"` 4. Save sorted by key **Canonical naming:** - Company: proper casing (Google, Amazon, JPMorgan Chase, PhonePe, IBM) - Role: UPPERCASE abbrevs (SDE, SSE, MLE), Title Case full names (Data Scientist). Preserve levels (SDE 2) - Location: official name (Bengaluru, Gurugram, Hyderabad) ## Step 4: Rebuild final_data.json ```bash cd {{workspace}} && uv run python -c "from leetcomp.transform import create_final_data; create_final_data()" ``` ## Step 5: Report Print summary: new posts fetched, parsed offers vs skipped, new mappings added per entity type, total final records. ```bash cd {{workspace}} && python3 -c "import json; print(f'Final records: {len(json.load(open(\"data/final_data.json\")))}')" ``` --- > Source: [0xku/leetcode-compensation](https://github.com/0xku/leetcode-compensation) — distributed by [TomeVault](https://tomevault.io).