# Apk Clues Extract > Extract vendor-specific GATT Service and Characteristic UUIDs from Android APK / XAPK files and emit a CLUES-formatted JSON (`CLUES_data_LLM_Android.json`) that conforms to `CLUES_schema.json`. Use when the user asks to 'scan APKs for UUIDs', 'extract Bluetooth UUIDs from Android apps', 'build CLUES data from APKs', or 'find GATT services in a folder of APKs'. Takes a path to either an individual APK/XAPK or a directory which it walks recursively. Pre-flights each APK with `aapt dump permissions` and skips any package that does not declare BLUETOOTH / BLUETOOTH_ADMIN / BLUETOOTH_SCAN / BLUETOOTH_CONNECT — never wastes time decompiling a non-BT app. Uses `jadx --no-res --no-debug-info` to decompile DEX, deletes the temp output after each file. Do NOT use for iOS apps, for non-(X)APK files (e.g. raw .dex outside an APK), or when the goal is to add UUIDs already known from public sources rather than discover them from an APK. - Skill: `xiamiya/apk-clues-extract` (Agent Skill) - Install (CLI): `npx skillmds@latest add xiamiya/apk-clues-extract` - Raw SKILL.md: https://api.skillmd.com/api/skills/xiamiya/apk-clues-extract/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: xiamiya (https://skillmd.com/u/xiamiya) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/xiamiya/apk-clues-extract --- # APK CLUES UUID Extractor ## What this skill does Given a path to a single Android `.apk` / `.xapk` file or a directory of them, this skill discovers every Bluetooth Low Energy GATT-related UUID referenced by the app's decompiled code, classifies each as **GATT Service** or **GATT Characteristic** from local code context, and emits an array of CLUES-format records to `CLUES_Schema/data/CLUES_data_LLM_Android_APK_search.json` (sibling of the hand-curated `CLUES_Schema/data/CLUES_data_human_verified.json`, both conforming to `CLUES_Schema/CLUES_schema.json`). For each APK it: 1. **Verifies the package requests Bluetooth.** Runs `aapt dump permissions` and looks for `android.permission.BLUETOOTH`, `BLUETOOTH_ADMIN`, `BLUETOOTH_SCAN`, or `BLUETOOTH_CONNECT`. If none are present, the APK is logged as "no BT permissions; skipped" and the skill moves to the next file *without* running jadx — decompiling a non-BT app is wasted time. 2. **Extracts `package_id`, `version_code`, `version_name`** from `aapt dump badging`. These populate the `android_info_array` for every UUID found in the app. 3. **Decompiles to a temp dir.** `jadx --no-res --no-debug-info -d ` (jadx 1.5+ understands `.xapk` natively, so XAPKs do not need to be unzipped first). The `--no-res` flag skips XML resource decoding — UUIDs live in the code, not in the AndroidManifest. 4. **Scans decompiled `.java` files for UUID patterns** — regex finds the candidate sites, the script then *reads* the code around each hit rather than relying on the regex alone (per GreyNoise's "regex will not dynamically emulate DEX bytecode" warning): - Raw 128-bit UUID string literals (`"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"`) — picks up `UUID.fromString("...")`, `ParcelUuid.fromString("...")`, log strings, JADX deobfuscation hints, etc. - For every literal, the script also captures the **LHS Java identifier** of the assignment (`public static final UUID LIBRE3_DATA_SERVICE = UUID.fromString(...)`) — this becomes the `UUID_name` after light cleanup (strip trailing `_UUID` / `Uuid`, drop Java `m`/`s` member prefixes). Minified jadx leftovers (`f7217p`, single-letter aliases) are rejected — those records get `UUID_name: "Unknown"` and a later APK that has the same UUID with a human-meaningful field name will overwrite it. - When JADX has moved the semantic label away from the literal, the scanner also recovers names from **lowered enum rows** (`WIFI_CONTROL = new TwinService("WIFI_CONTROL", …, "uuid")`), **nested sealed-object constructors** that pass a UUID temp into `super(...)`, **anonymous singleton-object fields** (`BulkTransfer = new e0() { … UUID.fromString("uuid") … }`), **constructor parameter roles** preserved in another class (`new BluetoothServiceConfiguration(uuid1, uuid2, uuid3, …)` → `notifyCharacteristicUUID` from the callee signature), and **assigned wrapper constructors** where the target field is the best surviving label (`writeCharacteristic = new BleCharacteristic(serviceId, id)`). - Then the script performs a **read-the-code usage trace**: for every `(field, uuid)` pair, it searches the whole file for `getService(field)` / `getCharacteristic(field)` / `setCharacteristicNotification(...)` / etc. A hit here is **authoritative** — those calls are how the Android API itself decides what a UUID is, so they beat any naming-convention heuristic. 5. **Decides at the file level whether a `.java` file is BLE-relevant.** Two ways a file qualifies (either is enough): - **Hard signal:** the file references one of `android.bluetooth.BluetoothGatt*` / `android.bluetooth.le.ScanFilter` / `android.os.ParcelUuid` anywhere. - **Constants-class fallback:** the file contains **≥3 `UUID.fromString("128-bit-literal")` calls** with no BLE imports. This catches "pure constants" classes like Abbott Libre3's `DCSGKSConstants.java` (15 UUIDs, zero `BluetoothGatt*` refs — the BLE-aware code lives in *other* classes that import this one) or Harman's `e2/d.java`. A non-BLE config class with a couple of analytics/telemetry UUIDs in URLs (Adobe Edge, Azure App Insights, OneTrust) won't trigger the fallback because those UUIDs sit inside string concatenations, not `UUID.fromString` calls. A file that hits neither signal is treated as "probably not BLE" — UUIDs in it only survive if a `BluetoothGatt*` identifier sits within ±8 lines of the literal. 6. **Filters Bluetooth-SIG standard UUIDs and already-published member UUIDs and curated CLUES UUIDs.** Any UUID matching the SIG base `0000XXXX-0000-1000-8000-00805F9B34FB` is collapsed to its 16-bit form. Then: - UUID16s in the SIG-allocated **standard** ranges (0x0000 / 0x0001–0x012F SDP protocols / 0x1000–0x12FF SDP service classes / 0x1800–0x18FF GATT services / 0x2800–0x29FF declarations & descriptors / 0x2A00–0x2BFF characteristics) are dropped — they are not vendor-specific. - UUID16s **already listed in `~/Blue2thprinting/Analysis/public/assigned_numbers/uuids/member_uuids.yaml`** (the SIG's public vendor-allocated UUID16 list) are also dropped from the output — they're already public, so there's no value re-recording them in CLUES_data_LLM. The hits *are* surfaced internally for the company-inference step below. - **UUIDs already in `~/Blue2thprinting/Analysis/CLUES_Schema/data/CLUES_data_human_verified.json`** (the hand-curated, human-verified canonical CLUES file — renamed from `CLUES_data.json` when data files moved under `CLUES_Schema/data/`) are dropped from the LLM output. Those entries are the project's highest-trust source; re-emitting an auto-extracted version risks a curator accepting a regression on merge. Configurable via `--curated-clues PATH` (pass an empty string to disable the skip-list). The startup log reports how many curated UUIDs were loaded. - UUID16s outside all three filters (typically vendor-internal IDs like `0xE104` or `0x8A21`) and all UUID128s not in curated CLUES are kept. 7. **Classifies each remaining UUID as Service vs Characteristic.** Cascading decision: - **Definitive usage trace (read-the-code):** if the captured field name is passed to `getService(...)` anywhere in the file → Service. Passed to `getCharacteristic(...)` / `writeCharacteristic(...)` / `setCharacteristicNotification(...)` / similar → Characteristic. This is the same authoritative sink set BLEScope uses for its value-set analysis. - **Anchor line:** if no definitive usage trace, look at the UUID's own source line for naming-convention hints (`_SERVICE` / `_CHAR_` / `BluetoothGattService` / `BluetoothGattCharacteristic`). - **Window fallback:** if the anchor line doesn't disambiguate, look at the surrounding ±8 lines. - **File-level last resort:** if everything is silent (minified field names, e.g. `f15558a`), classify based on whether the file overall mentions `BluetoothGattService`, `BluetoothGattCharacteristic`, or both. Files using only `BluetoothGattCharacteristic` default to Characteristic; otherwise default to Service. 8. **Infers the `company` field per APK.** Cascading resolution, in order of preference (always preferring SIG-official member names when they're a plausible match): - **`KNOWN_SDK_UUIDS` table hit (highest priority).** If the UUID is in the curated Phase-4 cache of third-party-SDK UUIDs, the table's entry wins — beats every other inference. This is how the skill auto-attributes well-known shared SDKs (Kubi/Zoom, Nordic UART, TI OAD, Microchip ISSC, HM-10, Ojmar/OCS-Smart) without per-host-app guessing. - **Single-APK third-party-SDK detection.** Each UUID literal is annotated with the Java `package …;` declaration of the `.java` file it was found in. If that declaring-package's tokens have **zero overlap** with the host APK's `package_id`, the UUID is flagged as `"Third-party SDK code (declared in Java package — distinct from host ; Phase-4 identification needed)"`. This catches SDK leakage on a **single-APK basis** without waiting for ≥3 host apps to share the UUID (which is the Phase-4 cluster threshold). Example caught on the May 2026 corpus: `5301` (declared in `com.stidmobileid.developmentkit`) and `00009800-…-00177a000000` (declared in `com.assaabloy.mobilekeys.api.ble`) were flagged on the very first Sesame Smart Spaces scan, even though only one host APK was processed. - **`member_uuids.yaml` with token-overlap gate.** If the APK exposes a UUID16 that's listed in `member_uuids.yaml`, *and* that member name shares at least one ≥3-letter token with the package_id, the SIG-official member name wins. So `com.abbott.lingo.wellness` + UUID16 `0xFDE3` → "Abbott Diabetes Care" (token `abbott` matches). The overlap gate is essential: many apps reference UUIDs from shared SDKs (Amazon, Bose, Sony, Qualcomm, Taobao, …) and we **must not** mis-tag a fitness app as "Taobao" just because it links a Taobao-published UUID16. No overlap → fall through to the package heuristic. - **Package-name heuristic.** Hard-coded `KNOWN_VENDOR_PACKAGES` map covers ~50 common vendors (Abbott, Dexcom, Harman, Garmin, Withings, Nordic, etc.); fallback is the first non-TLD-like component capitalized (`com.proctur.app222423` → "Proctur (inferred from package id)"; `branded.com.publicansmanhasset` → "Publicansmanhasset (inferred from package id)"). TLD-like and whitelabel tokens (`com`, `co`, `branded`, `app`, `android`, …) are skipped. - **Last resort:** `"Unknown"` only when all sources are silent. 9. **Merges across APKs with semantic upgrades.** When the same UUID is seen in multiple APKs, the record's `UUID_name` is replaced whenever a new APK has a higher-scoring identifier (e.g., `LED_COLOR_UUID` beats `f7217p` beats `Unknown`). Same goes for `company` — a non-`Unknown` value replaces an `Unknown`. Each new APK appends a row to `android_info_array` and an entry to `evidence_array`, so the full provenance is preserved. **Evidence deduplication (`dedupe_evidence_array`).** Two evidence entries are treated as duplicates and collapsed into a single survivor when *any* of these holds: - They share the same `URL` (case-insensitive, after strip) — one URL evidence per Phase-3 mapping, no matter how many sibling records merged in. - Both descriptions begin with `Found in decompiled DEX of v (versionCode )` with identical `(package_id, version_name, version_code)`. This is the user-reported pattern where a single APK has the same UUID referenced from multiple Java files — Phase 1 would emit one evidence entry per file (each with a different `Classification source:` or Java field name), but the underlying fact ("this UUID is in this APK build") is one. The `android_info_array` already carries the unique-per-(pkg, version_code) view of the same fact, so multiple evidence rows are pure noise. - The full `(description, submitter)` tuple matches (and neither has a URL). When two entries collide, the **more informative survivor** wins — preferring entries that captured a Java field name, an authoritative `usage:` classification source, or a third-party-SDK declaring-package annotation. Order is preserved: the survivor lives at the position of the first occurrence, so curator-edited entries stay at the top. Dedup runs in three places, so re-running phases or merging on top of existing data is idempotent: - Inside `merge_records` after each per-UUID extend. - As a final pass in `save_output` before writing. - Inside `apply_apk_url_evidence.py` and `apply_uuid_name_resolution.py` after their append step, so re-running those scripts never restacks the same URL or Phase-5 note. 7. **Attempts to pair Characteristics with their parent Service** when the Service UUID appears within the same `.java` file. If no parent can be found locally, `parent_UUID` is omitted (allowed by the schema). 8. **Emits one CLUES record per unique UUID.** Each record carries an `android_info_array` entry tagging which package(s) it was found in. When the same UUID turns up in multiple APKs, the entries are merged: the UUID gets a single record with multiple `android_info_array` items. 9. **Deletes the temp jadx output dir.** Every APK's decompilation is removed before the next one starts. The persistent output is only the JSON file. ## Prerequisites — confirm before starting - **`jadx` ≥ 1.5** must be installed. `brew install jadx`. Earlier versions don't understand `.xapk` and would need a manual `unzip → base.apk` step. - **`aapt`** from the Android build-tools must be reachable. `brew install android-commandlinetools` installs it under `/opt/homebrew/share/android-commandlinetools/build-tools//aapt`. The script auto-discovers the newest installed `aapt`. - **`python3`** (any 3.9+). - **Disk space for jadx output.** A single decompilation can produce 1–4× the APK size in `.java` files. The script writes to `$TMPDIR` (or `/tmp` if unset). Make sure the tmpfs has at least 4 GB free before pointing the skill at a folder of fat APKs (e.g. Dexcom Stelo 170 MB → ~600 MB of Java). ## How to invoke ### User-facing (Claude Code slash-command form) End users trigger the skill through the standard slash-command. The arguments after the slash are interpreted as one or more paths (APK files, XAPK files, or directories), optionally followed by any of the Python CLI flags listed below: ``` /apk-clues-extract /Volumes/2TB_ExFAT/__BLUUID_APKS/0002 \ /Volumes/2TB_ExFAT/__BLUUID_APKS/0003 \ "/Users/user/Downloads/DexcomStelo/Stelo by Dexcom_2.1.0.2972_APKPure.xapk" ``` **Slash-arg pass-through.** The harness gives the agent the entire free-form argument string, so any flag of `extract_clues.py` (`--parallel`, `--output`, `--session-output`, `--no-resume`, `--submitter`, …) is valid directly on the slash command. The agent appends them verbatim to the Python invocation: ``` /apk-clues-extract /Volumes/2TB_ExFAT/__BLUUID_APKS/0002 --parallel 4 --output /tmp/clues.json --session-output /tmp/just-the-new-ones.json ``` **Append behavior (important).** With no flag changes, the skill **merges into the existing `--output` file** — it does NOT start fresh. New UUIDs append; UUIDs already in the file gain new `android_info_array` / `evidence_array` entries from this run. The checkpoint sidecar (`.processed.json`) skips APKs whose SHA-1 is already recorded. To start fresh: either point `--output` at a new path, delete the file beforehand, or pass `--no-resume` (which ignores the checkpoint but still merges into the existing JSON — usually not what you want for "start fresh"). **Capturing just-this-run changes.** When you want a separate file listing only the UUIDs that *this invocation* added to the corpus (so you can review or share what one APK / one folder contributed without diffing the whole file by hand), pass `--session-output`: ``` /apk-clues-extract /Volumes/2TB_ExFAT/__BLUUID_APKS/0004 \ --output CLUES_data_LLM_Android_APK_search.json \ --session-output /tmp/run-0004-new-uuids.json ``` After the run, `/tmp/run-0004-new-uuids.json` contains only records whose UUID was *not* present in the `--output` file at the start of the run. UUIDs that this run merely added new package observations to are NOT included — their full state is in `--output`, and they pre-date the session. ### What the agent runs internally When the skill is invoked, the agent translates the user's paths into one call to `extract_clues.py`. The full command-line interface: ```bash python3 /Users/user/.claude/skills/apk-clues-extract/scripts/extract_clues.py \ [ ...] \ [--output CLUES_data_LLM_Android_APK_search.json] \ [--session-output FILE.json] # write only UUIDs not present in --output at start of run [--semantic-upgrades-output FILE.json] # write Unknown->named / Unknown->company upgrades [--submitter "Claude (Opus 4.7)"] \ [--parallel N] # default 1 (serial). N=4 is a good fit for an 8-core box. [--progress-interval 100] # checkpoint progress + Unknown-name count every N APKs [--no-resume] # ignore checkpoint and re-process every APK from scratch [--search-unknown] # re-process APKs that contributed UUID_name=='Unknown', then run deep Phase-5 RE [--phase5-deep-re] # run only the deep Phase-5 RE pass on existing Unknown records [--phase5-deep-re-output FILE.json] # optional mapping report from the deep Phase-5 RE pass [--keep-tmp] # only for debugging; default is to wipe the tempdir after each file ``` `` can be any mix of APK files, XAPK files, and directories — directories are walked recursively. Duplicates (same APK reached via multiple paths) are skipped via the checkpoint. **Worked example** — the agent's bash translation of the slash-command above: ```bash python3 /Users/user/.claude/skills/apk-clues-extract/scripts/extract_clues.py \ /Volumes/2TB_ExFAT/__BLUUID_APKS/0002 \ /Volumes/2TB_ExFAT/__BLUUID_APKS/0003 \ "/Users/user/Downloads/DexcomStelo/Stelo by Dexcom_2.1.0.2972_APKPure.xapk" \ --output /Users/user/Blue2thprinting/Analysis/CLUES_Schema/data/CLUES_data_LLM_Android_APK_search.json \ --parallel 4 ``` Equivalent serial form (each call accumulates into the same output file via the merge logic — useful when inputs arrive incrementally, e.g. the user pulls one APK at a time from a phone): ```bash OUT=data/CLUES_data_LLM_Android_APK_search.json # path is relative to CLUES_Schema/ python3 .../extract_clues.py /Volumes/2TB_ExFAT/__BLUUID_APKS/0002 --output "$OUT" --parallel 4 python3 .../extract_clues.py /Volumes/2TB_ExFAT/__BLUUID_APKS/0003 --output "$OUT" --parallel 4 python3 .../extract_clues.py "/Users/user/Downloads/DexcomStelo/Stelo by Dexcom_2.1.0.2972_APKPure.xapk" --output "$OUT" ``` The single-invocation form is preferred when the inputs are known upfront — the parallel pool stays warm across all paths. Both produce identical output because of the checkpoint + merge. **Default behavior: all six phases + a final post-flight sort run.** After Phase 1, the agent **automatically** runs Phase 2 (Play Store company enrichment), Phase 3 (re-download URL evidence), Phase 4 (shared-SDK identification), Phase 5 (re-investigation of UUIDs still named "Unknown" in this session), Phase 6 (extract lessons learned from Phases 2–5 and patch `extract_clues.py` so the next first-pass benefits), and a final post-flight invocation of `CLUES_Schema/scripts/SortCLUES.py` that re-sorts every data file in `CLUES_Schema/data/` into the project's canonical (company, UUID_purpose, UUID) order so git diffs stay reviewable. These phases are agent-driven loops, not separate slash commands. The user gets a fully-resolved CLUES file *and* a smarter Phase-1 extractor from one invocation rather than having to remember 7 separate commands. To opt out of one or more phases, pass any of these flags on the slash command and the agent will skip them: - `--phase1-only` — stop after Phase 1 (raw, offline extraction; also skips the post-flight sort) - `--no-phase-2` — skip Play Store company enrichment - `--no-phase-3` — skip URL evidence - `--no-phase-4` — skip shared-SDK cluster identification - `--no-phase-5` — skip re-investigation of Unknown UUID names - `--no-phase-6` — skip self-improvement (do not patch `extract_clues.py`) - `--no-sort` — skip the post-flight `SortCLUES.py` re-sort (use when sibling data files have uncommitted hand-edits the user wants preserved) Reasons to opt out: - **Reproducibility-sensitive runs:** Phase 1 is the only deterministic phase (network-independent). If you need a result that's bit-stable across re-runs, pass `--phase1-only`. - **1000+ APK corpora:** the Phase-4 loop is per-cluster O(decompile + WebSearch); 50 clusters can add 1-3 hours. If you'll do that curation later, pass `--no-phase-4`. - **Read-only audit:** if the user is reviewing the skill without authorizing source edits, pass `--no-phase-6` so `extract_clues.py` is untouched. - **Offline environment:** `--no-phase-2 --no-phase-3 --no-phase-4 --no-phase-5 --no-phase-6` is equivalent to `--phase1-only`. The agent's flag-handling: it parses the slash-command args, peels off `--phase1-only / --no-phase-N`, passes the remaining args to `extract_clues.py`, then decides phase-by-phase whether to invoke the helpers below. **Parallelism (`--parallel N`).** Each worker runs the per-APK pipeline (aapt + jadx + scan + classify) in its own process. The main thread serializes merge + output writes. jadx is internally multi-threaded (9 threads by default), so the practical ceiling is `min(cpu_count() // 2, len(targets))` — beyond that you'll thrash. On a 1,000-APK corpus this brings wall time from ~10h down to ~2.5h on an 8-core machine. **Resumability (checkpoint).** A sidecar file `.processed.json` records the SHA-1 + status of every APK the driver has seen. Re-running the same command will skip APKs already recorded with `status: ok:*`. To force a full re-scan, pass `--no-resume`. The checkpoint also survives Ctrl-C: at most the in-progress APKs are lost. SHA-1 is over the first 4 MiB of the APK + its size — fast and sufficient to detect a republished build. **Targeted re-extraction + deep Phase-5 RE (`--search-unknown`).** Re-runs Phase 1 against only the APKs that originally yielded records with `UUID_name: "Unknown"` (or empty/missing) in the current `--output`, then runs the deep Phase-5 reverse-engineering pass over those same Unknown UUIDs. The target set is derived from each Unknown record's `android_info_array[].package_path` field, deduped to unique APK paths. For each derived APK, the checkpoint's `status: ok:*` is *ignored* (so the APK is re-processed even though it's already in the sidecar), but the rest of the checkpoint is preserved untouched — non-target APKs aren't re-scanned and aren't dropped from the sidecar. This is the right tool when Phase 1's classifier has improved or when the remaining names need behavioral recovery from decompiled code. Positional path args act as a filter when given — only Unknown-record APKs whose absolute path matches or sits under one of those paths are processed. Paths that no longer exist on disk are skipped with a warning. The flag makes the positional path argument optional. Example: ```bash # retry every Unknown-yielding APK in the corpus python3 .../extract_clues.py --search-unknown --output data/CLUES_data_LLM_Android_APK_search.json --parallel 4 # retry only Unknown-yielding APKs that live under one specific folder python3 .../extract_clues.py /Volumes/2TB_ExFAT/__BLUUID_APKS/0998 --search-unknown --output data/CLUES_data_LLM_Android_APK_search.json --parallel 4 ``` Idempotent + convergent: each invocation re-derives the Unknown set from the current `--output`, so successive runs queue progressively fewer APKs as names get resolved. In `--search-unknown` mode, `--session-output` can legitimately contain zero records because no UUIDs were newly discovered; use `--semantic-upgrades-output /tmp/upgrades.json` when you want the audit artifact that matters for this mode: existing records whose `UUID_name` or `company` improved. If `--search-unknown` and `--session-output` are both set but `--semantic-upgrades-output` is omitted, the extractor automatically writes a sibling `*_semantic_upgrades.json` report. Long runs emit built-in checkpoint progress every `--progress-interval` APKs (default 100), including the remaining `UUID_name: "Unknown"` count. The deep Phase-5 portion of `--search-unknown` decompiles each target APK with `jadx --show-bad-code` in addition to the normal flags, because JADX often hides the callback/listener body that contains the useful BLE dataflow unless that flag is enabled. It then performs a targeted local reverse-engineering pass: - Captures loose UUID assignments, including instance fields such as `this.avion_char_out_uuid = "..."`, not only `UUID.fromString(...)`. - Traces obfuscated UUID constants through service-discovery comparisons into `BluetoothGattCharacteristic` alias fields. - Classifies service UUIDs from `BluetoothGattService.getUuid().compareTo(...)` and `getService(...)`, correcting Phase-1 file-level defaults when needed. - Reads alias behavior: write helpers, `writeCharacteristic`, `readCharacteristic`, `setCharacteristicNotification`, descriptor notification enables, and `onCharacteristicChanged`. - Uses nearby UI/log/action strings (`BluConsole`, `EXTRAS_CONSOLE_TEXT`, send-button command paths, proxy PDU flow) to synthesize conservative names such as `ARUBA_BLUCONSOLE_INPUT_CHARACTERISTIC` or to preserve exact recovered field names such as `avion_char_out_uuid`. Use `--phase5-deep-re` when you want **only** this deep Phase-5 pass against the current `--output` without re-running normal extraction first. It uses the same Unknown-record target derivation and accepts the same optional positional path filter: ```bash python3 .../extract_clues.py /tmp/two-apks \ --output data/CLUES_data_LLM_Android_APK_search.json \ --phase5-deep-re \ --phase5-deep-re-output /tmp/deep_phase5_mapping.json \ --semantic-upgrades-output /tmp/deep_phase5_upgrades.json ``` - The default `--output` is `data/CLUES_data_LLM_Android_APK_search.json` (relative to whatever directory the script is run from — typically `CLUES_Schema/`, which is what the agent always passes explicitly). Pre-existing entries in that file are **loaded** at start so re-running on more APKs accumulates results. - `--submitter` defaults to the active LLM model name (`Claude (Opus 4.7)` for this skill). Override only if running under a different model. - `--keep-tmp` retains the per-APK jadx output (under `$TMPDIR/apk_clues__/`) for manual inspection. Off by default — production runs should delete to save disk. ## Output format The output JSON conforms to `CLUES_schema.json` and is a flat top-level array of records. A typical record looks like: ```json { "UUID": "f8083535-849e-531c-c594-30f1f86a4ea5", "company": "Unknown", "UUID_name": "Unknown", "UUID_purpose": "Custom GATT Service observed in Android app com.dexcom.stelo (Stelo by Dexcom).", "UUID_usage_array": [ "GATT Service" ], "evidence_array": [ { "description": "Found in decompiled DEX of com.dexcom.stelo v2.1.0.2972 (XAPK 'Stelo by Dexcom_2.1.0.2972_APKPure.xapk'). Referenced in BluetoothGattService construction.", "submitter": "Claude (Opus 4.7)" } ], "android_info_array": [ { "package_id": "com.dexcom.stelo", "version_code": 2972, "version_name": "2.1.0.2972", "package_path": "/Users/user/Downloads/DexcomStelo/Stelo by Dexcom_2.1.0.2972_APKPure.xapk", "description": "Detected by apk-clues-extract scan; usage 'GATT Service'." } ] } ``` Characteristics carry `parent_UUID` when the parent service was identifiable in the same Java file: ```json { "UUID": "f8083536-849e-531c-c594-30f1f86a4ea5", "company": "Unknown", "UUID_name": "Unknown", "UUID_purpose": "GATT Characteristic under f8083535-849e-531c-c594-30f1f86a4ea5 in com.dexcom.stelo.", "UUID_usage_array": [ "GATT Characteristic" ], "parent_UUID": "f8083535-849e-531c-c594-30f1f86a4ea5", "evidence_array": [ ... ], "android_info_array": [ ... ] } ``` Because the extractor cannot infer the marketing/legal company name from an APK alone (only the package_id), every record starts with `company: "Unknown"` and `UUID_name: "Unknown"`. A human reviewer (or a separate LLM pass) is expected to fill those in later by cross-referencing the `package_id` against the developer's public docs / store listing — that step is intentionally out of scope. ## Storage layout — single file or 16 hex-bucket shards Once `data/CLUES_data_LLM_Android_APK_search.json` grows past ~100 MB it stops being a useful single file: git diffs become unreviewable, editors hang loading it, and GitHub rejects pushes that contain it (100 MB hard cap, no LFS). The skill solves this by sharding the output across 16 files named after the first hex character of each record's `UUID`: ``` data/CLUES_data_LLM_Android_APK_search_0.json # UUIDs starting with "0" data/CLUES_data_LLM_Android_APK_search_1.json # UUIDs starting with "1" … data/CLUES_data_LLM_Android_APK_search_f.json # UUIDs starting with "f" ``` The split is **storage only** — the global `(company, UUID_purpose, UUID)` sort order is preserved across the combined dataset; shards just hold contiguous slices of that order. Whether a given file is sharded or single-file is decided **exclusively by `CLUES_Schema/scripts/SortCLUES.py`** (via its `SPLIT_FILES` allowlist). Every script in this skill (`extract_clues.py`, all Phase 2–6 helpers) calls `clues_io.load_clues(path)` / `clues_io.save_clues(data, path)` — those helpers transparently read either form and **preserve whatever layout they find on disk** when writing back. No script in this skill ever decides to split or un-split on its own. Implications: - **For the agent.** Always pass the *unsplit base path* (`data/CLUES_data_LLM_Android_APK_search.json`) to every CLI in this skill, regardless of which form is actually on disk. The loaders DTRT. Never hand-edit individual shards — use the helper scripts or write your edits through `save_clues`, otherwise the next post-flight sort will reorder records across shard boundaries and your hand-edits may end up in a different shard than expected. - **For schema validation.** `check-jsonschema` doesn't transparently combine shards. Glob the file pattern when the output is split: `data/CLUES_data_LLM_Android_APK_search_*.json` (see the [Validation](#validation) section below). The pattern works for either form because globs match the single-file basename too when no shards exist. - **For child characteristics whose parent service has a UUID starting with a different hex letter.** The child still carries `parent_UUID`, but the parent's record lives in a different shard. This is intentional — the alternative (moving children to follow parents) would defeat the by-UUID bucketing and make `load_clues` non-trivial. Consumers that want parent/child pairing should follow `parent_UUID` after loading the combined dataset. - **For Phase 6 self-improvement.** `KNOWN_NON_BLE_UUIDS` and `KNOWN_SDK_JAVA_PACKAGES` updates live in `scripts/extract_clues.py`, not in the data files, so the split doesn't affect Phase 6 at all. ## Phase 2 — web-enrichment of "(inferred from package id)" records The static extractor (Phase 1) produces the strongest `company` value it can without a network. When neither `KNOWN_VENDOR_PACKAGES` nor a `member_uuids.yaml` overlap fires, the field falls back to a capitalized package component plus the marker `(inferred from package id)`. **In-APK strings are almost never enough to identify the publisher** — they describe the *product* or the *device family*, not the *developer account*. Example: `com.tntkhang.gtswatchface`. The APK contains zero strings mentioning the publisher; the closest in-APK hint is `amazfitwf-46c6d.firebaseio.com` (just says "Amazfit watch face" — the product). Only a web search of the package id reliably resolves the publisher (it returns Play Store / AppBrain / APKPure listings consistently crediting "SmartWatchCenter"). Phase 2 is a Claude Code agent loop that automates this lookup. The two helper scripts in `scripts/` make it a 3-step recipe — any agent with `WebSearch` plus the standard read/write/bash tools can run it without further code: ### Step 2.1 — list which packages still need enrichment ```bash python3 /Users/user/.claude/skills/apk-clues-extract/scripts/list_inferred_companies.py \ /path/to/CLUES_data_LLM_Android_APK_search.json ``` Stdout is a JSON document with two keys: `needs_enrichment` (a list of `{package_id, current_company, record_count}` for packages still tagged `(inferred from package id)`) and `already_resolved` (a list of package_ids that already have a curated company). Records the agent should focus on are the ones in `needs_enrichment`, sorted by `record_count` so high-impact packages go first. ### Step 2.2 — bulk-scrape the Play Store first For most public apps, the publisher is in plain HTML on `play.google.com/store/apps/details?id=`. There's a helper that scrapes that page (`` / JSON-LD `"author":{...,"name":"..."}` / `/store/apps/dev?id=...` patterns) for many package_ids in one shot: ```bash python3 /Users/user/.claude/skills/apk-clues-extract/scripts/list_inferred_companies.py CLUES.json \ | jq -r '.needs_enrichment[].package_id' \ | python3 /Users/user/.claude/skills/apk-clues-extract/scripts/scrape_play_store_companies.py \ > /tmp/scrape_mapping.json ``` Stdout is a flat `{package_id: company_name}` object — feed it straight to `apply_company_enrichment.py` (Step 2.3). Stderr reports per-package status (`ok` / `unreachable` / `not_listed` / `no_author_field`). For packages the scraper couldn't resolve (typically: removed from Play Store, region-locked, never published publicly), fall back to a free-form WebSearch per the next paragraph. ### Step 2.2b — web-search any package_ids the scraper couldn't resolve For each `package_id` in `needs_enrichment`, the agent should call `WebSearch` with a query like: ``` developer publisher android app ``` and read the top 5–10 results. Reliable signals, in order of preference: 1. **Play Store listing** (`play.google.com/store/apps/details?id=`) — the "developer" attribution is authoritative. 2. **AppBrain developer page** (`appbrain.com/dev//`) — usually titles itself with the dev account name. 3. **APKPure / APKCombo / APKamp** — explicit "App by " lines in the page body. 4. **Microsoft 365 / Capterra / LinkedIn** — useful when the publisher rebranded (e.g. "Witco de MONBUILDING &CO" on Microsoft 365 disambiguates the rebrand history). Special cases the agent should handle: - **Rebrand history.** If the search results show a name change (`MonBuilding → Witco`, `St. Jude Medical → Abbott`, etc.), record the *current* legal name and the SIG-style `" (formerly )"` form so the value lines up with how Bluetooth SIG names absorbed members. Cross-check by also searching the old name to confirm it's the same legal entity. - **White-label / customer-specific packages.** `com.monbuilding.app.legende` is a white-labeled instance of the Witco platform for the *Legende* building — the BLE UUIDs belong to Witco, not to "Legende". Use the platform publisher, not the customer name. `branded.com.` is another common white-label pattern — `` is the *customer*, and the publisher you want is the white-label-platform vendor (search the white-label vendor's site for a customer list to confirm). - **Ambiguous results.** If two reasonable answers exist (e.g., the SDK vendor vs. the app publisher), prefer the *app publisher* — the UUIDs as-shipped in this APK are tied to the publisher's identity, not the SDK's. - **No clear answer.** Leave the package out of the mapping — don't guess. Records stay as `(inferred from package id)` for a future curator. The output is a single flat JSON object mapping resolved package_ids to confirmed company names, e.g.: ```json { "com.foo.bar": "Foo Corp", "com.legacy.app": "NewName (formerly OldName)", "branded.com.acme": "Acme Whitelabel Platforms" } ``` Save it to disk (any path; `/tmp/enrichment.json` is fine). > **Company-field rule.** Each value must be the **bare company name** — nothing else. Do NOT append country, product line, app name, what BLE is used for, acquisition history, or any other context inside the value. Bad: `"Canon Inc. (Japan) — Camera Connect app; BLE for EOS / PowerShot cameras"`. Good: `"Canon Inc."`. Bad: `"Anki (now Digital Dream Labs)"`. Good: `"Anki"` (use the historically-attributed name; acquisition notes go elsewhere). The lone exceptions are: (a) the `(formerly OldName)` rebrand suffix, which SIG-style member entries also use, and (b) acronyms that are part of the formal name like `(IBV)`. `apply_company_enrichment.py` runs the value through `company_sanitizer.sanitize_for_write` and will auto-strip any other parenthetical / em-dash commentary — appending the stripped text to the record's `evidence_array` and emitting a stderr `[sanitize]` warning — so the data ends up clean either way. The warning means "put it in evidence next time, not in the company field". ### Step 2.3 — apply the mapping ```bash python3 /Users/user/.claude/skills/apk-clues-extract/scripts/apply_company_enrichment.py \ /path/to/CLUES_data_LLM_Android_APK_search.json \ /tmp/enrichment.json ``` This script: - Replaces `company` on every record whose `android_info_array` references a mapped package — but **only** when the existing value is the `(inferred from package id)` placeholder or `Unknown`. A previously curated value is never overwritten. - Re-sorts the JSON by UUID for stable diffs and writes it back atomically. - Prints to stdout a suggested `KNOWN_VENDOR_PACKAGES` diff (one `"": "",` line per resolved package), keyed by the same first-non-TLD-like component the Phase-1 extractor uses for lookup. The agent should then `Edit` `scripts/extract_clues.py` to paste those lines into the dict — that way every future Phase-1 run hits the table directly and the same package never needs enrichment again. This is the **preferred** outcome over patching records ad hoc; it's reusable and benefits the entire CLUES community. After applying, re-validate the schema and you're done: ```bash cd /Users/user/Blue2thprinting/Analysis/CLUES_Schema/ source venv/bin/activate check-jsonschema --base-uri ./CLUES_schema.json \ --schemafile ./CLUES_schema.json ./data/CLUES_data_LLM_Android_APK_search.json ``` ### Opt-out conditions (Phase 2) Phase 2 runs by default. Skip it via `--no-phase-2` (or `--phase1-only`) when: - The user only wants raw extraction (e.g., they're running on a corpus of 1,000+ APKs and will curate companies later). Phase 2 is per-package O(1 web search), which scales linearly with distinct unknown packages — fine for tens, costly for hundreds. - The CLUES file is going to be reviewed by a human before merge — let the human do the lookups so the citations they leave in commit messages are theirs, not the agent's. - Some `(inferred from package id)` entries are intentional placeholders pending more APK samples; the user should flag those explicitly so the enrichment loop skips them. ## Phase 3 — add a re-download URL to every record's `evidence_array` A CLUES record without a re-download URL is hard to reproduce: another researcher who wants to validate a UUID has no way to grab the exact same APK build. Phase 3 closes that gap by finding an apkpure / apkcombo / archive.org / Play Store URL for each `(package_id, version_name)` pair in the output and adding it as a CLUES `evidence_array` item. **Every URL is CLI-verified before commit**, so the JSON never ships a dead link. Three helpers in `scripts/` make this a 3-step loop, parallel in structure to Phase 2: ### Step 3.1 — list which APK versions still need a URL ```bash python3 /Users/user/.claude/skills/apk-clues-extract/scripts/list_apk_versions.py \ /path/to/CLUES_data_LLM_Android_APK_search.json ``` Stdout is a JSON document with `needs_url` (list of `{package_id, version_name, version_code, record_count, company, local_path}`, sorted by record_count so high-impact APKs go first) and `already_has_url` (list of `"@"` keys that already carry an APK-cache URL in their `evidence_array`). The Phase-3 agent should only work on entries in `needs_url`. ### Step 3.2 — find a URL per APK version and CLI-verify For each `(package_id, version_name)`, the agent should: 1. **Search the web** with a query like: ``` apkpure OR apkcombo …(truncated)