Perf Benchmarking

Perf benchmarking (on-device, stream-chat-react-native)

How to measure SDK performance for real, against the SampleApp running on a physical Android device, using the committed perf/ toolkit. This skill exists because the methodology is full of traps that produce confidently-wrong numbers; follow the rules and recipes here instead of improvising.

Prerequisites for almost everything below:

• The installed app is a DEBUG build wired to Metro — check this FIRST, before anything else. A release build embeds the JS bundle in the APK and ignores Metro entirely, so none of your src edits or throwaway instrumentation ever run, AND the Hermes inspector is disabled so CPU profiling can't attach. Symptoms are silent and misleading: the app runs fine, navigation works, but every marker you add stays absent and you waste an hour chasing phantom "stale bundle" / "adb reverse" causes (this happened). Verify with one command — adb shell run-as io.getstream.reactnative.sampleapp ls >/dev/null 2>&1 && echo DEBUG || echo "RELEASE — rebuild" (a release build reports "package not debuggable"); or check the flags: adb shell dumpsys package io.getstream.reactnative.sampleapp | grep flags= should include DEBUGGABLE. If it's release, install debug first: yarn workspace sampleapp android (builds, installs over the release app, sets up adb reverse). Confirm the device→Metro reverse exists: adb reverse --list should show tcp:8081.
• Metro running: yarn workspace sampleapp start (the device pulls the JS bundle from it; Metro bundles the SDK from src via the react-native package.json field, so editing package/src/** hot-reloads on relaunch — no yarn build needed).
• A device/emulator connected: adb devices shows one. The SampleApp package is io.getstream.reactnative.sampleapp.

What we're measuring for

The question is always: is this change a net positive for performance, and by how much? There are two axes, and either one improving (without regressing the other) is a win worth shipping:

• Lighter — less work per message / per action: fewer native calls, fewer listeners, fewer renders, less JS-thread time. A genuine win even if render latency is unchanged.
• Faster — the render / draw itself takes less time.

Good-practice bar for this SDK: if messages (or the action under test) end up lighter and/or render faster, that's a win. Always quantify the delta (counts and/or ms) and state the direction — net positive / neutral / regression. Report both axes honestly: don't dress a lightness win up as "faster" (rule 4), but don't undersell a lightness win either — it counts.

Non-negotiable rules

1. Measure on a real device, never Node/Jest. The RN jest preset mocks AccessibilityInfo and native modules, so a micro-benchmark of any native-touching code is meaningless. Proven the hard way: a Node bench said ~6µs/call; the device said ~86µs — 14× off — purely because Jest mocked the native call to ~free.
2. Prefer deterministic counts over timing. Counting native calls / listeners / renders is reproducible and trustworthy. Wall-clock timing is noisy; a single timing sample proves nothing. If you report timing, take ≥5 runs, give median + spread, and label it debug-build.
3. Count version-agnostically — instrument the native API, not one call site. Wrap e.g. AccessibilityInfo.isScreenReaderEnabled once at app startup and count every caller. Instrumenting a single hook undercounts and hides that a dependency may dominate. Real example: after centralizing our hook, the SDK made 1 call — but the app still made ~30, because RNGH Pressable independently queries it per instance.
4. Report "lighter" and "faster" as distinct wins — don't conflate them. Removing work from a useEffect (off the render/draw path) is a real win even though it doesn't lower draw latency. Both axes count (see "What we're measuring for"); just never claim "renders faster" from effect-phase work — name the axis that improved and quantify it.
5. Match the instrument to the change size.
   • Structural per-call / per-render change → counting (Pattern 1).
   • Big JS-thread cost change → Hermes CPU profile (Pattern 2).
   • "What component re-renders too much / too often" → React DevTools profile (analyze-react-profile.js).
   • Memory / jank / dropped frames → android-heap-dump.sh.
   • A CPU-profile --diff is blind to sub-noise changes and is polluted by line-number-shift phantom deltas when you diff across a code edit. Do not use it to "prove" a tiny change.
6. A/B fairly — same data, same path, multiple runs. Same scenario, instrumentation, device, channel (with enough messages), and swipe count; only the code under test differs. Crucially, both builds must traverse the identical content — the same images, the same messages, the same items in the same order. Content-dependent costs (image decode, layout, text shaping) otherwise confound the result, and the tail percentiles (95th/99th) are dominated by the single heaviest item traversed — so a different start position or swipe path makes them meaningless (this bit us once: two gallery runs opened on different images, and the 99th moved 50ms purely from decoding a bigger bitmap, not from the code). Pin the start (e.g. force a fixed gallery index via throwaway instrumentation) and drive a fixed path so each run does identical work. And run each side ≥3× and report the spread — a single run per side cannot separate a real delta from run-to-run noise, especially at the tail; a difference inside the runs' spread is not a result.
7. Drive the device on real mount signals, not sleep. Android debug navigation is slow and variable. Wait on testIDs (channel-preview-button → message-flat-list), never a fixed sleep after a tap — or your swipes land on the list / a half-mounted screen.
8. Respect git-hands-off. To A/B a committed change, produce the baseline by editing the file back locally (or git stash if it's uncommitted), measure, then restore by re-writing the committed version. Never git commit, git revert, or rewrite history to benchmark.

The toolbox (perf/)

perf/profiles/ is gitignored. See perf/README.md for the canonical capture/analyze docs.

Execution patterns

1) Deterministic call/listener counting — the reliable default

Best for "did this change make the app do less work?" Counts are exact and reproducible; no timing noise.

a. Version-agnostic instrument — a throwaway file in the SampleApp, e.g. examples/SampleApp/srPatch.ts, wrapping the native API at startup:

import { AccessibilityInfo } from 'react-native';
const g = global as any;
g.__SR_CALLS__ = 0; g.__SR_LISTENERS__ = 0; g.__SR_US__ = 0;
const now = g.performance?.now ? () => g.performance.now() : () => Date.now();
const origIsSR = AccessibilityInfo.isScreenReaderEnabled;
let stacks = 0;
AccessibilityInfo.isScreenReaderEnabled = function () {
  g.__SR_CALLS__ += 1;
  if (stacks < 4) { stacks += 1; console.log(`[SR_STACK] #${g.__SR_CALLS__}\n${new Error().stack}`); } // who calls it
  const t0 = now(); const r = origIsSR.call(AccessibilityInfo); g.__SR_US__ += (now() - t0) * 1000; return r;
};
const origAdd = AccessibilityInfo.addEventListener;
AccessibilityInfo.addEventListener = function (name: any, h: any) {
  if (name === 'screenReaderChanged') g.__SR_LISTENERS__ += 1;
  return origAdd.call(AccessibilityInfo, name, h);
};

Import it first in examples/SampleApp/index.js (import './srPatch'; at the top) so the patch is in place before any provider mounts.

b. Read the counters from a React.Profiler in the screen under test (Profiler onRender fires reliably; setInterval logging has proven flaky). Wrap the message list in ChannelScreen.tsx:

const MsgProfiler = ({ children }) => {
  const phase, dur) =>
    console.log(`[SR_M] ${phase} render=${dur.toFixed(0)}ms calls=${(global as any).__SR_CALLS__||0} listeners=${(global as any).__SR_LISTENERS__||0} us=${((global as any).__SR_US__||0).toFixed(0)}`);
  return <React.Profiler id='m'
};

c. Run + read: relaunch, drive the scenario (Pattern 3), then adb logcat -d -s "ReactNativeJS:V" | grep -E "SR_M|SR_STACK". Take the max calls/listeners seen (they accumulate as rows mount). The [SR_STACK] traces tell you who makes the calls — read them: a React effect frame (commitHookPassiveMountEffects) is one of ours; an asyncGeneratorStep/tryCallTwo chain with no React frames is a dependency.

d. A/B: measure with the change, then revert it (Pattern 4) and measure again, identical scenario. Report the delta in counts.

2) Hermes CPU profile + diff (non-interactive)

For real JS-thread cost. The capture script waits for Enter, so drive it from the background with a timed auto-stop while you run the scenario in the foreground:

# background: starts profiling, auto-stops after 50s, saves + analyzes
( sleep 50; printf '\n' ) | node perf/capture-hermes-profile.js perf/profiles/after.cpuprofile   # run_in_background
# foreground (inside the window): let it connect (~5s), then drive
sleep 5 && bash perf/drive-channel-scenario.sh 8

Then desymbolicate + diff:

curl -s 'http://localhost:8081/index.map?platform=android&dev=true&minify=false' -o /tmp/dev.map.json
node perf/desymbolicate-cpuprofile.js perf/profiles/after.cpuprofile /tmp/dev.map.json
node perf/analyze-cpuprofile.js --diff perf/profiles/baseline.cpuprofile perf/profiles/after.cpuprofile --grep '<fn>'

Heed rule 5: if the change is sub-noise, the diff will show nothing real (only line-shift phantom deltas) — that's expected, not a failure; fall back to Pattern 1.

3) Driving the device — scenarios on scenario-lib.sh

Scenarios are thin scripts on top of perf/scenario-lib.sh, which provides device-driving verbs so each scenario stays declarative and waits on real mount signals (never sleep). Verbs: relaunch · wait_for <testid> [secs] · tap_testid <id> · tap_until <tap_id> <expect_id> [tries] (tap + retry until the next screen's testID appears — taps occasionally don't register) · tap_text <substr> / tap_text_until <substr> <expect_id> (tap a node by its visible text — e.g. open a channel by display name) · tap_header_action [ymax] / tap_header_until <expect_id> (tap the right-most header action when RN flattens its testID away) · swipe_up/down/left/right [ms] · scroll [n] [ms] · scroll_down [n] [ms] [settle] · count_testid <prefix> (uses grep -o | wc -l, never grep -c) · wait_for_log <pattern> [secs] (poll logcat for states the view tree can't see).

Pagination direction depends on whether the list is inverted. To load more rows you must scroll toward the list's growing end and trigger its onEndReached. The message list is inverted (newest at bottom, older above) — paginate by scrolling up (scroll, the lib's swipe_up). The Photos & Videos media grid is a normal list (newest first, older below) — paginate by scrolling down (scroll_down, the lib's swipe_down). Use the wrong direction and you scroll against the anchored edge: nothing loads, the loaded set looks capped, and every "scaling" run silently measures the same N (this happened — preload depths 4/12/25 all yielded 44 assets until the grid was scrolled down, which grew it to 165).

Driving the lib inline runs under zsh, which breaks it. The scenario scripts have a #!/usr/bin/env bash shebang, so bash perf/drive-*.sh is fine. But source perf/scenario-lib.sh from an interactive tool shell runs under zsh, where "$id[^\"]*" inside tap_testid's grep is parsed as an array subscript (bad math expression: operand expected at \^"'). Always drive via bash perf/drive--scenario.sh …`, never by sourcing the lib into zsh.

The channel scenario (perf/drive-channel-scenario.sh) is the reference shape:

source "$(dirname "$0")/scenario-lib.sh"
wait_for channel-preview-button 30 || exit 1
tap_until channel-preview-button message-flat-list || exit 1   # navigate (retries the tap)
echo "MOUNTED"; scroll "${1:-10}"
echo "rows=$(count_testid message-list-item-)"

Adding a new scenario (gallery, threads, reactions, …): copy perf/scenario-template.sh → perf/drive-<name>-scenario.sh, discover the start/target testIDs by dumping the tree (adb shell uiautomator dump /sdcard/ui.xml && adb pull /sdcard/ui.xml /tmp/ui.xml; grep -oE 'resource-id="[^"]+"' /tmp/ui.xml | sort -u), then express the flow with the verbs. Always use tap_until <x> <next-screen-testid> for navigation and wait_for before interacting. Known testIDs so far: channel-list-view, channel-preview-button, message-flat-list, message-list-item-<uuid>. Pick a channel/state with enough content (a list preview like "Heya!" is just the last message — the channel still has full history).

4) A/B around a committed change (git-hands-off-safe)

1. Confirm the change's state: git status. If committed, the feature files won't show as modified.
2. Make the baseline: edit the changed file(s) back to the pre-change version (re-Write them). Metro hot-reloads src on relaunch.
3. Relaunch + measure (baseline).
4. Restore by re-writing the committed version verbatim (read it back from git/HEAD content). Do not git checkout/revert over the developer's work; just rewrite the file content.
5. Measure the changed version; report the delta.

Anti-patterns to avoid (each = a real mistake made; do the fix)

• Measuring against a release build without checking → src edits + instrumentation silently never run, the Hermes inspector won't attach, and the failure looks exactly like a stale bundle or a missing adb reverse (an hour lost chasing both). → adb shell run-as <pkg> ls (or check dumpsys package … flags= for DEBUGGABLE) as step zero; install debug with yarn workspace sampleapp android if needed.
• Node/Jest microbench of native-touching code → mocked to ~free, ~14× wrong. → Measure on device.
• Instrumenting one call site → undercount; misses dependency callers. → Wrap the native API (Pattern 1a).
• CPU-profile --diff for a tiny / cross-edit change → phantom line-shift deltas dominate, real signal is sub-noise. → Use counting (Pattern 1).
• grep -c on uiautomator XML → the dump is one line, so it returns 1 regardless. → grep -oE '…' | wc -l.
• uiautomator dump mid-scroll-animation → returns a partial/sparse tree (looked like "1 row" when there were 15). → Dump after the list settles.
• Paginating the wrong direction for the list's inversion → scroll against the anchored edge, nothing loads, the loaded set looks capped and every "scaling" run measures the same N. → Inverted list (message list) paginates by scrolling up (scroll); normal list (media grid) paginates by scrolling down (scroll_down).
• Sourcing scenario-lib.sh into the tool's zsh shell → tap_testid's "$id[^\"]*" is read as a zsh array subscript → bad math expression. → Drive via bash perf/drive-*.sh, never source into zsh.
• Fixed sleep after a tap, then swipe → debug nav is slow; swipes hit the list / half-mounted screen. → Wait on message-flat-list (Pattern 3).
• Reading cold first-mount numbers → dominated by startup/JIT/module-init (~2500ms "mount" is mostly not your code). → Use warm re-opens; never quote the cold mount as the row cost.
• Claiming "faster"/"slower" from one render sample → it's within noise. → Lead with the deterministic count; only call timing differences real with N runs + non-overlapping spreads.
• A/B runs that traverse different content (different images/messages, or a different start index / swipe path between baseline and after) → content-dependent cost (image decode, layout, text shaping) confounds the delta, and the 95th/99th are set by the single heaviest item traversed, so wins and losses are noise. → Pin the start (force a fixed index via throwaway instrumentation) and drive an identical path so both builds do byte-identical work; run each side ≥3× and compare spreads, not single numbers.

Environment gotchas

• RN 0.85 Fusebox inspector returns 401 on the WebSocket upgrade unless the Origin header matches the Metro host, and Node's built-in undici WebSocket fails the handshake (close 1006 + TypeError). capture-hermes-profile.js already handles both: it resolves the ws package (e.g. package/node_modules/ws) and passes { headers: { Origin: 'http://localhost:8081' } }. If you write a new capture client, do the same.
• Dev-mode profiles keep function names but collapse every frame to one bundle URL — desymbolicate for per-package attribution.
• Always tear down throwaway instrumentation afterward (srPatch.ts, the Profiler wrap, the index.js import) and restore any files you edited for the baseline.

Worked example (the case this skill came from)

Change: centralize useScreenReaderEnabled from a per-row hook (each Message row queried AccessibilityInfo.isScreenReaderEnabled() + added a listener on mount, ×2/row) into a single ScreenReaderProvider (one query + one listener app-wide; consumers read via context).

Measured version-agnostically over open+scroll on the same channel:

• Without the change: 83 calls / 84 listeners / 22.1 ms synchronous JS-thread time.
• With the change: 30 calls / 31 listeners / 12.4 ms.
• Removed ~53 calls, ~53 listeners, ~9.7 ms. Render time: unchanged (within noise) → the honest framing is lighter, not faster.

The [SR_STACK] traces revealed the residual ~30 wasn't ours: 1 call = our provider's effect; the other ~29 = react-native-gesture-handler's Pressable, whose useIsScreenReaderEnabled does the exact per-instance query+listener — one layer down, not fixable from our SDK. Lesson: rule 3 (count every caller) is what exposed this; hook-only counting had wrongly said "1". Full record: memory screen_reader_hook_native_query_cost.md.

Execution checklist

• DEBUG build installed (adb shell run-as <pkg> ls succeeds), not release — checked BEFORE measuring.
• Metro running, exactly one device connected (adb devices); adb reverse --list shows tcp:8081.
• Picked the right instrument for the change (rule 5).
• Chose a channel with enough messages; drove it on testIDs, not sleeps.
• Instrumented the native API (counts every caller), not a single call site.
• Captured [SR_STACK]/equivalent to attribute the calls (ours vs dependency).
• A/B used identical scenario + instrumentation; only code-under-test differed.
• Reported counts primarily; any timing has N runs + spread + "debug build".
• Stated lighter vs faster honestly; no "faster" claim from effect-phase work.
• Tore down all throwaway instrumentation; restored any baseline-edited files.
• Left git to the developer (no commits/reverts).

Reference files

• perf/README.md — canonical capture/analyze/desymbolicate/heap docs.
• perf/capture-hermes-profile.js, perf/capture-server.js — Hermes CPU capture (the ws+Origin clients).
• perf/analyze-cpuprofile.js (--inside/--diff/--grep), perf/analyze-react-profile.js, perf/desymbolicate-cpuprofile.js.
• perf/scenario-lib.sh — device-driving verbs (source it). perf/scenario-template.sh — stub for a new scenario. perf/drive-channel-scenario.sh — reference scenario. perf/android-heap-dump.sh — memory/jank.
• .claude/skills/accessibility/SKILL.md — the project-skill format this mirrors.

iOS footnote (not yet validated)

Everything here was validated on Android. The Hermes capture/analyze scripts are platform-agnostic — the Metro inspector connection is identical, so capture-hermes-profile.js, analyze-cpuprofile.js, etc. should work as-is on an iOS simulator. Only the device-driving (Pattern 3, drive-channel-scenario.sh) is Android-specific: for iOS, replace adb/uiautomator with xcrun simctl + the accessibility (AX) hierarchy or idb ui to find/tap testIDs, and screen-reader state maps to VoiceOver (getCurrentVoiceOverState). Treat an iOS run as unproven until done once and folded back in here.