Debug Agent Hw Test

Debug AgentHwTest Failures

End-to-end workflow for building, running, and debugging FBOSS AgentHwTest failures on switches. Drives iterative test-debug-fix cycles.

Prerequisites: Device Access

This skill requires access to a lab switch to deploy and run tests. Read the device access reference (see Reference Routing below) to learn the available commands for your environment (upload, download, run commands on switch).

Invocation Modes

The skill supports two modes:

Mode selection: If the user provides a single test name, run Mode A. If the user provides a file path, read the file and run Mode B.

Test List File Format (Mode B)

One test name per line. Lines starting with # are comments, blank lines are ignored.

# /tmp/test_list.txt
AgentVoqSwitchTest.addRemoveNeighbor
AgentAclCounterTest.AclStatTest

Inputs

Gather these before starting:

Deriving Switch IDs from Config

Switch IDs are platform-specific. Extract them from the materialized config file:

python3 -c "
import json, sys
with open(sys.argv[1]) as f:
    cfg = json.load(f)
for sid, info in cfg['sw']['switchSettings']['switchIdToSwitchInfo'].items():
    print(f'switchId={sid} switchIndex={info[\"switchIndex\"]}')
" /root/<user>/<platform>_multi.agent.materialized_JSON

Known platform switch IDs:

The --switch_id_for_testing flag takes the switch ID (not the switch index).

Build-Before-Run Rule

MANDATORY: Always build binaries on every skill invocation before running any test. Never skip the build step, even if binaries already exist on the switch from a previous session or invocation. Stale binaries produce incorrect results because the source code may have changed.

On every skill invocation:

1. Build the required binaries (mono or multi-switch)
2. Strip and copy them to the switch (strip_and_copy.sh strips the binary and prints its md5; the separate upload step uses md5 dedup — an unchanged binary skips the network copy)
3. Copy the config file and test scripts to the switch
4. Then proceed to run the test

The md5 dedup in the upload step makes this safe and fast — if the binary hasn't changed, the copy is skipped. But the build must always run to ensure the binary reflects the current source.

Debug Loop

Follow this iterative cycle for each failing test:

1. Build the test binary (mono or multi-switch) — never skip this step
2. Load the binary onto the switch
3. Copy firmware if testing on a Broadcom DNX switch (see build-and-load.md)
4. Run the test (cold or warm boot)
5. Analyze output — logs, crashes, vendor diag shell
6. Fix or adjust — code change, feature flag, XLOG insertion
7. Re-run to verify

Broadcom DNX firmware prerequisite: All tests on Broadcom DNX switches (Jericho3, Ramon3, etc.) require the firmware db/ directory at /tmp/db/ on the switch. See build-and-load.md for details. Without this, hw_agent processes abort with FW: ... is not accessible error:-1. This does not apply to Broadcom XGS or Leaba/Cisco platforms.

Discipline: Work on one test at a time. Try up to 5 iterations before categorizing and moving to the next test.

Batch Execution Loop (Mode B)

When running in batch mode, follow this loop for each test in the list:

for each test in list:
  1. RUN the test (cold boot + warm boot, across all switch_ids)
  2. if PASS on all switch_ids:
       - Record as PASS_NO_CHANGE
       - Proceed immediately to next test (no debug needed)
  3. if FAIL on any switch_id:
       - Enter debug loop (up to 5 iterations)
       - Use debug references: analyze-logs, crash-debug, vendor-diag-shell, etc.
       - Categorize result (PASS_FEATURE_FLAG, PASS_FBOSS_FIX, FAIL_FBOSS, etc.)
       - After 5 iterations or categorization, MOVE ON to next test
  4. Update the running results table after each test

Key principle: Passing tests get zero debug time. Failing tests get bounded debug time (max 5 iterations). The batch always makes forward progress — never get stuck on one test.

Session Tracking

Maintain a results table at /tmp/agent_hw_test_results.md to track progress across the debug session. Update after each test attempt.

Mode A — Single Test Results Table

| Test Name                              | Switch ID | Cold Boot | Warm Boot |
|----------------------------------------|-----------|-----------|-----------|
| AgentVoqSwitchTest.addRemoveNeighbor   | 0         | PASS      | PASS      |
| AgentVoqSwitchTest.addRemoveNeighbor   | 4         | PASS      | PASS      |

Mode B — Batch Results Table

| # | Test Name                              | Switch ID | Cold Boot | Warm Boot          | Category         | Iterations | Notes                        |
|---|----------------------------------------|-----------|-----------|--------------------|------------------|------------|------------------------------|
| 1 | AgentVoqSwitchTest.addRemoveNeighbor   | 0         | PASS      | PASS               | PASS_NO_CHANGE   | 1          |                              |
| 1 | AgentVoqSwitchTest.addRemoveNeighbor   | 4         | PASS      | PASS               | PASS_NO_CHANGE   | 1          |                              |
| 2 | AgentAclCounterTest.AclStatTest        | 0         | FAIL      | SKIPPED (cold fail) | FAIL_VENDOR      | 4          | SAI_STATUS_NOT_SUPPORTED     |
| 2 | AgentAclCounterTest.AclStatTest        | 4         | FAIL      | SKIPPED (cold fail) | FAIL_VENDOR      | 4          |                              |

At the end of a batch run, also print a summary:

=== Summary ===
Total tests: 2
  PASS_NO_CHANGE:    1
  PASS_FEATURE_FLAG: 0
  PASS_FBOSS_FIX:    0
  PASS_VENDOR_FIX:   0
  FAIL_FBOSS:        0
  FAIL_VENDOR:       1
  TODO_ASIC_CONFIG:  0

Pass rate: 1/2 (50%)

Test Categorization Taxonomy

Categorize every test into one of these 7 outcomes:

For FAIL_VENDOR (or PASS_VENDOR_FIX), produce a vendor-escalation package — SAI Replayer log + hw_config + a case report the vendor can reproduce from — using references/vendor-escalation.md. It is vendor-generic (Broadcom CSP / Cisco TAC / NVIDIA support).

Scripts

On-Switch Scripts (uploaded and run on the switch)

These scripts run on the switch. Upload them once per session, then run in the background.

Local Helper Scripts (run on build host only)

Vendor firmware scripts: See build-environment.md for environment-specific SDK path resolution and firmware preparation scripts.

Remote Execution Pattern

Upload scripts to the switch, run tests in the background, and fetch logs when complete. First resolve the device-access reference via the Reference Routing table below, then follow run-tests.md for the full execution pattern.

Reference Routing

For each reference pair below, load the facebook/ version first if it exists in your checkout. Otherwise load the references/ version. Treat the selected file as the source of truth for that topic.

This routing is client-agnostic:

• internal checkouts typically provide facebook/ overrides with lab-specific device access and build instructions
• OSS checkouts fall back to references/ with standard ssh/scp and open-source build instructions
• client-specific command forms are optional conveniences only; the resolved reference must always contain a direct runnable path as fallback