J6 Ucp Hbm Infer

hbm_infer X86 Python Client Code Generation

Generate production-ready X86-side Python client code for hbm_infer — the Python SDK that connects to a BPU board via gRPC, deploys HBM models over SSH, runs inference, and manages board resources.

Quick Decision: Which Mode?

Standard Mode — Quick Reference

from hbm_infer.hbm_rpc_session import HbmRpcSession

session = HbmRpcSession(
    host="<board_ip>",
    local_hbm_path="<hbm_file_path>",
)
# Use session, then always close
session.close_server()

HbmRpcSession.init parameters:

Key methods:

• session(data, output_config=None, model_name=None) — Run inference
• session.get_model_names() → List[str]
• session.get_input_info(model_name=None) → Dict[str, Dict]
• session.get_output_info(model_name=None) → Dict[str, Dict]
• session.show_input_output_info(model_name=None) — Print model I/O info
• session.get_profile(model_name=None) → Cumulative timing stats (avg/min/max in ms)
• session.get_profile_last_frame(model_name=None) → Last-frame timing stats (ms)
• session.close_server() — Must call explicitly to release board resources

Flexible Mode — Quick Reference

from hbm_infer.hbm_rpc_session_flexible import (
    HbmRpcSession, init_server, deinit_server, init_hbm, deinit_hbm
)

server = init_server(host="<board_ip>")
handle = init_hbm(local_hbm_path="<hbm_file_path>", hbm_rpc_server=server)
session = HbmRpcSession(hbm_handle=handle, hbm_rpc_server=server)
# Use session...
session.close_server()
deinit_hbm(handle)
deinit_server(server)

Flexible mode separates lifecycle into three levels:

1. Server (init_server / deinit_server) — Board connection and file deployment
2. HBM Handle (init_hbm / deinit_hbm) — Model file on board
3. Session (HbmRpcSession.__init__ / close_server) — Inference session

init_server parameters: host, username="root", password=None, ssh_port=22, remote_root="/map/hbm_infer/" init_hbm parameters: local_hbm_path, hbm_rpc_server

Flexible mode HbmRpcSession.__init__ takes hbm_handle and hbm_rpc_server instead of host and local_hbm_path. All other parameters (frame_timeout, with_profile, etc.) are identical to standard mode.

Transmission Optimization with HTensor

HTensor reduces data transfer between X86 and board. Three optimization scenarios:

1. Fixed/Periodic Input — Cache on Board

from hbm_infer.hbm_rpc_session import HbmRpcSession, HTensor

fixed_input = HTensor(
    data=torch.ones((1, 3, 224, 224), dtype=torch.int8),
    device=["cpu", "bpu"],  # "cpu" = X86, "bpu" = board copy
    key="model.input_0",    # Unique board-side identifier
)
# Update data periodically — only changed frames get transferred
fixed_input.data = torch.ones((1, 3, 224, 224), dtype=torch.int8) * 2

HTensor.init parameters:

HTensor properties: data (get/set), device (get only), key (get only), shape (get only, auto-maintained)

2. Output Filtering — Skip Unused Outputs

output_config = {
    "unused_output": {"device": None}  # Don't transfer back
}
result = session(data=input, output_config=output_config)
# result["unused_output"] is HTensor with data=None, but shape is available

3. Model Chaining — Board-Side Passthrough

# Model0 output stays on board
model0_output_config = {
    "output_0": {"device": "bpu", "key": "model0.output_0"}
}
model0_output = session(data=model0_input, output_config=model0_output_config, model_name="model0")

# model0_output["output_0"] is HTensor — use directly as model1 input
model1_input = {"input_0": model0_output["output_0"]}
model1_output = session(data=model1_input, model_name="model1")

output_config Format

Dict[str, Dict[str, Any]] — first-level keys are output names, second-level keys:

Outputs in output_config return as HTensor; others return as np.ndarray or torch.Tensor.

Inference Input/Output Rules

• Input data: Dict[str, Union[np.ndarray, torch.Tensor, HTensor]]
• Output: Same type as input (torch in → torch out, numpy in → numpy out)
• No mixing torch.Tensor and np.ndarray in the same data dict
• All torch.Tensor inputs must have the same device
• Input names, shapes, and element types must match model specs

Model I/O Info Fields

get_input_info() / get_output_info() return Dict[str, Dict] — first-level keys are tensor names, second-level values contain:

tensor_type values: DATA_TYPE_S4, DATA_TYPE_U4, DATA_TYPE_S8, DATA_TYPE_U8, DATA_TYPE_F16, DATA_TYPE_S16, DATA_TYPE_U16, DATA_TYPE_F32, DATA_TYPE_S32, DATA_TYPE_U32, DATA_TYPE_F64, DATA_TYPE_S64, DATA_TYPE_U64, DATA_TYPE_BOOL8

quanti_type values: QUANTI_TYPE_NONE, QUANTI_TYPE_SCALE

Multi-Model Session

When local_hbm_path contains multiple models:

• Model names ≠ HBM file names — call session.get_model_names() to list them
• Must specify model_name in get_input_info, get_output_info, __call__, get_profile
• Model file names and graph names must be unique across the session

Profile Data Structure

When with_profile=True, get_profile() returns cumulative stats (avg/min/max in ms):

get_profile_last_frame() returns the same keys with single-frame values (ms).

Critical Rules

1. Always call close_server() — Board processes and storage will leak otherwise. Do NOT rely on __del__.
2. Always call deinit_hbm() and deinit_server() in flexible mode — same reason.
3. Use try/finally to guarantee cleanup even on exceptions.
4. Compression is software-level — Increases latency, only use to reduce network load / increase throughput. Avoid for float I/O; try for image input / segmentation output.
5. Board-side logs: ${remote_root}/${session_token}/log/s.log for debugging.
6. Multi-model session: user must guarantee unique names — Model file names and internal graph names must not duplicate across the session; duplicates silently overwrite files on the board.
7. Model chaining must use the same session — One session maps to one board-side process; board-side data (HTensor with device="bpu") can only be reused within the same process.
8. L2M model inference requires remote_environment and core_id — Set HB_DNN_USER_DEFINED_L2M_SIZES via remote_environment to allocate L2 memory per BPU core (e.g. {"HB_DNN_USER_DEFINED_L2M_SIZES": "6:6:6:6"} for 4-core, 6MB each). Set core_id to match the cores in the env var. If the user needs L2M inference but hasn't specified this env var, ask for the L2M size requirement.

Code Generation Workflow

When generating hbm_infer code:

1. Ensure hbm_infer is installed: Check with pip3 list | grep hbm_infer. If not installed, run:

   pip3 install hbm_infer -i https://pypi.hobot.cc/hobot-local/simple
   

2. Determine mode: Ask or infer standard vs. flexible based on use case
3. Read model info: Generate code that calls show_input_output_info() or get_input_info() / get_output_info() to discover shapes and types
4. Prepare inputs: Match model input names, shapes, and dtypes exactly
5. Apply optimizations: Use HTensor for fixed/periodic inputs, output_config for filtering/chaining
6. Add cleanup: Always wrap in try/finally with close_server() (and deinit_* for flexible mode)
7. Add profiling if needed: Set with_profile=True and call get_profile() or get_profile_last_frame()
8. Consult FAQ when in doubt: If uncertain about API usage, error handling, or edge cases, read references/faq.md for troubleshooting guidance

For detailed API signatures, complete examples, and troubleshooting, read the reference files:

• references/standard_mode.md — Standard mode full API and examples
• references/flexible_mode.md — Flexible mode full API and examples
• references/trans_optimization.md — HTensor and transmission optimization details
• references/faq.md — Troubleshooting and FAQ