Native Consumers

Native Consumers

Overview

The NATIVE transport carries encoded video over the network (NVIDIA StreamSDK / NVSS). A consumer connects to a running native server by IP + port, receives the encoded stream, and decodes it client-side via StreamSDK's NvStreamingMedia inbound-video API (which uses NVDEC on NVIDIA GPUs) into host BGRA8 frames. On each wait_frame the consumer gets a zero-copy host view of the decoded pixels.

Unlike SHM and CUDASHM, the native consumer needs neither the same host nor the same GPU as the producer — only an NVIDIA GPU + driver on the consumer machine for the NvStreamingMedia decode. Producer and consumer can run on two different machines. That cross-machine capability is the trade for the encode + decode latency the local transports avoid.

The unified ovstream.Client / ovstream_create_client is backend-agnostic — the transport is chosen by ClientType. This skill covers ClientType.NATIVE; the same Client also speaks ClientType.SHM (host-resident, same machine, see the shm-consumers skill) and ClientType.CUDASHM (GPU-resident, same host, see the cudashm-consumers skill). For the full transport trade-off, see protocol-selection.

Python: ovstream.Client(ovstream.ClientType.NATIVE)

The minimal pattern:

with ovstream.Client(ovstream.ClientType.NATIVE, server_ip="10.0.0.5") as client:
    while client.is_alive():
        frame = client.wait_frame(timeout_ms=500)
        if frame is None:
            continue
        # frame.as_numpy() returns a host BGRA8 view of the decoded pixels
        pixels = frame.as_numpy()
        # ... process ...

The config is connection parameters, not a stream name: server_ip (required), signal_port (default 49100), stream_port (default 47999), and cuda_device (default -1 = device 0, the GPU the client decodes / converts on). Ports are validated to 0..65535 (a ValueError is raised otherwise).

wait_frame(timeout_ms=...) blocks up to the timeout, returning None on timeout or when no new frame has arrived. is_alive() flips to False when the StreamSDK connection drops. The returned ovstream.Frame has frame.source == ovstream.ClientType.NATIVE, and the pixels are host-resident (frame.data / frame.as_numpy() / frame.as_buffer()), frame.pitch_bytes is the row stride, and frame.sequence is a monotonic counter. The same reverse-channel send_message / send_input_event / send_unicode / on_message API is available — the producer's regular callbacks fire for native events too.

For a visual reader using OpenCV with live mouse / keyboard forwarding, see examples/python/native_client/main.py.

C / C++: <ovstream/ovstream_client.h>

The C API surface (type is OVSTREAM_CLIENT_NATIVE, with config.native.server_ip set, and optionally signal_port / stream_port / cuda_device):

• ovstream_create_client(type, &config, &client) — connect to a server by IP + port. Initializes the StreamSDK session and the NvStreamingMedia decoder.
• ovstream_client_wait_frame(client, timeoutMs, &frame) — block for a decoded frame; populates frame.data with the host BGRA8 pixels.
• ovstream_client_is_alive(client, &alive) — watchdog; reflects whether the StreamSDK connection is up.
• ovstream_client_get_producer_device(client, &device) — the client's decode / convert device (native.cuda_device, or 0).
• ovstream_destroy_client(client) — disconnect the StreamSDK session and tear down the decoder.
• ovstream_client_send_input_event(client, &event) — see Reverse channel.
• ovstream_client_send_message(client, message) — see Reverse channel.
• ovstream_client_send_unicode(client, text) — see Reverse channel.
• ovstream_client_set_message_callback(client, cb, user_data) — see Reverse channel.

The returned ovstream_frame_t carries source == OVSTREAM_CLIENT_NATIVE and the decoded pixels in frame.data (host-resident, like SHM). Useful when writing a headless C consumer, a desktop viewer, or any non-Python client of a native server. For a minimal headless example, see examples/c/native_client/main.cpp.

Retry on connect

nvstConnectToServer fails synchronously when the server isn't reachable yet, so a consumer started before its producer fails immediately rather than waiting. The examples wrap ovstream_create_client in a short retry loop (a deadline of ~10 seconds) so the producer / consumer launch order is forgiving. Mirror that pattern in production: catch the create failure, sleep briefly, and retry until a deadline.

Reverse channel: driving the producer from the consumer

Native isn't a one-way pipe. Alongside the video stream, the consumer can push events to the producer over the StreamSDK input / message channels. The producer's regular on_message / on_input / on_unicode callbacks fire for these the same way they do for WebRTC / SHM / CUDASHM — producer code is transport-agnostic.

Python:

client = ovstream.Client(ovstream.ClientType.NATIVE, server_ip="10.0.0.5")
client.send_message("hello from the native client")
client.send_unicode("é")  # IME / composed-text event
client.send_input_event(
    ovstream.InputEvent(
        type=ovstream.InputEventType.KEYBOARD,
        keyboard=ovstream.KeyboardEvent(key_code=65, scan_code=0,
                                        modifiers=0,
                                        key_state=ovstream.KeyState.DOWN),
    )
)
client.on_message = lambda text: print(f"from producer: {text}")

C:

• ovstream_client_send_input_event(client, &event) — keyboard / mouse event. Gamepad is not forwarded on the native path (keyboard / mouse / unicode are).
• ovstream_client_send_message(client, message) — text payload. Native caps each payload at 65535 bytes (the StreamSDK custom-message limit).
• ovstream_client_send_unicode(client, text) — IME / composed-text event. Same 65535-byte cap.
• ovstream_client_set_message_callback(client, cb, user_data) — register a server → client text-message handler.

Resilience

• If the connection drops (producer exits or the network breaks), wait_frame returns a timeout / invalid-state result and is_alive flips. The consumer should detect that and exit cleanly.
• Producer or consumer can start first; the consumer waits via the retry-on-connect loop above.

Key Types / Functions

Common Pitfalls

• Consumer needs an NVIDIA GPU + driver. The decode runs through StreamSDK's NvStreamingMedia, which uses NVDEC on NVIDIA GPUs, so a machine with no NVIDIA GPU can't be a native consumer. This is the cost of lifting the same-host / same-GPU constraint of SHM / CUDASHM.
• Connect is synchronous and fails fast. Start the producer first, or use a retry-on-connect loop (the examples do). A consumer started before its producer otherwise fails immediately.
• Ports must match the server. signal_port (default 49100) and stream_port (default 47999) must match what the producer was started with. Both must be reachable across the network / firewall.
• No gamepad input. Keyboard, mouse, and unicode forward to the producer; gamepad does not on the native path.
• Message / unicode payloads cap at 65535 bytes. The StreamSDK custom-message limit. Chunk larger payloads at the application layer.
• No browser viewer. Native requires a StreamSDK client (the native_client examples, or your own). To reach a browser, run a WebRTC server alongside the native one.
• Link only ovstream::ovstream_client. A native consumer links the ovstream::ovstream_client target (never the server-side ovstream library), includes <ovstream/ovstream_client.h>, and has a runtime dependency on the bundled NvStreamingMedia + NVCodec libraries for the decode plus StreamClientShared for the StreamSDK session.