Plugin Inference Middleware

Implements NemoInferenceMiddleware plugins for in-process inference request/response interception in IGW. Use when building a middleware plugin, implementing process_request or process_response, handling MiddlewareCall config (inline or config_id), exposing config entity CRUD APIs, or wiring up the nemo.inference_middleware entry-point. Trigger keywords: inference middleware, NemoInferenceMiddleware, process_request, process_response, MiddlewareCall, config_id, ImmediateResponse, VirtualModel middleware, middleware plugin.

NVIDIA NeMo Updated 2.3k repo stars

File contents

nvidia-nemo/nemo-platform commit 577f78cc44

Frequently asked questions

npx get-skill add nvidia-nemo/plugin-inference-middleware-2