Auditing Ml Inference Endpoint Abuse

Auditing ML inference endpoint abuse: a served model is an asset and a meter, both attackable

A model exposed as an inference endpoint is two things worth attacking at once: a metered resource that costs money per call, and a confidential asset that queries can reconstruct. The abuses follow from that. If the endpoint is unauthenticated or weakly keyed, anyone can call it. If there is no per-caller rate or spend limit, a caller runs up unbounded inference cost, a denial-of-wallet against expensive model serving. Beyond cost, the model itself leaks to a determined querent: systematic queries reconstruct the model or its decision boundary (model extraction), and carefully chosen queries recover whether a specific record was in the training set or reconstruct sensitive training data (membership and inversion inference). Over-informative responses, full class probabilities, raw embeddings, confidence vectors, make both extraction and inversion far easier. The audit treats the endpoint as a cost meter and a confidential asset and checks the controls on both. You audit this by calling the endpoint as an attacker would: unauthenticated, at volume, and systematically.

When to use

• A model is deployed as a callable prediction or embedding endpoint (classifier, recommender, scorer).
• The endpoint may be unauthenticated, weakly keyed, or lack per-caller rate and spend limits.
• Responses may return full probabilities or embeddings, and the model or its training data is confidential.

Scope check

Test inference endpoints only on models and services you own or are authorized to assess, on non-production deployments. Volume and systematic querying incur real cost and exercise a real confidentiality boundary, so use a test deployment and never run up spend or extract from a model that is not yours. If you can't name the authorization, stop.

The loop

1. Establish the intended access, cost, and confidentiality bounds first. Name who may call the endpoint, the per-caller rate and spend limit, and what the response should reveal. This is the false-positive killer: an endpoint that authenticates callers, enforces per-caller rate and spend limits, and returns only the minimal answer (a label, a top-k, a coarse score) is behaving correctly. Name the intended bounds, then test each.

2. Check authentication and per-caller identity. Confirm the endpoint requires authentication and attributes each call to a caller, rather than being open or sharing one key. An unauthenticated or shared-key endpoint has no one to rate-limit or bill, so every abuse below is unbounded.

3. Test rate and spend bounds (denial-of-wallet). Call the endpoint at volume and confirm a per-caller rate limit and a spend or quota cap stop it. Where inference is expensive (large models, GPU, per-token billing), confirm the cost of a single caller's traffic is bounded. No per-caller ceiling means one caller can run up arbitrary cost.

4. Test model-extraction exposure. Assess whether systematic queries can reconstruct the model or its decision boundary: whether query volume is bounded, whether responses are informative enough to train a surrogate, and whether extraction-shaped query patterns are detected. An endpoint that answers unlimited fine-grained queries hands over the model over time.

5. Test membership, inversion, and response verbosity. Check whether the response returns more than needed, full probability vectors, raw embeddings, exact confidences, and whether that enables membership inference (was this record trained on) or model inversion (reconstructing sensitive inputs). Confirm the response is minimized to the decision the caller needs.

6. Confirm and record. Confirm by abusing the endpoint on a test deployment: calling it unauthenticated, running unbounded cost, training a surrogate from its answers, or recovering membership from verbose responses, without running up real spend or extracting a real model. Kill the lead if the endpoint authenticates callers, enforces per-caller rate and spend limits, and returns minimized responses that resist extraction and inversion. Record the abusive query stream, the cost or reconstruction sink, and the missing auth, bound, or over-informative response.

Where the endpoint leaks

• Open or shared-key access. An unauthenticated endpoint or one shared key means no per-caller identity to limit or bill, so cost and extraction are unbounded.
• No per-caller cost ceiling. Missing rate and spend limits let one caller run up arbitrary inference cost, denial-of-wallet against expensive serving.
• Unlimited fine-grained queries. An endpoint that answers systematic queries without volume bounds or pattern detection lets a caller reconstruct the model.
• Over-informative responses. Full probability vectors, raw embeddings, and exact confidences make extraction, membership inference, and inversion far easier than a minimal answer would.
• No abuse detection. Extraction and inference attacks have distinctive query patterns; an endpoint that never looks for them cannot slow them down.

Worked example (a confirm and a kill)

Confirm. A classification endpoint requires only a shared API key embedded in the web client and returns the full probability vector for every class. A single extracted key drives high-volume systematic queries with no per-caller limit, running up substantial GPU inference cost and, from the fine-grained probabilities, training a surrogate that reproduces the model's decisions. Confirmed denial-of-wallet plus model extraction via shared key and verbose responses, high, remediation = authenticate and attribute each caller, enforce per-caller rate and spend limits, and return only the top label or a coarse score rather than full probabilities.

Kill. The endpoint authenticates each caller and attributes calls to them, enforces a per-caller rate and spend limit that bounds cost, returns only the minimal decision (top label or coarse score, no raw embeddings or full vector), and flags extraction-shaped query patterns. High-volume, systematic, and verbose-response abuse are all bounded. Killed, kill_reason = "per-caller auth, rate and spend limits, and minimized responses; neither unbounded cost nor model or training-data reconstruction is reachable."

Rationalizations to reject

• "The key is in the client, so callers are known." → A client-embedded key is extractable and shared; treat the endpoint as open unless each caller authenticates and is attributed.
• "Inference is cheap." → Per call, maybe; at unbounded volume on GPU or per-token billing it is not. Confirm a per-caller spend ceiling exists.
• "You cannot steal a model through an API." → Systematic queries reconstruct the decision boundary over time; bound query volume and minimize responses to make it impractical.
• "We return probabilities because clients want them." → Full vectors and embeddings enable extraction and inversion; return the minimal answer the client actually needs.
• "No one is attacking it." → Extraction and membership inference have distinctive patterns; without detection you would not know, and the cost accrues regardless.

Executing this in practice

You need the endpoint's authentication, its per-caller rate and spend limits, the verbosity of its responses, and any abuse detection. Call it unauthenticated, at volume, and with systematic and membership-probing query sets on a test deployment, watching cost and what the responses reveal. Reading the serving configuration shows the intended bounds; unbounded cost or a working surrogate shows whether they hold.

Related

• auditing-ml-model-supply-chain - the load-time trust of the model artifact; this skill is the run-time abuse of the same model once it is served.
• reviewing-rate-limiting-and-abuse-controls - the general rate and abuse-control discipline this applies to the specific economics of model inference.
• auditing-ai-agent-permissions - denial-of-wallet through an agent's tool use is the agent-side version of the cost abuse here.
• hunting-code-interpreter-and-tool-sandbox-escape - the execution side of the AI surface; serving abuse and sandbox escape are the two halves of an exposed model runtime.
• FINDING-SCHEMA.md - source = the unbounded or systematic query stream, sink = the run-up cost or reconstructed model or training data, evidence = the missing auth, bound, or over-informative response.