NoorQureshi Model extraction & data inference
When it applies
You have query access to an ML/LLM endpoint and want to show it leaks the model itself, its training data, or confidential context — IP theft or privacy impact, not just a bad answer.
Why it works
Query access is more powerful than it looks. Outputs (labels, probabilities, embeddings, generations) carry information about the model and its data. Enough targeted queries reconstruct a functional copy, reveal whether a record was in training, or regurgitate memorized secrets.
Method
- Model stealing: query systematically (esp. if confidence scores/logits are returned) to train a surrogate that mimics the target — proves the model can be cloned via the API.
- Membership inference: compare model behaviour (confidence, loss) on candidate records to infer whether a specific record was in the training set (privacy impact).
- Training-data / secret extraction (LLM): prompt for memorized data — PII, keys, or the
system prompt/hidden context (overlaps
ai-prompt-injection); look for verbatim regurgitation. - Embedding inversion: if an embeddings API is exposed, reconstruct approximate input text from vectors.
- Cost/DoS angle: unbounded/unthrottled querying is itself a finding (LLM10).
Gotchas
- Tie it to impact: a stolen surrogate, a confirmed membership leak, or verbatim secret output — not "it answered a lot".
- Respect scope/RoE — extraction requires many queries; get authorization and mind rate/cost limits.
- Defenders: rate-limit, strip logits, add output filtering, and monitor query patterns.
Verify success
Demonstrated leakage: a working surrogate, a reliable membership inference, or verbatim training-data/secret extraction.
References
OWASP LLM Top 10 (2025); "Stealing ML models via prediction APIs" (Tramèr et al.); membership-inference literature.