MLSecOps standards — security of the model lifecycle and the AI supply chain
Criteria verified as of August 2026. Re-verify on the web before committing to anything (§8).
1. Scope and triggers
Applies to securing the model lifecycle and its supply chain: provenance and integrity of the
weights, serialisation formats and their capacity for arbitrary execution, scanning and signing of
model artifacts, inventory (AIBOM), security of the training pipeline and of the model registry,
the deployed endpoint as an attack surface, the risk classes specific to the lifecycle (poisoning,
backdoors, extraction, inversion, membership inference, adversarial examples), AI red teaming
methodology, and the mapping to the frameworks in force (ATLAS, OWASP GenAI, NIST AI RMF, AI Act
art. 15).
Posture: defensive and authorised, always. Attacks are described as risk class,
indicator and mitigation; never as a reproducible procedure. See §7.
Domain thesis: a third-party model is a third-party binary. A weights file downloaded from
a public hub has the same trust profile as an executable pulled off the internet — and, in several
formats in mass use, it literally executes code when loaded. Everything that follows derives
from treating it as such: provenance, integrity, signature, scanning, isolation.
Second thesis: the AI supply chain is the software supply chain, plus the data and the
weights. It does not replace it: it extends it. The real 2026 incidents were not exotic attacks
against models, but classic compromises of CI and package repositories that reached AI (§3.3).
The control that removes the most AI risk is still pinning dependencies by digest.
Triggers: .safetensors, .pt, .pth, .bin, .ckpt, .pkl, .gguf, .h5/.keras,
torch.load, weights_only, pickle, joblib.load, trust_remote_code=True, from_pretrained,
picklescan, modelscan, "malicious model", "public hub", "Hugging Face", "model registry",
MLflow, "sign the model", "AIBOM", "ML-BOM", "CycloneDX", "SPDX AI profile", "data
poisoning", "data poisoning", "model backdoor", "backdoor", "model extraction",
"unauthorised distillation", "model inversion", "membership inference", "membership
inference", "adversarial example", "AI red team", garak, PyRIT, promptfoo redteam,
deepteam, "MITRE ATLAS", AML.T, AML.CS, "OWASP Top 10 LLM", ASI01, "NIST AI RMF",
"AI 600-1", "AI Act article 15", "LiteLLM", "TeamPCP".
Not applicable: see
llm-app-engineering-standards (prompt injection is theirs, together with treating model
output as untrusted input, structured output and spend limits. I do not claim it:
here it appears only as one more technique in the ATLAS catalogue when mapping coverage);
ai-agents-standards (agent containment, sandbox, egress, human approval of
irreversible actions, lethal trifecta, OWASP ASI01–ASI10 agentic risks applied to loop design);
rag-standards (per-document access control over the retrieved fragments and index
deletion — poisoning of the retrieval corpus is mitigated there; here it is a risk
class of the lifecycle);
mcp-standards (tool poisoning, rug pull, server shadowing and authorisation of MCP servers);
claude-api (installed skill, canonical reference on the Anthropic side: nothing about Claude
models —id, price, limits, parameters— is asserted from memory);
appsec-standards (application vulnerability classes —IDOR, SSRF, XSS, insecure
deserialisation as a category, STRIDE, ASVS— and SAST/DAST/SCA selection. Here, insecure
deserialisation applied to the model artifact and the classes specific to the AI lifecycle);
offensive-security-standards (the authorised exercise and its governance are theirs, without
exception: written authorisation, RoE, scope, window, deconfliction, stop conditions, report,
retest, legal framing. Precise boundary: if the question is "may I attack this and under what
role?", it is theirs; if it is "what is tested in an AI system and with what method?", it is
mine. The AI red teaming of §3.7 runs inside their RoE, never outside);
llm-evaluation-standards (the measurement apparatus. Boundary declared on
both sides: adversarial attack methodology is mine; the case set, the judge, the rubric,
the threshold and the CI gate it is measured with are theirs. A red team finding becomes a
case in their eval set: that is the handoff);
vulnerability-management-standards (CVE triage, CVSS/EPSS/KEV, remediation SLA and VEX. The
CVEs of the ML stack —torch, transformers, inference servers, harnesses— come in there,
not here);
cicd-standards (SBOM, signing with cosign/Sigstore and SLSA provenance in the pipeline are
theirs, as is runner hardening and ephemeral identity via OIDC. Here only what is specific
to the model: what is signed when the artifact is weights, and what is inventoried when there is
also data);
cryptography-pki-standards (the signing primitive, the algorithm choice and the custody and
rotation of keys);
secrets-management-standards (manager, ephemeral credentials, rotation — the §3.3 incident is a
case of CI credential theft, and the structural mitigation is theirs);
container-runtime-security-standards and kubernetes-standards (isolation of the compute where
an untrusted model is loaded: seccomp, capabilities, non-root, admission);
detection-engineering-standards (rule authorship and lifecycle: Sigma, YARA, tests,
ATT&CK coverage. Here we say which AI event must be emitted and which anomaly matters; the rule
is written and governed there);
incident-response-forensics-standards (technical and forensic response to the compromise, chain of
custody, eradication and mass credential rotation);
privacy-engineering-standards (personal data in training, model memorisation,
de-identification, data subject rights and how AI systems fit under GDPR/AI Act: already
covered — not duplicated here. Membership inference appears in §3.5 as a technical risk
class; its treatment as a privacy risk is theirs);
grc-compliance-standards (management framework, SoA, audit evidence, formal risk
acceptance);
ai-governance-standards (AI Act as a regime, policies, inventory of
AI systems and organisational risk management. Boundary: governance is theirs, the verifiable
technical control is mine. If the answer is a signed document, it is theirs; if the answer is
running a scan or a signature verification, it is mine);
mlops-standards (the operational lifecycle of an in-house model —dataset versioning
with DVC/lakeFS, experiments in MLflow/W&B, model registry with model cards, states and
approval, training orchestration, feature store, canary deployment and weight rollback,
drift and retraining—. This skill is its security face: mlops defines the registry and
promotion between environments; I define how that chain is protected, signed, isolated and audited.
Reproducibility is a quality requirement there and a security control here (§3.10));
local-inference-standards (serving open weights on your own infrastructure: engine, quantisation,
sizing, endpoint security as a port);
gpu-computing-standards (the GPU as a resource: driver, MIG/MPS, accelerator isolation);
data-platform-standards (where the training data lives and its encryption at rest);
iac-standards, identity-access-management-standards, bcdr-standards (infrastructure,
identity and recovery of the ML environment).
2. Default decisions
Verify the latest version and the maintenance status on the web before pinning anything (§8). This
ecosystem has a high tool mortality rate and repositories that get archived and moved.
| Decision |
Default |
Reason |
| Weights format |
safetensors, whenever it exists |
Stores tensors only: no executable code, no deserialisation hooks. It is the only mass-use format designed to eliminate this attack class |
| Formats with arbitrary execution on load |
Vetoed without a sandbox: pickle and everything wrapping it (.pkl, classic PyTorch .pt/.bin/.ckpt, joblib), and Keras models with Lambda layers |
Pickle deserialisation executes code by design. The module's own documentation warns about it |
torch.load over a not fully trusted source |
weights_only=True (and verify your version's default) |
Restricts what can be deserialised. It mitigates, it does not eliminate — there is research on evading restricted loaders |
trust_remote_code=True |
FORBIDDEN by default |
Equivalent to running an arbitrary binary from the internet. There is a recent RCE CVE from hardcoding it (§3.2) |
| Source of the weights |
Internal replica with pinned digest, not a direct hub download at runtime |
A pull at deployment time against a public hub is a mutable dependency in the critical path |
| Artifact identification |
Cryptographic digest (SHA-256), never a tag or "latest" |
Same principle as container images. A tag is mutable |
| Signing and verification |
cosign/Sigstore over the model artifact, verification before loading |
Mechanics and key custody: cicd-standards and cryptography-pki-standards |
| Model scanning |
picklescan and/or modelscan as a gate, with calibrated expectations (§3.2) |
They are denylists: they detect what is known. Necessary, insufficient |
| Loading an unverified model |
Sandbox with no network and no credentials, unprivileged user, read-only FS |
Loading is execution. Containment: container-runtime-security-standards |
| Inventory |
AIBOM/ML-BOM in CycloneDX for the pipeline; SPDX AI profile when the recipient is regulatory |
See §3.4 on real maturity |
| Threat framework |
MITRE ATLAS as the reference taxonomy, complementing ATT&CK |
§3.8 |
| Application risk catalogue |
OWASP Top 10 for LLM Applications 2025 + Top 10 for Agentic Applications 2026 (ASI01–ASI10) |
§3.8 |
| AI red teaming |
garak (broad sweep, CI) + PyRIT (multi-turn campaigns) + promptfoo redteam (in PR) |
§3.7 |
Verified status of the tools (August 2026)
| Tool |
Version |
Date |
Notes |
garak (NVIDIA) |
0.15.1 |
2026-06-05 |
Alive, active repo. Model-level probe scanner |
| PyRIT (Microsoft) |
v1.0.1 |
2026-07-30 |
⚠️ The repository moved: Azure/PyRIT is archived (2026-03-27). The active one is microsoft/PyRIT. Any tutorial pointing at Azure/PyRIT is obsolete |
promptfoo (redteam mode) |
0.121.20 |
2026-07-31 |
Alive, high cadence. Governance: verify the acquisition by OpenAI (§8) |
picklescan |
1.0.5 |
2026-07-01 |
Alive. Integrated into the Hugging Face scanning pipeline |
modelscan (Protect AI) |
0.8.8 |
2026-02-18 |
⚠️ ~6 months without a release. Use it, but pin the version and do not take it as the sole control |
deepteam |
v1.0.4 |
2025-11-12 |
⚠️ No releases in ~9 months. Do not adopt for new work without re-evaluating |
| CycloneDX (spec) |
1.7 (2025-10-21), patches 1.7.1 (2026-06-02) |
— |
ECMA-424; verify the edition in force (§8) |
| SPDX (spec) |
3.0.1 (2024-12-17), 3.1-RC1 (2026-01-24) |
— |
The AI Profile lives in the 3.x line. ISO/IEC 5962:2021 codifies SPDX 2.2.1, not the current one |
| MITRE ATLAS |
content v2026.06 (2026-06-30), format v6.0.0 |
— |
§3.8 |
3. Structure and conventions
3.1 The model as a software artifact
The three non-negotiable controls, in order:
- Provenance: where it came from, who published it, which exact version, against which data it
was trained (as far as that is knowable), and which licence it carries. Without recorded
provenance there is no possible answer to "are we affected?" when the hub pulls a model.
- Integrity: digest computed at ingestion and verified before each load. It is always
referenced by digest, never by name or by tag.
- Signature: the artifact is signed on entry into the internal registry and the signature is
verified before loading, not only at deployment. A verification that only happens in the
pipeline does not protect against substitution of the file in the store.
Operational corollary: the model registry is a production-grade asset, not a drawer of
.ckpt in a shared bucket. Least-privilege access control, audited writes, immutability of
published versions, and separation of who trains, who promotes and who deploys. An attacker with
write access to the registry needs no AI attack at all: they replace the file.
3.2 Third-party weights = third-party binaries
Formats that execute code on load (risk class: insecure deserialisation /
arbitrary execution):
- Pickle and everything built on top of it:
.pkl, the classic PyTorch .pt/.bin/.ckpt,
joblib. The mechanism is the protocol's own reconstruction hook — not a bug, the
design. The pickle interpreter processes opcodes as they arrive, without first validating that
the file is intact, which enables evasion techniques based on deliberately corrupted files.
- Keras/TensorFlow with
Lambda layers: they execute arbitrary Python code embedded in the model.
The problem is not exclusive to pickle.
trust_remote_code=True: downloads and executes code from the model's repository. There is a
recent CVE (CVE-2026-6859, InstructLab, verify) from hardcoding it in a training
script: a malicious model on the hub was enough to achieve RCE on any user. It
requires no scanner evasion whatsoever: it is the functionality doing its job.
Recommended safe format: safetensors — it stores tensor data only, with no code
or deserialisation hooks. It is the default choice and, when a model is published only in
pickle format, that is in itself a risk signal to evaluate (and, if accepted, the conversion to
safetensors happens inside the sandbox, not on the engineer's workstation).
Model scanning — necessary, insufficient, and it must be said:
picklescan and modelscan work by denylist of dangerous functions. They detect
what is known.
- Verified history of evasion: JFrog reported three 0-days in
picklescan (fixed in
0.0.31, Sept 2025), each one allowing detection to be bypassed; the nullifAI technique
(ReversingLabs) evaded the hub's scanning with a deliberately broken pickle; and recent academic
work (ShadowPickle) reports evasion of ten scanners and four model hubs.
- A public hub that flags a model as "unsafe" normally does not block it: it lets you
download and run it at your own risk. The label is not a control.
- Consequence: the scanner is a hygiene gate, not a guarantee. The control that really
bounds the damage is isolation at load time plus preferring safetensors.
Verified real incidents (use as an argument, not as an anecdote): malicious models have been
found on public hubs that open a reverse shell to an external IP when loaded; a year-on-year
increase on the order of 5× in the upload rate of malicious models is reported; and in
February 2026, 341 malicious skills were detected in a public registry of agent skills
distributing an infostealer. The public repository of AI artifacts is today an active malware
distribution vector.
3.3 The AI supply chain — the didactic case of 2026
The compromise of LiteLLM on PyPI (March 2026) is the canonical example that the AI supply
chain breaks where any software supply chain breaks. Verified chain:
- 19 March: the actor (TeamPCP) compromises the GitHub Actions of Trivy — an open-source
security scanner. Since most pipelines reference Actions by mutable tag
instead of pinned commit SHA, the organisations running Trivy in CI started
executing the malicious code immediately.
- Among the harvested credentials was the LiteLLM PyPI publishing token, whose
pipeline invoked Trivy in a secret- and CVE-scanning script.
- 24 March:
litellm 1.82.7 and 1.82.8 are published with a malicious payload. 1.82.8,
~13 minutes later, adds persistence via a .pth file (litellm_init.pth).
- The
.pth mechanism is what must be understood: Python's site module executes the
contents of any .pth in site-packages during interpreter initialisation —
before any import and before any application code. There is no need to import
the library: python --version is enough to trigger it. The payload was doubly base64-encoded
to reduce visibility to basic static analysis.
- Payload: credential harvesting (more than 50 categories: SSH keys, AWS with IMDSv2 and Secrets
Manager, GCP, Azure, Kubernetes,
.env, shell history, git credentials, Docker
registries, Terraform state), lateral movement in Kubernetes (reading secrets in all
namespaces, creating privileged pods mounting the host filesystem) and a persistent
backdoor via a user systemd service. Encrypted exfiltration (AES-256-CBC with a session key
wrapped in RSA-4096) to a domain not affiliated with the project.
- Window: ~40 minutes until quarantine on PyPI. Tens of thousands of installations are reported
in that interval. The campaign continued with
telnyx (27 March) and other registries.
- Who was not affected: the deployments that pinned dependencies in a
requirements.txt
inside the official image.
Lessons that become controls, not anecdotes:
- Pin by digest/SHA, not by tag, for GitHub Actions as well as dependencies and base
images. It is the control that separated the affected from the unaffected.
- Installing a package is executing code. There is no "install and then review".
- A security scanner in your CI is a dependency with credentials. The compromised link
was a defensive tool. Apply the same criteria to it as to any other dependency.
- Ephemeral publishing credentials (OIDC / trusted publishing) instead of long-lived static
tokens →
cicd-standards, secrets-management-standards.
- A host or CI job that installed the compromised artifact is treated as full credential
exposure, not as "check whether the package is present": mass rotation, hunting for
persistence and review of activity in Kubernetes →
incident-response-forensics-standards.
- The catalogue precedent is consistent: the CVEs and maintenance status of the ML stack
(
torch, transformers, inference servers, harnesses) are governed by
vulnerability-management-standards; what is AI-specific is that the weights and the data are two
more links, and no classic SCA covers them.
3.4 AIBOM / ML-BOM — real status, without optimism
A classic SBOM inventories neither weights, nor training data, nor the provenance of the model.
Hence the AIBOM. Honest status as of August 2026: it is an emerging standard, not a mature one.
- CycloneDX supports ML-BOM/AI-BOM and is the practical option for CI. Spec 1.7 published
2025-10-21 (with 1.7.x patches in 2026), adopted as ECMA-424. It is the line with the most
traction in tooling.
- SPDX 3.x defines an AI Profile and a Dataset Profile (model type, training
method, data handling, explainability, limitations, energy consumption). Status:
3.0.1 (Dec 2024) with 3.1-RC1 (Jan 2026). Warning: the ISO/IEC 5962:2021 standard codifies
SPDX 2.2.1, not the version with the AI profile — citing "SPDX is ISO" as proof of AIBOM
maturity is incorrect.
- Field reality: the tool generates what the upstream model declared. If the publisher did not
document data or licence, the AIBOM comes out with gaps — some generators explicitly score
that incompleteness, and that score is the useful datum.
Criteria: generate an AIBOM because inventory is a prerequisite for everything else (and because
procurement is starting to demand it), but do not treat it as a security control. It is an
inventory, and an inventory with declared gaps. The control is still digest + signature + scan + sandbox.
3.5 Lifecycle attacks — risk class, indicator, mitigation
Described as risk. Never as a recipe.
| Class |
Risk |
Indicators |
Mitigation |
| Data poisoning |
Manipulating the training or fine-tuning corpus to degrade or bias the model |
Anomalous contributions to the dataset; quality drift after retraining; duplicate or near-duplicate samples from a single source |
Dataset provenance and access control; curation and review of external sources; anomaly/duplicate detection; reproducibility so you can bisect which batch caused it |
| Model poisoning / backdoor |
Latent malicious behaviour triggered by a specific trigger |
Discrepancy between aggregate metrics (good) and behaviour in specific cases; artifact without provenance |
Only models with provenance and signature; evaluation with adversarial cases (§3.7 → llm-evaluation-standards); controlled retraining on verified data |
| Retrieval corpus poisoning |
Inserting content into the base the system retrieves from |
New documents with embedded instructions; retrieval spikes from a specific source |
Write access control over the corpus; treat retrieved content as untrusted → rag-standards, llm-app-engineering-standards |
| Unauthorised extraction / distillation |
Reconstructing model capability by querying its API en masse |
Anomalous query volume per account; systematic or high-entropy queries; many new accounts with a common pattern; consumption spikes misaligned with product usage |
Rate limits and quotas per user/organisation; anomalous-use detection; identity verification at sign-up; §3.6 |
| Model inversion |
Reconstructing features of the training data from the outputs |
Queries aimed at extracting memorisations; outputs that literally reproduce training fragments |
Minimisation of training data and de-identification → privacy-engineering-standards; limit on output detail; output filtering |
| Membership inference |
Determining whether a specific record was in the training set |
Repeated queries about specific records |
See the realism note below. Minimisation, avoiding overfitting, differential privacy when the risk justifies it → privacy-engineering-standards |
| Adversarial examples |
Perturbed inputs that induce a wrong classification or behaviour |
Error rate concentrated on inputs close to each other; inputs with imperceptible perturbations |
Robustness as a requirement (AI Act art. 15); validation at the edge; out-of-distribution input detection; decision redundancy in critical uses |
Realism note — laboratory versus production. Distinguishing them is mandatory:
- Query-based extraction is a threat demonstrated at industrial scale. In February 2026,
frontier providers disclosed extraction campaigns against their models (on the order of 10⁵
prompts in one campaign, and tens of thousands of fraudulent accounts generating millions of
exchanges in another). This is no longer theoretical: size your rate limits accordingly.
- Membership inference performs far worse than the headlines suggest. Systematic
work on pretrained models finds that most attacks barely beat
chance when pretraining is ~1 epoch (little overfitting). It is materially more
relevant on finely tuned models or with heavily repeated data. Treat it as a real risk
but conditioned on the training regime, not as a universal threat.
- Poisoning and backdoors are amply demonstrated in the laboratory; their
exploitation in production requires access to the pipeline or the artifact — which is exactly what
§3.1–§3.3 protect. The real mitigation is supply-chain, not ML.
3.6 The deployed model as a surface
- Rate limits and quotas per user and per organisation, not just global. The global one does not
stop an extraction campaign distributed across accounts.
- Anomalous-use detection as a first-class control: volume, entropy and systematicity of
the queries, mass correlated account sign-ups, sweep patterns. What is logged and what
alerts: §3.9; rule authorship belongs to
detection-engineering-standards.
- Identity verification and friction at sign-up when the endpoint exposes valuable capability.
- Watermarking: low preventive value. It is a forensic and post hoc instrument:
by the time you detect the mark, the substitute model is already trained. The current literature
classifies it as an attribution and litigation tool, not a defence, and its robustness against
distillation and paraphrasing remains the open problem. Do not sell it internally as protection.
- Protection against unauthorised distillation: the controls that work are the boring ones —
limits, quotas, anomaly detection, enforceable terms of service and correlation between
accounts. Technical defences (output perturbation, distillation resistance) are an active area
of research, not a product. With the model publicly accessible, there is no infallible
barrier: the realistic goal is to raise the cost and detect, not to prevent.
3.7 AI red teaming — methodology
Hard precondition: it runs inside an authorised exercise. Written authorisation,
the RoE, the scope, the window, deconfliction, stop conditions and the report belong to
offensive-security-standards and are not improvised here. What this skill contributes is the
AI-specific methodology.
What distinguishes it from classic pentesting:
- The target is not only code execution: it is the behaviour of the system. A failure can
be an output, not a shell.
- It is not deterministic: an attack that works once may not repeat. A finding without a measured
success rate over N attempts is not a finding, it is an anecdote → the measurement apparatus belongs to
llm-evaluation-standards.
- The surface includes data, weights, prompt, tools and memory, not only network and application.
- Two objectives that overlap but are not the same: safety (harmful content, policy
violation) and security (exfiltration, system compromise, unauthorised use of tools).
Declare which one you are pursuing; the teams, the criteria and the report recipients differ.
Recommended organisation (aligned with industry practice and with the NIST AI RMF cycle
— Govern, Map, Measure, Manage):
- Map: model the system and its risks based on ATLAS and the applicable Top 10 (LLM or ASI).
- Measure: automate breadth — probe sweep in CI on every model deployment.
- Manage: manual, multi-turn depth on what is critical, with a defined periodicity;
mitigation and monitoring in production with a response plan.
- Handoff: every finding becomes a permanent case in the evaluation set
(
llm-evaluation-standards) and, where appropriate, a detection rule
(detection-engineering-standards). A red team whose only result is a PDF is money
burned: half the value is in the regression it leaves installed.
Tools in force (versions in §2):
garak — model-level probe scanner. Breadth and regression, cheap, fits in CI.
Limited agentic and RAG coverage.
- PyRIT (
microsoft/PyRIT, watch out for the repository change) — programmable orchestration of
multi-turn campaigns. Depth on critical applications.
promptfoo in redteam mode — adversarial regression in the PR, with presets mapped to the OWASP
Top 10.
- They never replace systematic measurement. Red teaming is directed exploration; coverage and
the threshold are evaluation.
Forbidden: including in internal documentation ready-to-use payloads, specific jailbreaks for
third-party products or specific guardrail bypasses. What is documented is the class of technique, the
indicator and the mitigation, with a reference to the framework identifier (e.g. the corresponding
ATLAS technique).
3.8 Frameworks
- MITRE ATLAS — the ATT&CK of AI. Knowledge base of tactics, techniques, mitigations and
real cases against AI systems (
AML.T*, AML.M*, AML.CS*).
- Relationship with ATT&CK: it complements, it does not replace. It reuses the model and much of the
tactics of ATT&CK and adds those specific to the AI domain. Both are used together: ATT&CK for
the enterprise part of the intrusion, ATLAS for the AI surface.
- Verified status: since May 2026 the project separated the versioning of content and
format. The content follows a
YYYY.MM.N scheme — latest verified release
v2026.06 (2026-06-30). The data format follows SemVer, v6.0.0, which introduced a
platforms field on all techniques (Predictive AI, Generative AI, Agentic AI,
Enterprise), Pydantic validation schemas and a REST API. Previous releases used conflated
SemVer (v5.6.0, May 2026).
- Practical implication: if your tool or your Navigator layer consumes the old format's
ATLAS.yaml, the move to v6.0.0 affects you. Filter coverage by platforms so you do not
claim agentic coverage you do not have.
- Recent content incorporates real cases from the agentic wave (exfiltration via indirect
injection in productivity assistants, model extraction, AI services as a C2 relay,
RCE in agent framework plugins).
- OWASP GenAI Security Project — two lists, and the right one must be used:
- Top 10 for LLM Applications 2025 (
LLM01:2025–LLM10:2025). Verified: it remains the
edition in force; there is no 2026 edition. Applies to chatbots, copilots and RAG.
- Top 10 for Agentic Applications 2026 (
ASI01–ASI10): Agent Goal Hijack, Tool Misuse &
Exploitation, Agent Identity & Privilege Abuse, Agentic Supply Chain Compromise, Unexpected
Code Execution, Memory & Context Poisoning, Insecure Inter-Agent Communication, Cascading Agent
Failures, Human-Agent Trust Exploitation, Rogue Agents. It extends the LLM one, it does not
replace it: ASI04 covers dynamic composition at runtime where LLM03 covered the static chain
prior to deployment. Agent design against these risks belongs to ai-agents-standards;
here, the supply-chain part (ASI04) and the execution part (ASI05).
- NIST AI RMF — AI RMF 1.0 (
NIST AI 100-1, January 2023) remains the base document;
there is no published 2.0. The generative AI profile NIST AI 600-1 is still the one from
July 2024, unrevised. The 2026 activity is additive: critical infrastructure profile
(concept paper, April 2026), cybersecurity overlays for AI systems (COSAiS,
single- and multi-agent), and a standards initiative for agents via CAISI (February 2026). Its
practical usefulness is the Govern/Map/Measure/Manage cycle as the skeleton of the programme, not as a
technical catalogue: its threat model predates the agentic era.
- AI Act (EU) — security part, art. 15: high-risk systems must reach an
appropriate level of accuracy, robustness and cybersecurity, with the accuracy metrics declared
in the instructions for use, and resilience against attempts to alter their use, outputs or
performance — explicitly citing data and model poisoning, adversarial examples and
confidentiality breaches. The required level is contextual (purpose, state of the art,
risk), not a universal percentage. ⚠️ The timetable is in dispute: the application date
of the high-risk obligations (2 August 2026 under the original text) may have
shifted because of the Digital Omnibus package — verify in the primary source before committing to
a date (§8). The regime, the inventory and risk classification belong to
ai-governance-standards; here only the technical control that satisfies
art. 15.
3.9 Detection and response applied to AI
What is logged (minimum, correlatable and with a defined retention):
- Artifact lifecycle: download or ingestion of a model (source, digest), signature
verification (success and failure), loading of a model into a process, and every write to the
model registry.
- Pipeline: who launched a training run, on which dataset version, with which image, and which
artifact it produced.
- Inference endpoint: caller identity, volume, tokens, latency, tools
invoked, guardrail rejections, and authorisation errors.
- Agent: tool calls, network destinations, irreversible actions and approvals.
What alerts (symptoms, not noise):
- Signature verification failure or loading of an artifact with an unknown digest.
- Write to the model registry outside the authorised pipeline.
- Query pattern consistent with extraction (§3.6) or correlated account sign-ups.
- Jump in the guardrail rejection rate (sign of a jailbreak campaign) or a sudden drop to zero
(sign of a broken or evaded guardrail).
- Egress from the inference process or from the load sandbox towards a disallowed destination.
- Installation in CI of a dependency version outside the pinned list.
The authorship, testing and lifecycle of these rules belong to detection-engineering-standards; the
response and forensics to the compromise, to incident-response-forensics-standards.
3.10 Training pipeline security
- Who can touch the data: minimal and audited access to the dataset; separation between who
contributes data, who trains and who promotes the model.
- Reproducibility as a security control, not only a quality one: without being able to reproduce a
training run you cannot bisect which data batch introduced the anomalous behaviour. Version
dataset, code, configuration and image, and record the triple in the resulting artifact.
- Compute isolation: training and, above all, the loading of unverified models
run in environments without production credentials, without access to the management plane and with
restricted egress. The training node usually has very broad storage credentials:
it is a high-value target.
- Chain of custody of the artifact end to end: training → registry (signed) →
deployment (verified). Every hop, audited.
- Personal data in training: legal basis, minimisation, retention and rights belong to
privacy-engineering-standards. Here only the protection of the store and of access.
4. Quality and CI gates
Increasing order of cost. The first ones break the build.
- Verified dependency pinning (seconds, breaks the build): no GitHub Action, base
image or dependency referenced by mutable tag. It is the control that separated the affected from the
unaffected in §3.3.
- Secret scanning (seconds, breaks the build) →
cicd-standards,
secrets-management-standards.
- SCA of the ML stack (minutes, breaks the build at critical severity): training and
inference dependencies. Triage and SLA:
vulnerability-management-standards.
- Weights format policy (seconds, breaks the build): no artifact in a format with
arbitrary execution enters the registry without an approved and recorded exception. No
trust_remote_code=True in the code.
- Model scanning (
picklescan/modelscan) over every ingested artifact (minutes,
breaks the build). With the right expectation: it detects what is known (§3.2).
- Signature and digest verification before promoting and before loading (seconds, breaks the
deployment).
- Generation and publication of the AIBOM as a release artifact (minutes): inventory, with its
declared gaps.
- Red teaming sweep with
garak on every model deployment (minutes, breaks the
promotion by an agreed threshold) — the threshold and variance management belong to
llm-evaluation-standards.
- Deep multi-turn campaign (PyRIT) with a defined periodicity, within the RoE
(
offensive-security-standards), pre-release for high-risk systems.
- Coverage verification against ATLAS and the applicable Top 10 (periodic review, not a gate):
filtered by
platforms so as not to claim non-existent agentic coverage.
5. Stack security
In addition to everything above, which is already §5 de facto:
- Least-privilege inference runtime: non-root, read-only FS, capabilities
dropped, seccomp, no production credentials in the process that loads the model →
container-runtime-security-standards, kubernetes-standards.
- Filtered egress from the training node and from the inference runtime. It is the control
that turns a malicious load into a failed attempt, and the one that cuts exfiltration.
- Segregation by trust: an unverified third-party model does not share a namespace, node or
credentials with production workloads.
- Zero-trust between services (mTLS, workload identity) in the ML plane as in
any other →
identity-access-management-standards, networking-standards.
- Input and output guardrails at the endpoint: they are defence-in-depth mitigation, not a
perimeter. They are tested with red teaming and their activation rate is watched (§3.9).
- Backups of the model registry and the dataset, encrypted and immutable, with tested restore
→
bcdr-standards, backup-recovery-standards. Retraining from scratch can cost more than
any other state recovery plan.
6. Performance and operability
- Cost of scanning: an artifact of tens or hundreds of GB is not scanned in the critical path
of the deployment. Scan at ingestion into the internal registry, once only, and verify the digest
on every use.
- Internal cache of verified artifacts keyed by digest. It eliminates the download from the public
hub at runtime, which is both a security and an availability risk.
- Red teaming budget: the CI sweep is sized (number of probes × cost per
call) or it will be cut on its own. Deep campaigns are planned by quarter, not by sprint.
- False positives from the model scanner: if the gate produces noise, the exception is documented with
an owner and an expiry. A permanent exception without an owner is the gate disabled with more steps.
- Programme metrics: % of artifacts with provenance and signature, % ingested in a safe format,
time from a model's publication to its verification, ATLAS coverage filtered by
platform, red team findings converted into evaluation cases.
7. Long-term sustainability
- Cadence: review the mapping to ATLAS and the applicable Top 10 every quarter (ATLAS publishes
content monthly); review the maintenance status of the scanning and red teaming tools
every half-year; re-evaluate the model inventory at every release.
- Deprecation: a model no longer served is retired from the registry with its AIBOM archived, it is not
left "just in case" — every live artifact is surface.
- Conscious debt: every exception to the format or signing policy is recorded with a reason,
an owner, a compensating control and an expiry date.
PROHIBITIONS
- ❌ FORBIDDEN to load third-party weights without verifying provenance, digest and signature.
- ❌ FORBIDDEN
trust_remote_code=True without an approved exception, a sandbox and a record. It is
execution of arbitrary third-party code.
- ❌ FORBIDDEN to reference models, images, Actions or dependencies by mutable tag or
latest.
Digest or SHA. It is the direct lesson of the §3.3 incident.
- ❌ FORBIDDEN to load an unverified model outside a sandbox with no network and no credentials. The
load is execution.
- ❌ FORBIDDEN to treat a model scanner as a guarantee. It is a denylist with a
documented history of evasion.
- ❌ FORBIDDEN to treat a public hub's "unsafe" label as a block. It is not one.
- ❌ FORBIDDEN to download weights from a public hub at deployment time. Internal replica.
- ❌ **FORBIDDEN to give production, management plane or model registry credentials to the
process tha
…(truncated)
1---2name: mlsecops-standards3description: Security of the model lifecycle and the AI supply chain. Use when verifying provenance and integrity of third-party model weights, preferring safetensors over pickle-backed formats (.pt, .bin, joblib, Keras Lambda layers) and auditing trust_remote_code, scanning weights with picklescan or modelscan and understanding their evasion limits, signing model artifacts and pinning them by digest in a model registry, triaging an AI-stack supply-chain compromise (the LiteLLM PyPI backdoor and its .pth persistence, a poisoned CI scanner, malicious models or agent skills in a public hub), producing an AIBOM or ML-BOM with CycloneDX or the SPDX AI profile, handling data and model poisoning, model backdoors, query-based extraction and unauthorized distillation, model inversion and membership inference as risk classes with indicators and mitigations, rate limiting and anomalous-use detection on a deployed inference endpoint, hardening the training pipeline and its compute isolation, running AI red teaming with garak, PyRIT 4---56# MLSecOps standards — security of the model lifecycle and the AI supply chain78Criteria verified as of **August 2026**. Re-verify on the web before committing to anything (§8).910## 1. Scope and triggers1112Applies to **securing the model lifecycle and its supply chain**: provenance and integrity of the13weights, serialisation formats and their capacity for arbitrary execution, scanning and signing of14model artifacts, inventory (AIBOM), security of the training pipeline and of the model registry,15the deployed endpoint as an attack surface, the risk classes specific to the lifecycle (poisoning,16backdoors, extraction, inversion, membership inference, adversarial examples), **AI** red teaming17methodology, and the mapping to the frameworks in force (ATLAS, OWASP GenAI, NIST AI RMF, AI Act18art. 15).1920**Posture**: **defensive and authorised, always.** Attacks are described as **risk class,21indicator and mitigation**; never as a reproducible procedure. See §7.2223**Domain thesis**: **a third-party model is a third-party binary.** A weights file downloaded from24a public hub has the same trust profile as an executable pulled off the internet — and, in several25formats in mass use, **it literally executes code when loaded**. Everything that follows derives26from treating it as such: provenance, integrity, signature, scanning, isolation.2728**Second thesis**: **the AI supply chain is the software supply chain, plus the data and the29weights.** It does not replace it: it extends it. The real 2026 incidents were not exotic attacks30against models, but classic compromises of CI and package repositories that reached AI (§3.3).31**The control that removes the most AI risk is still pinning dependencies by digest.**3233Triggers: `.safetensors`, `.pt`, `.pth`, `.bin`, `.ckpt`, `.pkl`, `.gguf`, `.h5`/`.keras`,34`torch.load`, `weights_only`, `pickle`, `joblib.load`, `trust_remote_code=True`, `from_pretrained`,35`picklescan`, `modelscan`, "malicious model", "public hub", "Hugging Face", "model registry",36`MLflow`, "sign the model", "AIBOM", "ML-BOM", "CycloneDX", "SPDX AI profile", "data37poisoning", "data poisoning", "model backdoor", "backdoor", "model extraction",38"unauthorised distillation", "model inversion", "membership inference", "membership39inference", "adversarial example", "AI red team", `garak`, `PyRIT`, `promptfoo redteam`,40`deepteam`, "MITRE ATLAS", `AML.T`, `AML.CS`, "OWASP Top 10 LLM", `ASI01`, "NIST AI RMF",41"AI 600-1", "AI Act article 15", "LiteLLM", "TeamPCP".4243**Not applicable**: see44`llm-app-engineering-standards` (**prompt injection is theirs**, together with treating model45output as untrusted input, structured output and spend limits. **I do not claim it**:46here it appears only as one more technique in the ATLAS catalogue when mapping coverage);47`ai-agents-standards` (agent containment, sandbox, egress, human approval of48irreversible actions, lethal trifecta, OWASP ASI01–ASI10 agentic risks applied to loop design);49`rag-standards` (per-document access control over the retrieved fragments and index50deletion — poisoning of the retrieval corpus is **mitigated** there; here it is a risk51class of the lifecycle);52`mcp-standards` (tool poisoning, rug pull, server shadowing and authorisation of MCP servers);53`claude-api` (**installed skill, canonical reference on the Anthropic side**: nothing about Claude54models —id, price, limits, parameters— is asserted from memory);55`appsec-standards` (**application vulnerability classes** —IDOR, SSRF, XSS, insecure56deserialisation as a category, STRIDE, ASVS— and SAST/DAST/SCA selection. Here, insecure57deserialisation **applied to the model artifact** and the classes specific to the AI lifecycle);58`offensive-security-standards` (**the authorised exercise and its governance are theirs, without59exception**: written authorisation, RoE, scope, window, deconfliction, stop conditions, report,60retest, legal framing. **Precise boundary: if the question is "may I attack this and under what61role?", it is theirs; if it is "what is tested in an AI system and with what method?", it is62mine.** The AI red teaming of §3.7 runs **inside** their RoE, never outside);63`llm-evaluation-standards` (the measurement apparatus. **Boundary declared on64both sides: adversarial attack methodology is mine; the case set, the judge, the rubric,65the threshold and the CI gate it is measured with are theirs.** A red team finding becomes a66case in their eval set: that is the handoff);67`vulnerability-management-standards` (**CVE triage, CVSS/EPSS/KEV, remediation SLA and VEX. The68CVEs of the ML stack —`torch`, `transformers`, inference servers, harnesses— come in there**,69not here);70`cicd-standards` (**SBOM, signing with cosign/Sigstore and SLSA provenance in the pipeline are71theirs**, as is runner hardening and ephemeral identity via OIDC. **Here only what is specific72to the model**: what is signed when the artifact is weights, and what is inventoried when there is73also data);74`cryptography-pki-standards` (the signing primitive, the algorithm choice and the custody and75rotation of keys);76`secrets-management-standards` (manager, ephemeral credentials, rotation — the §3.3 incident is a77case of CI credential theft, and the structural mitigation is theirs);78`container-runtime-security-standards` and `kubernetes-standards` (isolation of the compute where79an untrusted model is loaded: seccomp, capabilities, non-root, admission);80`detection-engineering-standards` (**rule authorship and lifecycle**: Sigma, YARA, tests,81ATT&CK coverage. Here we say **which AI event must be emitted and which anomaly matters**; the rule82is written and governed there);83`incident-response-forensics-standards` (technical and forensic response to the compromise, chain of84custody, eradication and mass credential rotation);85`privacy-engineering-standards` (**personal data in training, model memorisation,86de-identification, data subject rights and how AI systems fit under GDPR/AI Act: already87covered — not duplicated here**. Membership inference appears in §3.5 as a **technical risk88class**; its treatment as a privacy risk is theirs);89`grc-compliance-standards` (management framework, SoA, audit evidence, formal risk90acceptance);91`ai-governance-standards` (AI Act as a regime, policies, inventory of92AI systems and organisational risk management. **Boundary: governance is theirs, the verifiable93technical control is mine.** If the answer is a signed document, it is theirs; if the answer is94running a scan or a signature verification, it is mine);95`mlops-standards` (the operational lifecycle of an in-house model —dataset versioning96with DVC/lakeFS, experiments in MLflow/W&B, model registry with *model cards*, states and97approval, training orchestration, *feature store*, canary deployment and weight rollback,98drift and retraining—. **This skill is its security face**: `mlops` defines the registry and99promotion between environments; I define how that chain is protected, signed, isolated and audited.100Reproducibility is a quality requirement there and a **security control here** (§3.10));101`local-inference-standards` (serving open weights on your own infrastructure: engine, quantisation,102sizing, endpoint security as a port);103`gpu-computing-standards` (the GPU as a resource: driver, MIG/MPS, accelerator isolation);104`data-platform-standards` (where the training data lives and its encryption at rest);105`iac-standards`, `identity-access-management-standards`, `bcdr-standards` (infrastructure,106identity and recovery of the ML environment).107108## 2. Default decisions109110> Verify the latest version and the maintenance status on the web before pinning anything (§8). This111> ecosystem has a high tool mortality rate and **repositories that get archived and moved**.112113| Decision | Default | Reason |114|---|---|---|115| Weights format | **`safetensors`, whenever it exists** | Stores tensors only: no executable code, no deserialisation hooks. It is the only mass-use format designed to eliminate this attack class |116| Formats with arbitrary execution on load | **Vetoed without a sandbox**: pickle and everything wrapping it (`.pkl`, classic PyTorch `.pt`/`.bin`/`.ckpt`, `joblib`), and Keras models with `Lambda` layers | Pickle deserialisation **executes code by design**. The module's own documentation warns about it |117| `torch.load` over a not fully trusted source | **`weights_only=True`** (and verify your version's default) | Restricts what can be deserialised. It mitigates, **it does not eliminate** — there is research on evading restricted loaders |118| `trust_remote_code=True` | **FORBIDDEN by default** | Equivalent to running an arbitrary binary from the internet. There is a recent RCE CVE from hardcoding it (§3.2) |119| Source of the weights | **Internal replica with pinned digest**, not a direct hub download at runtime | A pull at deployment time against a public hub is a mutable dependency in the critical path |120| Artifact identification | **Cryptographic digest (SHA-256), never a tag or "latest"** | Same principle as container images. A tag is mutable |121| Signing and verification | **cosign/Sigstore over the model artifact**, verification **before loading** | Mechanics and key custody: `cicd-standards` and `cryptography-pki-standards` |122| Model scanning | **`picklescan` and/or `modelscan` as a gate**, with calibrated expectations (§3.2) | They are denylists: they detect what is known. **Necessary, insufficient** |123| Loading an unverified model | **Sandbox with no network and no credentials**, unprivileged user, read-only FS | Loading **is** execution. Containment: `container-runtime-security-standards` |124| Inventory | **AIBOM/ML-BOM in CycloneDX** for the pipeline; SPDX AI profile when the recipient is regulatory | See §3.4 on real maturity |125| Threat framework | **MITRE ATLAS** as the reference taxonomy, **complementing ATT&CK** | §3.8 |126| Application risk catalogue | **OWASP Top 10 for LLM Applications 2025** + **Top 10 for Agentic Applications 2026 (ASI01–ASI10)** | §3.8 |127| AI red teaming | **`garak`** (broad sweep, CI) + **PyRIT** (multi-turn campaigns) + **promptfoo redteam** (in PR) | §3.7 |128129### Verified status of the tools (August 2026)130131| Tool | Version | Date | Notes |132|---|---|---|---|133| `garak` (NVIDIA) | 0.15.1 | 2026-06-05 | **Alive**, active repo. Model-level probe scanner |134| **PyRIT (Microsoft)** | v1.0.1 | 2026-07-30 | ⚠️ **The repository moved**: `Azure/PyRIT` is **archived** (2026-03-27). The active one is **`microsoft/PyRIT`**. Any tutorial pointing at `Azure/PyRIT` is obsolete |135| `promptfoo` (redteam mode) | 0.121.20 | 2026-07-31 | Alive, high cadence. Governance: verify the acquisition by OpenAI (§8) |136| `picklescan` | 1.0.5 | 2026-07-01 | Alive. Integrated into the Hugging Face scanning pipeline |137| `modelscan` (Protect AI) | 0.8.8 | 2026-02-18 | ⚠️ **~6 months without a release.** Use it, but pin the version and do not take it as the sole control |138| `deepteam` | v1.0.4 | 2025-11-12 | ⚠️ **No releases in ~9 months.** Do not adopt for new work without re-evaluating |139| CycloneDX (spec) | 1.7 (2025-10-21), patches 1.7.1 (2026-06-02) | — | ECMA-424; verify the edition in force (§8) |140| SPDX (spec) | 3.0.1 (2024-12-17), **3.1-RC1** (2026-01-24) | — | The AI Profile lives in the 3.x line. ISO/IEC 5962:2021 codifies **SPDX 2.2.1**, not the current one |141| MITRE ATLAS | content **v2026.06** (2026-06-30), format **v6.0.0** | — | §3.8 |142143## 3. Structure and conventions144145### 3.1 The model as a software artifact146147The three non-negotiable controls, in order:1481491. **Provenance**: where it came from, who published it, which exact version, against which data it150 was trained (as far as that is knowable), and which licence it carries. Without recorded151 provenance there is no possible answer to "are we affected?" when the hub pulls a model.1522. **Integrity**: digest computed at ingestion and verified before each load. **It is always153 referenced by digest**, never by name or by tag.1543. **Signature**: the artifact is signed on entry into the internal registry and **the signature is155 verified before loading**, not only at deployment. A verification that only happens in the156 pipeline does not protect against substitution of the file in the store.157158**Operational corollary**: the model registry is a **production-grade asset**, not a drawer of159`.ckpt` in a shared bucket. Least-privilege access control, audited writes, immutability of160published versions, and separation of who trains, who promotes and who deploys. An attacker with161write access to the registry needs no AI attack at all: they replace the file.162163### 3.2 Third-party weights = third-party binaries164165**Formats that execute code on load** (risk class: insecure deserialisation /166arbitrary execution):167168- **Pickle and everything built on top of it**: `.pkl`, the classic PyTorch `.pt`/`.bin`/`.ckpt`,169 `joblib`. The mechanism is the protocol's own reconstruction hook — not a bug, **the170 design**. The pickle interpreter processes opcodes as they arrive, **without first validating that171 the file is intact**, which enables evasion techniques based on deliberately corrupted files.172- **Keras/TensorFlow with `Lambda` layers**: they execute arbitrary Python code embedded in the model.173 The problem is not exclusive to pickle.174- **`trust_remote_code=True`**: downloads and executes code from the model's repository. There is a175 recent CVE (**CVE-2026-6859**, InstructLab, verify) from hardcoding it in a training176 script: a malicious model on the hub was enough to achieve RCE on any user. **It177 requires no scanner evasion whatsoever: it is the functionality doing its job.**178179**Recommended safe format**: **`safetensors`** — it stores tensor data only, with no code180or deserialisation hooks. It is the default choice and, when a model is published only in181pickle format, that is in itself a risk signal to evaluate (and, if accepted, the conversion to182safetensors happens **inside the sandbox**, not on the engineer's workstation).183184**Model scanning — necessary, insufficient, and it must be said:**185186- `picklescan` and `modelscan` work by **denylist** of dangerous functions. They detect187 what is known.188- Verified history of evasion: JFrog reported **three 0-days in `picklescan`** (fixed in189 0.0.31, Sept 2025), each one allowing detection to be bypassed; the **nullifAI** technique190 (ReversingLabs) evaded the hub's scanning with a deliberately broken pickle; and recent academic191 work (**ShadowPickle**) reports evasion of **ten scanners and four model hubs**.192- **A public hub that flags a model as "unsafe" normally does not block it**: it lets you193 download and run it at your own risk. The label is not a control.194- Consequence: **the scanner is a hygiene gate, not a guarantee.** The control that really195 bounds the damage is **isolation at load time** plus **preferring safetensors**.196197**Verified real incidents** (use as an argument, not as an anecdote): malicious models have been198found on public hubs that open a reverse shell to an external IP when loaded; a year-on-year199increase on the order of **5×** in the upload rate of malicious models is reported; and in200February 2026, **341 malicious *skills*** were detected in a public registry of agent skills201distributing an infostealer. **The public repository of AI artifacts is today an active malware202distribution vector.**203204### 3.3 The AI supply chain — the didactic case of 2026205206The compromise of **LiteLLM on PyPI (March 2026)** is the canonical example that **the AI supply207chain breaks where any software supply chain breaks**. Verified chain:2082091. **19 March**: the actor (**TeamPCP**) compromises the GitHub Actions of **Trivy** — an open-source210 security scanner. Since most pipelines reference Actions by **mutable tag211 instead of pinned commit SHA**, the organisations running Trivy in CI started212 executing the malicious code immediately.2132. Among the harvested credentials was the **LiteLLM PyPI publishing token**, whose214 pipeline invoked Trivy in a secret- and CVE-scanning script.2153. **24 March**: `litellm` **1.82.7** and **1.82.8** are published with a malicious payload. 1.82.8,216 ~13 minutes later, adds **persistence via a `.pth` file** (`litellm_init.pth`).2174. **The `.pth` mechanism is what must be understood**: Python's `site` module **executes** the218 contents of any `.pth` in `site-packages` during interpreter initialisation —219 **before any `import` and before any application code**. There is no need to import220 the library: `python --version` is enough to trigger it. The payload was doubly base64-encoded221 to reduce visibility to basic static analysis.2225. **Payload**: credential harvesting (more than 50 categories: SSH keys, AWS with IMDSv2 and Secrets223 Manager, GCP, Azure, Kubernetes, `.env`, shell history, git credentials, Docker224 registries, Terraform state), **lateral movement in Kubernetes** (reading secrets in all225 namespaces, creating privileged pods mounting the host filesystem) and a persistent226 backdoor via a user systemd service. Encrypted exfiltration (AES-256-CBC with a session key227 wrapped in RSA-4096) to a domain not affiliated with the project.2286. **Window**: ~40 minutes until quarantine on PyPI. Tens of thousands of installations are reported229 in that interval. The campaign continued with `telnyx` (27 March) and other registries.2307. **Who was not affected**: the deployments that **pinned dependencies** in a `requirements.txt`231 inside the official image.232233**Lessons that become controls, not anecdotes:**234235- **Pin by digest/SHA, not by tag**, for GitHub Actions as well as dependencies and base236 images. It is the control that separated the affected from the unaffected.237- **Installing a package is executing code.** There is no "install and then review".238- **A security scanner in your CI is a dependency with credentials.** The compromised link239 was a *defensive* tool. Apply the same criteria to it as to any other dependency.240- **Ephemeral publishing credentials** (OIDC / *trusted publishing*) instead of long-lived static241 tokens → `cicd-standards`, `secrets-management-standards`.242- **A host or CI job that installed the compromised artifact is treated as full credential243 exposure**, not as "check whether the package is present": mass rotation, hunting for244 persistence and review of activity in Kubernetes → `incident-response-forensics-standards`.245- The **catalogue precedent** is consistent: the CVEs and maintenance status of the ML stack246 (`torch`, `transformers`, inference servers, harnesses) are governed by247 `vulnerability-management-standards`; what is AI-specific is that **the weights and the data are two248 more links**, and no classic SCA covers them.249250### 3.4 AIBOM / ML-BOM — real status, without optimism251252A classic SBOM inventories neither weights, nor training data, nor the provenance of the model.253Hence the AIBOM. **Honest status as of August 2026: it is an emerging standard, not a mature one.**254255- **CycloneDX** supports ML-BOM/AI-BOM and is the practical option for CI. Spec **1.7** published256 2025-10-21 (with 1.7.x patches in 2026), adopted as **ECMA-424**. It is the line with the most257 traction in tooling.258- **SPDX 3.x** defines an **AI Profile** and a **Dataset Profile** (model type, training259 method, data handling, explainability, limitations, energy consumption). Status:260 **3.0.1** (Dec 2024) with **3.1-RC1** (Jan 2026). **Warning**: the ISO/IEC 5962:2021 standard codifies261 **SPDX 2.2.1**, not the version with the AI profile — citing "SPDX is ISO" as proof of AIBOM262 maturity is incorrect.263- **Field reality**: the tool generates what the upstream model declared. If the publisher did not264 document data or licence, **the AIBOM comes out with gaps** — some generators explicitly score265 that incompleteness, and that score is the useful datum.266267**Criteria**: generate an AIBOM because inventory is a prerequisite for everything else (and because268procurement is starting to demand it), but **do not treat it as a security control**. It is an269inventory, and an inventory with declared gaps. The control is still digest + signature + scan + sandbox.270271### 3.5 Lifecycle attacks — risk class, indicator, mitigation272273**Described as risk. Never as a recipe.**274275| Class | Risk | Indicators | Mitigation |276|---|---|---|---|277| **Data poisoning** | Manipulating the training or fine-tuning corpus to degrade or bias the model | Anomalous contributions to the dataset; quality drift after retraining; duplicate or near-duplicate samples from a single source | Dataset provenance and access control; curation and review of external sources; anomaly/duplicate detection; **reproducibility so you can bisect which batch caused it** |278| **Model poisoning / backdoor** | Latent malicious behaviour triggered by a specific trigger | Discrepancy between aggregate metrics (good) and behaviour in specific cases; artifact without provenance | Only models with provenance and signature; evaluation with adversarial cases (§3.7 → `llm-evaluation-standards`); controlled retraining on verified data |279| **Retrieval corpus poisoning** | Inserting content into the base the system retrieves from | New documents with embedded instructions; retrieval spikes from a specific source | Write access control over the corpus; treat retrieved content as untrusted → `rag-standards`, `llm-app-engineering-standards` |280| **Unauthorised extraction / distillation** | Reconstructing model capability by querying its API en masse | Anomalous query volume per account; systematic or high-entropy queries; **many new accounts with a common pattern**; consumption spikes misaligned with product usage | Rate limits and quotas per user/organisation; anomalous-use detection; identity verification at sign-up; §3.6 |281| **Model inversion** | Reconstructing features of the training data from the outputs | Queries aimed at extracting memorisations; outputs that literally reproduce training fragments | Minimisation of training data and de-identification → `privacy-engineering-standards`; limit on output detail; output filtering |282| **Membership inference** | Determining whether a specific record was in the training set | Repeated queries about specific records | **See the realism note below.** Minimisation, avoiding overfitting, differential privacy when the risk justifies it → `privacy-engineering-standards` |283| **Adversarial examples** | Perturbed inputs that induce a wrong classification or behaviour | Error rate concentrated on inputs close to each other; inputs with imperceptible perturbations | Robustness as a requirement (AI Act art. 15); validation at the edge; out-of-distribution input detection; decision redundancy in critical uses |284285**Realism note — laboratory versus production.** Distinguishing them is mandatory:286287- **Query-based extraction is a threat demonstrated at industrial scale.** In February 2026,288 frontier providers disclosed extraction campaigns against their models (on the order of **10⁵289 prompts** in one campaign, and tens of thousands of fraudulent accounts generating millions of290 exchanges in another). **This is no longer theoretical: size your rate limits accordingly.**291- **Membership inference performs far worse than the headlines suggest.** Systematic292 work on pretrained models finds that most attacks **barely beat293 chance** when pretraining is ~1 epoch (little overfitting). It is **materially more294 relevant on finely tuned models or with heavily repeated data**. Treat it as a real risk295 but **conditioned on the training regime**, not as a universal threat.296- **Poisoning and backdoors are amply demonstrated in the laboratory**; their297 exploitation in production requires access to the pipeline or the artifact — which is exactly what298 §3.1–§3.3 protect. **The real mitigation is supply-chain, not ML.**299300### 3.6 The deployed model as a surface301302- **Rate limits and quotas per user and per organisation**, not just global. The global one does not303 stop an extraction campaign distributed across accounts.304- **Anomalous-use detection** as a first-class control: volume, entropy and systematicity of305 the queries, mass correlated account sign-ups, sweep patterns. What is logged and what306 alerts: §3.9; rule authorship belongs to `detection-engineering-standards`.307- **Identity verification and friction at sign-up** when the endpoint exposes valuable capability.308- **Watermarking: low preventive value.** It is a **forensic and post hoc** instrument:309 by the time you detect the mark, the substitute model is already trained. The current literature310 classifies it as an **attribution and litigation** tool, not a defence, and its robustness against311 distillation and paraphrasing remains the open problem. **Do not sell it internally as protection.**312- **Protection against unauthorised distillation**: the controls that work are the boring ones —313 limits, quotas, anomaly detection, enforceable terms of service and correlation between314 accounts. Technical defences (output perturbation, distillation resistance) are an active area315 of research, **not a product**. With the model publicly accessible, **there is no infallible316 barrier**: the realistic goal is to **raise the cost and detect**, not to prevent.317318### 3.7 AI red teaming — methodology319320**Hard precondition**: it runs inside an authorised exercise. **Written authorisation,321the RoE, the scope, the window, deconfliction, stop conditions and the report belong to322`offensive-security-standards` and are not improvised here.** What this skill contributes is the323AI-specific methodology.324325**What distinguishes it from classic pentesting:**326327- The target is not only code execution: it is **the behaviour of the system**. A failure can328 be an output, not a shell.329- **It is not deterministic**: an attack that works once may not repeat. A finding without a measured330 success rate over N attempts is not a finding, it is an anecdote → the measurement apparatus belongs to331 `llm-evaluation-standards`.332- The surface includes **data, weights, prompt, tools and memory**, not only network and application.333- **Two objectives that overlap but are not the same**: *safety* (harmful content, policy334 violation) and *security* (exfiltration, system compromise, unauthorised use of tools).335 Declare which one you are pursuing; the teams, the criteria and the report recipients differ.336337**Recommended organisation** (aligned with industry practice and with the NIST AI RMF cycle338— *Govern, Map, Measure, Manage*):3393401. **Map**: model the system and its risks based on ATLAS and the applicable Top 10 (LLM or ASI).3412. **Measure**: automate breadth — probe sweep in CI on every model deployment.3423. **Manage**: manual, multi-turn depth on what is critical, with a defined periodicity;343 mitigation and monitoring in production with a response plan.3444. **Handoff**: **every finding becomes a permanent case in the evaluation set**345 (`llm-evaluation-standards`) and, where appropriate, a detection rule346 (`detection-engineering-standards`). **A red team whose only result is a PDF is money347 burned**: half the value is in the regression it leaves installed.348349**Tools in force** (versions in §2):350351- **`garak`** — model-level probe scanner. Breadth and regression, cheap, fits in CI.352 Limited agentic and RAG coverage.353- **PyRIT** (`microsoft/PyRIT`, **watch out for the repository change**) — programmable orchestration of354 multi-turn campaigns. Depth on critical applications.355- **`promptfoo` in redteam mode** — adversarial regression in the PR, with presets mapped to the OWASP356 Top 10.357- **They never replace systematic measurement.** Red teaming is directed exploration; coverage and358 the threshold are evaluation.359360**Forbidden**: including in internal documentation ready-to-use payloads, specific jailbreaks for361third-party products or specific guardrail bypasses. What is documented is **the class of technique, the362indicator and the mitigation**, with a reference to the framework identifier (e.g. the corresponding363ATLAS technique).364365### 3.8 Frameworks366367- **MITRE ATLAS** — the ATT&CK of AI. Knowledge base of tactics, techniques, mitigations and368 real cases against AI systems (`AML.T*`, `AML.M*`, `AML.CS*`).369 - **Relationship with ATT&CK: it complements, it does not replace.** It reuses the model and much of the370 tactics of ATT&CK and adds those specific to the AI domain. **Both are used together**: ATT&CK for371 the enterprise part of the intrusion, ATLAS for the AI surface.372 - **Verified status**: since May 2026 the project **separated the versioning of content and373 format**. The **content** follows a `YYYY.MM.N` scheme — latest verified release374 **v2026.06** (2026-06-30). The **data format** follows SemVer, **v6.0.0**, which introduced a375 `platforms` field on all techniques (`Predictive AI`, `Generative AI`, **`Agentic AI`**,376 `Enterprise`), Pydantic validation schemas and a REST API. Previous releases used conflated377 SemVer (v5.6.0, May 2026).378 - **Practical implication**: if your tool or your Navigator layer consumes the old format's379 `ATLAS.yaml`, **the move to v6.0.0 affects you**. Filter coverage by `platforms` so you do not380 claim agentic coverage you do not have.381 - Recent content incorporates real cases from the agentic wave (exfiltration via indirect382 injection in productivity assistants, model extraction, AI services as a C2 relay,383 RCE in agent framework plugins).384- **OWASP GenAI Security Project** — **two lists, and the right one must be used**:385 - **Top 10 for LLM Applications 2025** (`LLM01:2025`–`LLM10:2025`). **Verified: it remains the386 edition in force; there is no 2026 edition.** Applies to chatbots, copilots and RAG.387 - **Top 10 for Agentic Applications 2026** (`ASI01`–`ASI10`): Agent Goal Hijack, Tool Misuse &388 Exploitation, Agent Identity & Privilege Abuse, **Agentic Supply Chain Compromise**, Unexpected389 Code Execution, Memory & Context Poisoning, Insecure Inter-Agent Communication, Cascading Agent390 Failures, Human-Agent Trust Exploitation, Rogue Agents. It **extends** the LLM one, it does not391 replace it: ASI04 covers dynamic composition at runtime where LLM03 covered the static chain392 prior to deployment. Agent design against these risks belongs to `ai-agents-standards`;393 here, the supply-chain part (ASI04) and the execution part (ASI05).394- **NIST AI RMF** — **AI RMF 1.0** (`NIST AI 100-1`, January 2023) remains the base document;395 **there is no published 2.0**. The generative AI profile **`NIST AI 600-1` is still the one from396 July 2024, unrevised**. The 2026 activity is **additive**: critical infrastructure profile397 (concept paper, April 2026), cybersecurity overlays for AI systems (COSAiS,398 single- and multi-agent), and a standards initiative for agents via CAISI (February 2026). **Its399 practical usefulness is the Govern/Map/Measure/Manage cycle as the skeleton of the programme**, not as a400 technical catalogue: its threat model predates the agentic era.401- **AI Act (EU) — security part, art. 15**: high-risk systems must reach an402 appropriate level of **accuracy, robustness and cybersecurity**, with the accuracy metrics **declared403 in the instructions for use**, and resilience against attempts to alter their use, outputs or404 performance — **explicitly citing data and model poisoning, adversarial examples and405 confidentiality breaches**. The required level is **contextual** (purpose, state of the art,406 risk), not a universal percentage. ⚠️ **The timetable is in dispute**: the application date407 of the high-risk obligations (2 August 2026 under the original text) may have408 shifted because of the *Digital Omnibus* package — **verify in the primary source before committing to409 a date** (§8). **The regime, the inventory and risk classification belong to410 `ai-governance-standards`; here only the technical control that satisfies411 art. 15.**412413### 3.9 Detection and response applied to AI414415**What is logged** (minimum, correlatable and with a defined retention):416417- **Artifact lifecycle**: download or ingestion of a model (source, digest), signature418 verification (success **and failure**), loading of a model into a process, and **every write to the419 model registry**.420- **Pipeline**: who launched a training run, on which dataset version, with which image, and which421 artifact it produced.422- **Inference endpoint**: caller identity, volume, tokens, latency, tools423 invoked, guardrail rejections, and authorisation errors.424- **Agent**: tool calls, network destinations, irreversible actions and approvals.425426**What alerts** (symptoms, not noise):427428- Signature verification failure or **loading of an artifact with an unknown digest**.429- Write to the model registry outside the authorised pipeline.430- Query pattern consistent with extraction (§3.6) or correlated account sign-ups.431- Jump in the guardrail rejection rate (sign of a jailbreak campaign) **or a sudden drop to zero**432 (sign of a broken or evaded guardrail).433- Egress from the inference process or from the load sandbox towards a disallowed destination.434- Installation in CI of a dependency version outside the pinned list.435436The **authorship, testing and lifecycle** of these rules belong to `detection-engineering-standards`; the437**response and forensics** to the compromise, to `incident-response-forensics-standards`.438439### 3.10 Training pipeline security440441- **Who can touch the data**: minimal and audited access to the dataset; separation between who442 contributes data, who trains and who promotes the model.443- **Reproducibility as a security control**, not only a quality one: without being able to reproduce a444 training run you cannot bisect which data batch introduced the anomalous behaviour. Version445 dataset, code, configuration and image, and record the triple in the resulting artifact.446- **Compute isolation**: training and, above all, **the loading of unverified models**447 run in environments without production credentials, without access to the management plane and with448 restricted egress. The training node usually has very broad storage credentials:449 it is a high-value target.450- **Chain of custody of the artifact** end to end: training → registry (signed) →451 deployment (verified). Every hop, audited.452- **Personal data in training**: legal basis, minimisation, retention and rights belong to453 `privacy-engineering-standards`. Here only the protection of the store and of access.454455## 4. Quality and CI gates456457Increasing order of cost. The first ones break the build.4584591. **Verified dependency pinning** (seconds, **breaks the build**): no GitHub Action, base460 image or dependency referenced by mutable tag. It is the control that separated the affected from the461 unaffected in §3.3.4622. **Secret scanning** (seconds, **breaks the build**) → `cicd-standards`,463 `secrets-management-standards`.4643. **SCA of the ML stack** (minutes, **breaks the build** at critical severity): training and465 inference dependencies. Triage and SLA: `vulnerability-management-standards`.4664. **Weights format policy** (seconds, **breaks the build**): no artifact in a format with467 arbitrary execution enters the registry without an approved and recorded exception. No468 `trust_remote_code=True` in the code.4695. **Model scanning** (`picklescan`/`modelscan`) over every ingested artifact (minutes,470 **breaks the build**). With the right expectation: it detects what is known (§3.2).4716. **Signature and digest verification before promoting and before loading** (seconds, **breaks the472 deployment**).4737. **Generation and publication of the AIBOM** as a release artifact (minutes): inventory, with its474 declared gaps.4758. **Red teaming sweep** with `garak` on every model deployment (minutes, **breaks the476 promotion** by an agreed threshold) — the threshold and variance management belong to477 `llm-evaluation-standards`.4789. **Deep multi-turn campaign** (PyRIT) with a defined periodicity, within the RoE479 (`offensive-security-standards`), **pre-release** for high-risk systems.48010. **Coverage verification against ATLAS and the applicable Top 10** (periodic review, not a gate):481 filtered by `platforms` so as not to claim non-existent agentic coverage.482483## 5. Stack security484485In addition to everything above, which is already §5 de facto:486487- **Least-privilege inference runtime**: non-root, read-only FS, capabilities488 dropped, seccomp, no production credentials in the process that loads the model →489 `container-runtime-security-standards`, `kubernetes-standards`.490- **Filtered egress** from the training node and from the inference runtime. It is the control491 that turns a malicious load into a failed attempt, and the one that cuts exfiltration.492- **Segregation by trust**: an unverified third-party model does not share a namespace, node or493 credentials with production workloads.494- **Zero-trust between services** (mTLS, workload identity) in the ML plane as in495 any other → `identity-access-management-standards`, `networking-standards`.496- **Input and output guardrails** at the endpoint: they are defence-in-depth mitigation, **not a497 perimeter**. They are tested with red teaming and their activation rate is watched (§3.9).498- **Backups of the model registry and the dataset**, encrypted and immutable, with tested restore499 → `bcdr-standards`, `backup-recovery-standards`. Retraining from scratch can cost more than500 any other state recovery plan.501502## 6. Performance and operability503504- **Cost of scanning**: an artifact of tens or hundreds of GB is not scanned in the critical path505 of the deployment. Scan **at ingestion** into the internal registry, once only, and **verify the digest**506 on every use.507- **Internal cache of verified artifacts** keyed by digest. It eliminates the download from the public508 hub at runtime, which is both a security and an availability risk.509- **Red teaming budget**: the CI sweep is sized (number of probes × cost per510 call) or it will be cut on its own. Deep campaigns are planned by quarter, not by sprint.511- **False positives from the model scanner**: if the gate produces noise, the exception is documented with512 an owner and an expiry. A permanent exception without an owner is the gate disabled with more steps.513- **Programme metrics**: % of artifacts with provenance and signature, % ingested in a safe format,514 time from a model's publication to its verification, ATLAS coverage filtered by515 platform, red team findings converted into evaluation cases.516517## 7. Long-term sustainability518519- **Cadence**: review the mapping to ATLAS and the applicable Top 10 every quarter (ATLAS publishes520 content **monthly**); review the maintenance status of the scanning and red teaming tools521 every half-year; re-evaluate the model inventory at every release.522- **Deprecation**: a model no longer served is retired from the registry with its AIBOM archived, it is not523 left "just in case" — every live artifact is surface.524- **Conscious debt**: every exception to the format or signing policy is recorded with a reason,525 an owner, a compensating control and an **expiry date**.526527### PROHIBITIONS528529- ❌ **FORBIDDEN to load third-party weights without verifying provenance, digest and signature.**530- ❌ **FORBIDDEN `trust_remote_code=True`** without an approved exception, a sandbox and a record. It is531 execution of arbitrary third-party code.532- ❌ **FORBIDDEN to reference models, images, Actions or dependencies by mutable tag or `latest`.**533 Digest or SHA. It is the direct lesson of the §3.3 incident.534- ❌ **FORBIDDEN to load an unverified model outside a sandbox** with no network and no credentials. The535 load is execution.536- ❌ **FORBIDDEN to treat a model scanner as a guarantee.** It is a denylist with a537 documented history of evasion.538- ❌ **FORBIDDEN to treat a public hub's "unsafe" label as a block.** It is not one.539- ❌ **FORBIDDEN to download weights from a public hub at deployment time.** Internal replica.540- ❌ **FORBIDDEN to give production, management plane or model registry credentials to the541 process tha542543…(truncated)