In Toto

in-toto in Cloud-Native Engineering

Purpose and Use Cases

What Problem Does It Solve?

• Software supply chain security: in-toto provides end-to-end integrity for software artifacts throughout their lifecycle
• Provenance verification: Cryptographically verify that artifacts were built according to defined supply chain steps
• Build pipeline assurance: Ensure no unauthorized changes occurred during build, test, or deployment processes
• Attack surface reduction: Prevent supply chain attacks like compromised build tools or malicious code injection

When to Use

• High-security applications: Financial systems, healthcare, government, or critical infrastructure
• Compliance requirements: Auditable software development for SOC2, PCI-DSS, HIPAA, or FedRAMP
• Multi-vendor supply chains: When multiple organizations contribute to software delivery
• Open source governance: Verify third-party dependencies haven't been tampered with
• Regulatory compliance: When you need to prove software integrity to auditors

Key Use Cases

• Build pipeline verification: Confirm all build steps executed as defined
• Artifact provenance: Track origin and transformation history of software artifacts
• Release integrity: Ensure release artifacts haven't been modified after signing
• Continuous verification: Integrate in-toto checks into CI/CD pipelines
• Attack detection: Detect compromised build tools or malicious code injection

Architecture Design Patterns

Core Components

In-toto Layout

{
  "_type": "layout",
  "expires": "2025-12-31T23:59:59Z",
  "keys": {
    "alice-key-id": {
      "keytype": "rsa",
      "keyid_hash_algorithms": ["sha256"],
      "keyval": {
        "public": "-----BEGIN PUBLIC KEY-----..."
      }
    }
  },
  "roles": [
    {
      "name": "root",
      "threshold": 1,
      "keyids": ["alice-key-id"]
    },
    {
      "name": "staging",
      "threshold": 2,
      "keyids": ["alice-key-id", "bob-key-id"]
    },
    {
      "name": "production",
      "threshold": 3,
      "keyids": ["alice-key-id", "bob-key-id", "carol-key-id"]
    }
  ],
  "steps": [
    {
      "name": "build",
      "expected_artifact": "app-image.tar.gz",
      "expected_command": ["make", "build"],
      "pubkeys": ["alice-key-id"],
      "threshold": 1
    },
    {
      "name": "test",
      "expected_artifact": "app-image.tar.gz",
      "expected_command": ["make", "test"],
      "pubkeys": ["bob-key-id"],
      "threshold": 1
    }
  ],
  "inspect": [
    {
      "name": "verify-integrity",
      "command": ["in-toto-run", "--", "sha256sum", "app-image.tar.gz"],
      "expected_materials": [["MATCH", "app-image.tar.gz", "WITH", "PATH", "app-image.tar.gz"]],
      "expected_products": []
    }
  ]
}

• Layout definition: Root policy for supply chain verification
• Roles and thresholds: Multiple signatures required for critical steps
• Steps definition: Build steps with expected artifacts and commands
• Inspections: Post-build verification commands

Link Metadata

{
  "_type": "link",
  "name": "build",
  "material": {
    "source-code.tar.gz": {
      "sha256": "abc123..."
    }
  },
  "product": {
    "app-image.tar.gz": {
      "sha256": "def456..."
    }
  },
  "command": ["make", "build"],
  "byproducts": {},
  "duration": 42
}

• Step execution record: Documents a single supply chain step
• Materials: Input artifacts with cryptographic hashes
• Products: Output artifacts with cryptographic hashes
• Evidence: Commands executed and timing information

Verification Tool (in-toto-verifier)

• Layout validation: Verify layout file structure and signatures
• Link verification: Check all link files exist and are valid
• Hash matching: Verify product hashes match materials through the pipeline
• Signature verification: Ensure all required roles signed

Component Interactions

Builder (Alice)
    ↓ (builds artifact + creates link)
Link Metadata
    ↓ (signed with Alice's key)
In-toto Layout
    ↓ (defines verification policy)
Verifier (CI/CD)
    ↓ (validates all steps)
Artifact Release

Data Flow Patterns

Build Pipeline Flow

1. Developer pushes code
2. CI system triggers build
3. Build step creates link with materials/products
4. Link signed by build role key
5. Next step reads link as input material
6. Repeat for each pipeline step
7. Final artifact verified against layout
8. Signed metadata included with release

Verification Flow

1. Download artifact + in-toto metadata
2. Load layout policy
3. Verify all link signatures
4. Verify link chain: products of step N = materials of step N+1
5. Execute inspections
6. Report verification result

Design Principles

Non-Repudiation

• All supply chain steps must be signed by designated roles
• Key compromise detection through multiple signature requirements
• Immutable link metadata prevents post-hoc modifications

Least Privilege

• Each role signs only for authorized steps
• Key separation limits blast radius of compromise
• Threshold signatures prevent single-point failures

Verifiability

• All cryptographic material publicly auditable
• Self-contained metadata with no external dependencies
• Standard format enables interoperability

Integration Approaches

Integration with Other CNCF Projects

Grafeas Integration

# Grafeas Note for in-toto policy
apiVersion: grafeas.io/v1
kind: Note
metadata:
  name: projects/myproject/notes/in-toto-policy
spec:
  in_toto: {
    "step": {
      "name": "build",
      "threshold": 1
    }
  }

• Provenance tracking: Grafeas stores in-toto metadata
• Policy enforcement: Grafeas policies validate in-toto signatures
• Compliance reporting: Audit trail of software provenance

Tekton Chains Integration

apiVersion: chains.tekton.dev/v1beta1
kind: Configuration
metadata:
  name: chains
spec:
  artifacts:
    in_toto:
      predicate_type: in-toto.io/Link
      enabled: true
      keyless:
        subject: "https://tkn.dev/chains"

• Automatic link generation: Tekton pipelines create in-toto links
• Keyless signing: Use Fulcio for certificate-based signing
• Metadata storage: Store in-toto metadata in artifact registry

Cosign Integration

# Sign in-toto metadata with cosign
cosign sign-blob --output-signature=link.sig link.json

# Verify in-toto policy
cosign verify-blob --signature=link.sig \
  --certificate=link.pem \
  --certificate-identity=build@example.com \
  link.json

• Cryptographic signing: Cosign signs in-toto metadata
• Certificate-based signing: Use Fulcio for keyless signing
• Policy enforcement: Rekor provides transparency log verification

Kyverno Integration

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-in-toto
spec:
  rules:
    - name: check-in-toto
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Container images must have in-toto attestation"
        deny:
          conditions:
            - key: "{{ request.object.spec.containers[].image }}"
              operator: AnyNotIn
              value: "{{ 'images' | length(@) > 0 }}"

• Policy enforcement: Kyverno policies require in-toto attestation
• Admission control: Block pods without valid in-toto metadata
• Automated remediation: Policies can automatically add missing metadata

API Patterns

Layout API

• Layout definition: JSON Schema for supply chain policy
• Key management: RSA, ECDSA, Ed25519 key support
• Role management: Define roles with signature thresholds

Link API

• Link creation: Generate link metadata for each step
• Link signing: Sign links with cryptographic keys
• Link verification: Verify links against layout policy

Configuration Patterns

Layout Configuration

{
  "expires": "2025-12-31T00:00:00Z",
  "keys": {},
  "roles": {
    "root": {
      "threshold": 1,
      "keyids": []
    }
  },
  "steps": [
    {
      "name": "checkout",
      "pubkeys": [],
      "threshold": 1,
      "expected_artifacts": [],
      "expected_command": []
    }
  ]
}

Key Generation

# Generate in-toto key
in-toto-keygen -t ed25519 -u alice

# Export public key
cat alice.pub

# Import public key into layout
in-toto-sign -f layout.json -p alice.pub

Extension Mechanisms

Custom Predicate Types

• SLSA Provenance: Use in-toto with SLSA provenance predicate
• Cosign Attestations: Integrate with cosign for keyless signing
• Custom predicates: Define organization-specific predicates

Verification Integrations

• CLI tool: in-toto-verify for manual verification
• Library support: Go, Python, JavaScript libraries
• CI/CD integration: GitHub Actions, GitLab CI, Tekton

Common Pitfalls and How to Avoid Them

Configuration Issues

Expired Layout

Problem: Layout file expires and verification fails.

Solution:

• Set long expiration dates (1+ years)
• Implement layout rotation process
• Monitor expiration dates
• Use continuous delivery for layout updates

Missing Keys

Problem: Keys used in layout not available for verification.

Solution:

• Distribute public keys with layout
• Use keyless signing with certificate authorities
• Implement key backup and recovery
• Test verification with all keys available

Incorrect Thresholds

Problem: Threshold too high causes legitimate failures, too low reduces security.

Solution:

• Start with threshold of 1 for each role
• Increase thresholds as confidence grows
• Use separate keys for different purposes
• Document key rotation procedures

Performance Issues

Large Metadata Files

Problem: Many links create large metadata files.

Solutions:

• Use link aggregation for batches
• Implement link expiration and cleanup
• Store links in separate storage
• Use link metadata inlining where appropriate

Verification Slowdown

Problem: Complex layouts with many steps slow verification.

Solutions:

• Cache verification results
• Parallelize independent step verification
• Use optimized implementations
• Profile verification for bottlenecks

Operational Challenges

Key Compromise Recovery

Problem: Key compromise requires layout update and re-signing.

Solutions:

• Implement key rotation procedures
• Use multi-signature policies
• Document incident response procedures
• Test recovery procedures regularly

Build Tool Integration

Problem: Existing build tools don't naturally produce in-toto links.

Solutions:

• Wrap existing tools with in-toto runners
• Use in-toto plugins where available
• Implement build step instrumentation
• Gradual adoption strategy

Cross-Platform Compatibility

Problem: in-toto metadata format varies between implementations.

Solutions:

• Use standard in-toto specification
• Validate against JSON Schema
• Test with multiple verifier implementations
• Contribute improvements to standard

Security Pitfalls

Weakened Signature Requirements

Problem: Reducing thresholds during emergencies creates security gaps.

Solutions:

• Document emergency procedures
• Require post-incident review
• Automatically revert threshold changes
• Alert on threshold modifications

Insecure Key Storage

Problem: Private keys stored in source control or accessible locations.

Solutions:

• Use hardware security modules (HSM)
• Store keys in secrets management systems
• Implement key access auditing
• Use keyless signing where possible

Incomplete Supply Chain Coverage

Problem: Only some steps covered, allowing bypass through uncovered steps.

Solutions:

• Map all supply chain steps
• Include all manual and automated steps
• Cover developer laptop builds
• Verify intermediate artifacts

Coding Practices

Idiomatic Configuration

Layout File Structure

{
  "_type": "layout",
  "spec_version": "1.0",
  "expires": "2025-12-31T23:59:59Z",
  "keys": {
    "keyid1": {
      "keytype": "ed25519",
      "keyval": {
        "public": "publickeybase64"
      }
    }
  },
  "roles": {
    "root": {
      "threshold": 1,
      "keyids": ["keyid1"]
    }
  },
  "steps": [],
  "inspect": []
}

API Usage Patterns

Layout Creation

# ✅ GOOD — in-toto layout creation via CLI
# Generate in-toto layout using in-toto-tool
in-toto-record start --name build --expected-products app.tar.gz

# Create initial layout file
cat > layout.json << 'EOF'
{
  "_type": "layout",
  "spec_version": "1.0",
  "expires": "2025-12-31T23:59:59Z",
  "keys": {},
  "roles": {
    "root": { "threshold": 1, "keyids": [] }
  },
  "steps": [],
  "inspect": []
}
EOF

# Add steps to layout
in-toto-layout --add-step build --pubkey alice.pub --threshold 1

# Sign layout with root key
in-toto-sign -f layout.json -k root.key

Link Generation

// Go example
link := link.Link{
    Name:      "build",
    Materials: materials,
    Products:  products,
    Command:   command,
}

link.Sign(privateKey)

Observability Best Practices

Logging Verification

• Log verification pass/fail with details
• Include artifact hashes in logs
• Record verification timestamps
• Correlate with build IDs

Metrics Collection

• Link creation count by step
• Verification success/failure rates
• Key rotation frequency
• Layout expiration monitoring

Development Workflow

Layout Development

1. Define supply chain steps
2. Generate initial layout
3. Test with sample builds
4. Add signatures
5. Verify with sample artifacts
6. Deploy to CI/CD
7. Monitor verification results

CI/CD Integration

• Generate links during build
• Sign links with CI keys
• Store links with artifacts
• Verify before release
• Fail build on verification failure

Fundamentals

Essential Concepts

In-toto Metadata Format

• Layout file: Root policy defining supply chain
• Link files: Step execution records with artifacts and signatures
• Metadata structure: Standardized JSON format
• Cryptographic signatures: ECDSA, Ed25519, RSA support

Verification Process

• Layout loading: Parse policy from layout file
• Link validation: Check signatures and link chain
• Artifact matching: Verify materials flow to products
• Inspection execution: Run post-build verification commands

Terminology Glossary

Data Models and Types

Layout Schema

{
  "_type": "layout",
  "spec_version": "string",
  "expires": "string (ISO 8601)",
  "keys": {
    "keyid": {
      "keytype": "string",
      "keyval": {
        "public": "string"
      }
    }
  },
  "roles": {
    "rolename": {
      "threshold": "number",
      "keyids": ["string"]
    }
  },
  "steps": [
    {
      "name": "string",
      "pubkeys": ["string"],
      "threshold": "number",
      "expected_materials": [],
      "expected_products": []
    }
  ],
  "inspect": [
    {
      "name": "string",
      "command": ["string"],
      "expected_materials": [],
      "expected_products": []
    }
  ]
}

Link Schema

{
  "_type": "link",
  "name": "string",
  "material": {
    "artifact_path": {
      "sha256": "string"
    }
  },
  "product": {
    "artifact_path": {
      "sha256": "string"
    }
  },
  "command": ["string"],
  "byproducts": {},
  "duration": "number"
}

Lifecycle Management

Layout Lifecycle

1. Creation: Initial layout with minimal roles
2. Signing: Root key signs initial layout
3. Distribution: Layout distributed to all verifiers
4. Rotation: Periodic key and layout updates
5. Revocation: Compromised keys removed from layout
6. Expiration: Layout expires and must be renewed

Link Lifecycle

1. Creation: Build step generates link
2. Signing: Step key signs link
3. Storage: Link stored with artifact
4. Aggregation: Multiple links combined
5. Verification: Link verified against layout
6. Archival: Old links archived or deleted

State Management

Layout State

• Expiration tracking: Monitor layout expiration dates
• Key status: Active, revoked, rotated
• Role membership: Keys assigned to roles
• Step definitions: Current supply chain steps

Link State

• Step completion: Pending, complete, failed
• Signature status: Unsigned, signed, verified
• Artifact status: In material, in product
• Chain state: Connected to other links

Scaling and Deployment Patterns

Horizontal Scaling

Multiple Verifiers

• Parallel verification: Multiple verifiers check same artifact
• Load distribution: Distribute verification load
• High availability: Redundant verification services
• Cache sharing: Shared verification results

Link Aggregation

• Batch signing: Sign multiple links together
• Link aggregation files: Combine links for efficiency
• Incremental updates: Update only changed links

High Availability

Verification Service Redundancy

• Multiple verifier instances: Run multiple verifiers
• Load balancing: Distribute verification requests
• Failover verification: Secondary verifier for failover
• Caching: Cache verification results

Layout Distribution

• CDN distribution: Deliver layout via CDN
• Multiple sources: Verify layout from multiple sources
• Local caches: Cache layouts for offline verification
• Versioning: Support multiple layout versions

Production Deployments

CI/CD Pipeline Integration

1. Generate links during build
2. Sign links with appropriate keys
3. Store links with artifacts
4. Verify before deployment
5. Fail pipeline on verification failure

Deployment Gate

# GitHub Actions example
- name: Verify in-toto
  uses: in-toto/verify-action@v1
  with:
    layout: layout.json
    artifact: app.tar.gz

Upgrade Strategies

Layout Evolution

1. Create new layout with updated rules
2. Sign with new keys
3. Deploy new layout alongside old
4. Monitor verification results
5. Switch to new layout
6. Archive old layout

Key Rotation

1. Generate new keys
2. Update layout with new keys
3. Sign layout with old keys
4. Distribute updated layout
5. Stop using old keys
6. Securely delete old keys

Resource Management

Storage Optimization

• Compress links: Use gzip for storage
• Link expiration: Delete old links
• Aggregated storage: Store multiple links together
• Tiered storage: Hot/warm/cold storage tiers

Memory Constraints

• Streaming verification: Process large link files incrementally
• Memory limits: Enforce verification memory limits
• Link size limits: Limit individual link sizes
• Batch processing: Process links in batches

Additional Resources

Official Documentation

• in-toto GitHub: https://github.com/in-toto
• in-toto Specification: https://github.com/in-toto/attestation
• in-toto Python: https://pypi.org/project/in-toto/
• in-toto Go: https://pkg.go.dev/github.com/in-toto/in-toto-golang
• in-toto Ruby: https://rubygems.org/gems/in_toto

CNCF References

• in-toto in CNCF Landscape: https://landscape.cncf.io/?group=projects&filter=supply-chain
• CNCF Security Technical Advisory Group: https://github.com/cncf/sig-security

Tools and Libraries

• in-toto-rs: Rust implementation
• in-toto-golang: Go implementation
• in-toto-python: Python implementation
• in-toto-js: JavaScript implementation
• fulcio: Keyless signing CA
• cosign: Container signing tool
• slsa-generator: SLSA provenance generator

Tutorials and Guides

• in-toto Getting Started: https://in-toto.readthedocs.io/en/latest/
• Build Secure Systems with in-toto: https://www.usenix.org/conference/usenixsecurity20/presentation/melara
• Software Supply Chain Security with in-toto: https://arxiv.org/abs/2009.06734

Community Resources

• in-toto Mailing List: https://groups.google.com/g/in-toto
• in-toto Slack: #in-toto channel
• Stack Overflow: https://stackoverflow.com/questions/tagged/in-toto

Related Standards

• SLSA (Supply-chain Levels for Software Artifacts): https://slsa.dev/
• OCI Images: https://github.com/opencontainers/image-spec
• CycloneDX: https://cyclonedx.org/
• SPDX: https://spdx.dev/

OpenTelemetry Integration

• Tracing: Trace verification steps
• Metrics: Count verification pass/fail rates
• Logs: Verify and audit verification results

Troubleshooting

Common Issues

1. Deployment Failures

   • Check pod logs for errors
   • Verify configuration values
   • Ensure network connectivity

2. Performance Issues

   • Monitor resource usage
   • Adjust resource limits
   • Check for bottlenecks

3. Configuration Errors

   • Validate YAML syntax
   • Check required fields
   • Verify environment-specific settings

4. Integration Problems

   • Verify API compatibility
   • Check dependency versions
   • Review integration documentation

Getting Help

• Check official documentation
• Search GitHub issues
• Join community channels
• Review logs and metrics

Examples

Basic Configuration

# Basic configuration example
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{project_name}}-config
  namespace: default
data:
  # Configuration goes here
  config.yaml: |
    # Base configuration
    # Add your settings here

Kubernetes Deployment

# Kubernetes deployment for {{project_name}}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{project_name}}
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: {{project_name}}
  template:
    metadata:
      labels:
        app: {{project_name}}
    spec:
      containers:
      - name: {{project_name}}
        image: {{project_name}}:latest
        ports:
        - containerPort: 8080
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"

Kubernetes Service

# Kubernetes service for {{project_name}}
apiVersion: v1
kind: Service
metadata:
  name: {{project_name}}
  namespace: default
spec:
  selector:
    app: {{project_name}}
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  type: ClusterIP

When to Use

Use this skill when:

• Integrating a CNCF project into Kubernetes infrastructure — You need to configure, deploy, or troubleshoot a cloud-native tool within a cluster
• Designing cloud-native architecture — You are selecting and integrating CNCF tools to solve specific infrastructure challenges
• Resolving operational issues — A CNCF component is misbehaving, underperforming, or needs configuration changes

Core Workflow

1. Assess Requirements — Understand the use case, scale, integration needs, and existing infrastructure. Checkpoint: Document requirements, constraints, and success criteria.

2. Design Architecture — Plan component interactions, data flow, and deployment strategy using cloud-native best practices. Checkpoint: Verify the architecture addresses all requirements and follows CNCF conventions.

3. Implement & Configure — Create manifests, configurations, and deployment scripts. Include resource limits, health checks, and observability hooks. Checkpoint: Validate all YAML against schema and test in a staging environment.

4. Deploy & Monitor — Apply manifests to the cluster, verify component health, and confirm observability is working. Checkpoint: Confirm all pods/services are running, probes passing, and metrics/alerts configured.

Constraints

MUST DO

• Include at least one complete working YAML manifest example
• Note when content is auto-generated vs. manually verified
• Reference relevant CNCF project documentation

MUST NOT DO

• Deploy manifests without testing in a staging environment first
• Use deprecated API versions (e.g., apps/v1beta1)
• Omit resource limits and requests in Kubernetes manifests