Lakefs Versioning

Overview

LakeFS brings Git-like version control to data lakes. It enables branching, committing, merging, and reverting data changes — supporting atomic operations, isolated experimentation, and CI/CD for data.

Capabilities

• Create branches for isolated data experiments
• Commit atomic changes to data
• Merge branches with conflict detection
• Time travel to any previous data state
• Run pre-commit and pre-merge hooks (CI/CD)
• Works with S3, Azure Blob, GCS as storage backends
• Compatible with Spark, Presto, Trino, Hive, dbt

When to Use

Trigger phrases:

• "lakefs versioning"

• "LakeFS data versioning — Git-like branching for data lakes, atomic commits, time"

• Needing reproducible data pipelines

• Experimenting with data changes without risk

• Implementing CI/CD for data quality

• Rolling back failed data updates

• Supporting multi-tenant data isolation

When NOT to Use

• Task is about data processing, not versioning
• You need real-time data streaming (use streaming tools)
• Task is about data storage, not version control
• You don't have data pipeline infrastructure
• Task is about database migrations (use migration tools)
• You need data warehousing, not versioning

Pseudo Code

Implementation patterns for common use cases with this skill.

Installation

# Docker
docker run -p 8000:8000 treeverse/lakefs:latest

# Access at http://localhost:8000
# Default credentials: admin / admin

CLI Operations

# Install lakectl
brew install lakefs/tap/lakectl

# Configure
lakectl config
# Endpoint: http://localhost:8000
# Access Key: AKIAIOSFODNN7EXAMPLE
# Secret Key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

# Repository operations
lakectl repo create my-repo s3://my-bucket
lakectl branch create my-repo/main-branch

# Branch
lakectl branch create my-repo/experiment-1 --source my-repo/main

# Upload data
lakectl fs upload s3://my-repo/experiment-1/data/file.parquet --source ./file.parquet

# Commit
lakectl commit my-repo/experiment-1 -m "Add new data"

# Merge
lakectl merge my-repo/experiment-1 my-repo/main

# Diff
lakectl diff my-repo/main my-repo/experiment-1

# Time travel
lakectl fs ls my-repo/main@commit-abc123/data/

Python SDK

import lakefs

# Connect
repo = lakefs.repository("my-repo")

# Create branch
branch = repo.branch("experiment-1").create(source_reference="main")

# Upload object
obj = branch.object("data/new_file.parquet")
obj.upload(data=open("file.parquet", "rb"))

# Commit
branch.commit(message="Add new data", metadata={"author": "etl-pipeline"})

# Merge
branch.merge_into("main", message="Merge experiment results")

# List objects
for obj in branch.objects(prefix="data/"):
    print(obj.path, obj.size_bytes)

# Time travel
main_branch = repo.branch("main")
for obj in main_branch.objects(prefix="data/", ref="commit-abc123"):
    print(obj.path)

Spark Integration

# S3 gateway endpoint
spark.conf.set("spark.hadoop.fs.s3a.endpoint", "http://localhost:8000")
spark.conf.set("spark.hadoop.fs.s3a.access.key", "AKIAIOSFODNN7EXAMPLE")
spark.conf.set("spark.hadoop.fs.s3a.secret.key", "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY")

# Read from branch
df = spark.read.parquet("s3a://my-repo/experiment-1/data/")

# Write to branch
df.write.parquet("s3a://my-repo/experiment-1/output/")

Hooks (CI/CD)

# .lakefs.yaml
pre_commit:
  - name: check_format
    type: lua
    properties:
      script: |
        for _, obj in ipairs(objects) do
          if not string.match(obj.path, "%.parquet$") then
            error("Only parquet files allowed: " .. obj.path)
          end
        end

pre_merge:
  - name: validate_schema
    type: lua
    properties:
      script: |
        -- Run schema validation before merge
        -- Reject if breaking changes detected

Garbage Collection

# Configure retention policy
lakectl gc set-config my-repo --from-file gc-config.yaml

# gc-config.yaml
default_retention_days: 30
branches:
  - branch: main
    retention_days: 90
  - branch: experiment-*
    retention_days: 7

Common Patterns

Error Handling

Red Flags

• Not testing data versions before committing
• Ignoring data quality issues
• Not backing up data before versioning
• Missing rollback procedures
• Not documenting data changes

Verification

• Data versions are created correctly
• Rollback procedures are tested
• Data quality is maintained
• Changes are documented
• Backup procedures are in place

Process

1. Analyze the task requirements
2. Apply domain expertise
3. Verify output quality

Anti-Rationalization Table