Cubefs

related-skills: cncf-aws-dynamodb, cncf-aws-ecr, cncf-aws-rds, cncf-aws-s3

CubeFS in Cloud-Native Engineering

Category: storage
Status: Sandbox
Stars: 1,500
Last Updated: 2026-04-22
Primary Language: Go
Documentation: https://cubefs.io/

Purpose and Use Cases

CubeFS is a CNCF sandbox project that provides a distributed, high-performance file system for cloud-native environments.

What Problem Does It Solve?

Storage fragmentation across different use cases (object, block, file). CubeFS provides a unified storage solution with support for multiple protocols and high performance for containerized workloads.

When to Use This Project

Use CubeFS when you need:

• Distributed file system with POSIX compatibility
• High performance for container workloads
• Multi-protocol support (S3, NAS)
• Horizontal scalability
• Cloud-native storage for Kubernetes

Key Use Cases

• Kubernetes Persistent Storage: Storage for stateful applications
• Machine Learning Workloads: High-throughput data access
• Big Data Processing: Distributed file storage for analytics
• Media Processing: High-bandwidth file operations
• Hybrid Cloud Storage: Unified storage across clouds

Architecture Design Patterns

CubeFS Architecture

┌─────────────────────────────────────────┐
│           Client (FUSE/NAS/S3)          │
│        (POSIX/S3/NAS protocols)         │
└──────────────┬──────────────────────────┘
               │
               ▼
┌─────────────────────────────────────────┐
│          CubeFS Control Plane           │
│  ┌──────────┐  ┌────────────────────┐   │
│  │  Master  │  │  Metadata Service  │   │
│  │  (Leader)│  │  (Raft consensus)  │   │
│  └──────────┘  └────────────────────┘   │
└──────────────┬──────────────────────────┘
               │
               ▼
┌─────────────────────────────────────────┐
│           Data Plane (DataNodes)        │
│  ┌──────────┐  ┌────────────────────┐   │
│  │  Data    │  │  Data              │   │
│  │  Node 1  │  │  Node 2            │   │
│  └──────────┘  └────────────────────┘   │
└─────────────────────────────────────────┘

Components

1. Master

• Cluster management
• Metadata distribution
• Leader election via Raft

2. DataNode

• Data storage
• Block management
• Replication

3. Client

• POSIX FUSE interface
• S3 API
• NAS (NFS/CIFS)

Data Flow

1. Client → Master: Request file metadata
2. Master → Client: Return metadata location
3. Client → DataNode: Read/write data
4. DataNode → Client: Return data

Storage Pool

# Storage pool configuration
storagePools:
  - name: default
    tier: default
    replicas: 3
    compression: false
    encryption: false

Integration Approaches

Kubernetes Integration

# Deploy CubeFS to Kubernetes
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cubefs-master
spec:
  replicas: 3
  selector:
    matchLabels:
      app: cubefs-master
  template:
    metadata:
      labels:
        app: cubefs-master
    spec:
      containers:
      - name: cubefs-master
        image: cubefs/master:latest
        args:
          - "--config=/etc/cubefs/master.conf"
          - "--role=master"
        ports:
        - containerPort: 9100
---
  related-skills: cncf-aws-dynamodb, cncf-aws-ecr, cncf-aws-rds, cncf-aws-s3
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cubefs-datanode
spec:
  replicas: 3
  selector:
    matchLabels:
      app: cubefs-datanode
  template:
    metadata:
      labels:
        app: cubefs-datanode
    spec:
      containers:
      - name: cubefs-datanode
        image: cubefs/datanode:latest
        args:
          - "--config=/etc/cubefs/datanode.conf"
          - "--role=datanode"
        ports:
        - containerPort: 9200
        volumeMounts:
        - name: data
          mountPath: /data
      volumes:
      - name: data
        hostPath:
          path: /var/lib/cubefs
---
  related-skills: cncf-aws-dynamodb, cncf-aws-ecr, cncf-aws-rds, cncf-aws-s3
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: cubefs-fuse
spec:
  selector:
    matchLabels:
      app: cubefs-fuse
  template:
    metadata:
      labels:
        app: cubefs-fuse
    spec:
      hostNetwork: true
      containers:
      - name: cubefs-fuse
        image: cubefs/fuse:latest
        args:
          - "--config=/etc/cubefs/fuse.conf"
          - "--mountpoint=/cubefs"
        securityContext:
          privileged: true
        volumeMounts:
        - name: cubefs
          mountPath: /cubefs
      volumes:
      - name: cubefs
        hostPath:
          path: /cubefs

Persistent Volume

# CubeFS as Kubernetes PV
apiVersion: v1
kind: PersistentVolume
metadata:
  name: cubefs-pv
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  mountOptions:
    - allow_other
  csi:
    driver: cubefs.csi.cubefs.com
    volumeHandle: cubefs-volume-001
    nodePublishSecretRef:
      name: cubefs-secret
      namespace: kube-system
  storageClassName: cubefs
---
  related-skills: cncf-aws-dynamodb, cncf-aws-ecr, cncf-aws-rds, cncf-aws-s3
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cubefs-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: cubefs
  resources:
    requests:
      storage: 100Gi

S3 Compatibility

# CubeFS S3 API endpoint
curl http://cubefs-s3:7400/bucket-name/object-key

# S3 CLI usage
aws --endpoint-url=http://cubefs-s3:7400 s3 cp file.txt s3://bucket-name/

# MinIO client
mc alias set cubefs http://cubefs-s3:7400 access-key secret-key

related-skills: cncf-aws-dynamodb, cncf-aws-ecr, cncf-aws-rds, cncf-aws-s3

Common Pitfalls and How to Avoid Them

1. Replica Configuration

Pitfall: Incorrect replica count affecting availability.

# ❌ Incorrect - single replica
storagePools:
  - name: default
    replicas: 1  # Risky - no redundancy

# ✅ Correct - multiple replicas
storagePools:
  - name: default
    replicas: 3  # Recommended for production

2. Disk Space Management

Pitfall: DataNodes running out of disk space.

# ❌ Incorrect - no disk space monitoring
# No warnings or alerts for low disk space

# ✅ Correct - with monitoring
dataNodes:
  - name: datanode-1
    diskPaths:
      - /data/disk1
      - /data/disk2
    diskQuota: 1000GB
    lowWatermark: 0.8
    highWatermark: 0.95

3. Network Configuration

Pitfall: Network latency affecting performance.

# ❌ Incorrect - no network optimization
# Default network settings

# ✅ Correct - with network optimization
network:
  # Enable TCP optimization
  tcpNoDelay: true
  socketSendBuffer: 4MB
  socketReceiveBuffer: 4MB
  
  # Enable compression
  compression: true
  compressionType: lz4

4. Client Mount Options

Pitfall: Incorrect mount options affecting performance.

# ❌ Incorrect - default mount options
cubefs-fuse --mountpoint=/mnt/cubefs

# ✅ Correct - optimized mount options
cubefs-fuse \
  --mountpoint=/mnt/cubefs \
  --attr-timeout=120 \
  --entry-timeout=120 \
  --negative-timeout=60 \
  --allow-other \
  --enable-external-service=true

5. Master Failover

Pitfall: Master failover not properly configured.

# ❌ Incorrect - single master
masters:
  - name: master-1
    port: 9100

# ✅ Correct - multiple masters with raft
masters:
  - name: master-1
    port: 9100
  - name: master-2
    port: 9101
  - name: master-3
    port: 9102
raft:
  electionTimeout: 1000ms
  heartbeatInterval: 500ms

6. Metadata Performance

Pitfall: Metadata server bottleneck.

# ❌ Incorrect - single metadata server
metadataServices:
  - name: metadata-1

# ✅ Correct - distributed metadata
metadataServices:
  - name: metadata-1
  - name: metadata-2
  - name: metadata-3
  - name: metadata-4
metadata:
  cacheSize: 1GB
  cacheEntryCount: 1000000

related-skills: cncf-aws-dynamodb, cncf-aws-ecr, cncf-aws-rds, cncf-aws-s3

Coding Practices

Client Library

// CubeFS client library
package cubefs

import (
    "context"
    "fmt"
)

// Client wraps CubeFS API operations
type Client struct {
    masterAddr string
}

// NewClient creates a new CubeFS client
func NewClient(masterAddr string) *Client {
    return &Client{
        masterAddr: masterAddr,
    }
}

// CreateFile creates a new file
func (c *Client) CreateFile(ctx context.Context, path string, mode int) error {
    // Implementation
    return nil
}

// OpenFile opens an existing file
func (c *Client) OpenFile(ctx context.Context, path string) (*File, error) {
    // Implementation
    return nil, nil
}

// ReadFile reads a file
func (c *Client) ReadFile(ctx context.Context, path string) ([]byte, error) {
    // Implementation
    return nil, nil
}

// WriteFile writes to a file
func (c *Client) WriteFile(ctx context.Context, path string, data []byte) error {
    // Implementation
    return nil
}

// DeleteFile deletes a file
func (c *Client) DeleteFile(ctx context.Context, path string) error {
    // Implementation
    return nil
}

// ListDir lists a directory
func (c *Client) ListDir(ctx context.Context, path string) ([]string, error) {
    // Implementation
    return nil, nil
}

// File represents an open file
type File struct {
    fd int
    path string
}

// Read reads from file
func (f *File) Read(p []byte) (int, error) {
    // Implementation
    return 0, nil
}

// Write writes to file
func (f *File) Write(p []byte) (int, error) {
    // Implementation
    return 0, nil
}

// Close closes the file
func (f *File) Close() error {
    // Implementation
    return nil
}

CSI Driver

// CubeFS CSI Driver
package csi

import (
    "context"
    "fmt"
    
    "github.com/container-storage-interface/spec/lib/go/csi"
    "google.golang.org/grpc/codes"
    "google.golang.org/grpc/status"
)

// Driver implements CSI interface
type Driver struct {
    nodeID   string
    endpoint string
    client   *Client
}

// NewDriver creates a new CSI driver
func NewDriver(nodeID, endpoint string, client *Client) *Driver {
    return &Driver{
        nodeID:   nodeID,
        endpoint: endpoint,
        client:   client,
    }
}

// CreateVolume creates a new volume
func (d *Driver) CreateVolume(ctx context.Context, req *csi.CreateVolumeRequest) (*csi.CreateVolumeResponse, error) {
    name := req.GetName()
    if name == "" {
        return nil, status.Error(codes.InvalidArgument, "Volume name must be provided")
    }
    
    capacity := req.GetCapacityRange().GetRequiredBytes()
    
    // Create CubeFS volume
    err := d.client.CreateVolume(ctx, name, capacity)
    if err != nil {
        return nil, status.Errorf(codes.Internal, "Failed to create volume: %v", err)
    }
    
    return &csi.CreateVolumeResponse{
        Volume: &csi.Volume{
            VolumeId:      name,
            CapacityBytes: capacity,
            VolumeContext: req.GetParameters(),
        },
    }, nil
}

// DeleteVolume deletes a volume
func (d *Driver) DeleteVolume(ctx context.Context, req *csi.DeleteVolumeRequest) (*csi.DeleteVolumeResponse, error) {
    volumeID := req.GetVolumeId()
    if volumeID == "" {
        return nil, status.Error(codes.InvalidArgument, "Volume ID must be provided")
    }
    
    err := d.client.DeleteVolume(ctx, volumeID)
    if err != nil {
        return nil, status.Errorf(codes.Internal, "Failed to delete volume: %v", err)
    }
    
    return &csi.DeleteVolumeResponse{}, nil
}

// PublishVolume mounts a volume
func (d *Driver) PublishVolume(ctx context.Context, req *csi.NodePublishVolumeRequest) (*csi.NodePublishVolumeResponse, error) {
    volumeID := req.GetVolumeId()
    targetPath := req.GetTargetPath()
    
    err := d.client.MountVolume(ctx, volumeID, targetPath)
    if err != nil {
        return nil, status.Errorf(codes.Internal, "Failed to mount volume: %v", err)
    }
    
    return &csi.NodePublishVolumeResponse{}, nil
}

// UnpublishVolume unmounts a volume
func (d *Driver) UnpublishVolume(ctx context.Context, req *csi.NodeUnpublishVolumeRequest) (*csi.NodeUnpublishVolumeResponse, error) {
    volumeID := req.GetVolumeId()
    targetPath := req.GetTargetPath()
    
    err := d.client.UnmountVolume(ctx, volumeID, targetPath)
    if err != nil {
        return nil, status.Errorf(codes.Internal, "Failed to unmount volume: %v", err)
    }
    
    return &csi.NodeUnpublishVolumeResponse{}, nil
}

// NodeStageVolume stages a volume
func (d *Driver) NodeStageVolume(ctx context.Context, req *csi.NodeStageVolumeRequest) (*csi.NodeStageVolumeResponse, error) {
    volumeID := req.GetVolumeId()
    stagingTargetPath := req.GetStagingTargetPath()
    
    err := d.client.StagingVolume(ctx, volumeID, stagingTargetPath)
    if err != nil {
        return nil, status.Errorf(codes.Internal, "Failed to stage volume: %v", err)
    }
    
    return &csi.NodeStageVolumeResponse{}, nil
}

// NodeUnstageVolume unstages a volume
func (d *Driver) NodeUnstageVolume(ctx context.Context, req *csi.NodeUnstageVolumeRequest) (*csi.NodeUnstageVolumeResponse, error) {
    volumeID := req.GetVolumeId()
    stagingTargetPath := req.GetStagingTargetPath()
    
    err := d.client.UnstageVolume(ctx, volumeID, stagingTargetPath)
    if err != nil {
        return nil, status.Errorf(codes.Internal, "Failed to unstage volume: %v", err)
    }
    
    return &csi.NodeUnstageVolumeResponse{}, nil
}

Fundamentals

CubeFS Storage Model

1. Block

• Fixed-size data unit (default 4MB)
• Replicated across DataNodes
• Referenced by files

2. File

• Metadata + block references
• Stored in metadata service
• Supports POSIX operations

3. Volume

• Logical container for files
• Configurable replica count
• Storage pool assignment

Protocol Support

1. POSIX (FUSE)

• Full POSIX compliance
• Standard file operations

2. S3

• S3-compatible API
• Object storage operations

3. NAS

• NFS v3/v4 support
• CIFS/SMB support

Scaling and Deployment Patterns

Horizontal Scaling

# Scale CubeFS horizontally
# Add more DataNodes for capacity
# Add more metadata servers for metadata performance

Multi-Region

# Multi-region deployment
# Primary region: active traffic
# Secondary region: backup/failover
# Tertiary region: disaster recovery

Backup and Recovery

# Backup CubeFS data
# Use CubeFS snapshot feature
# Export to S3 for long-term storage

Additional Resources

Official Documentation

• CubeFS - Project website
• CubeFS Documentation - Complete documentation
• CubeFS GitHub - Source code

Community Resources

• CNCF CubeFS - CNCF project page
• CubeFS Slack - Community discussion

Learning Resources

• CubeFS Getting Started - Tutorial
• CubeFS Examples - Examples

This SKILL.md file was verified and last updated on 2026-04-22. Content based on CNCF CubeFS project and production usage patterns.

Troubleshooting

Common Issues

1. Deployment Failures

   • Check pod logs for errors
   • Verify configuration values
   • Ensure network connectivity

2. Performance Issues

   • Monitor resource usage
   • Adjust resource limits
   • Check for bottlenecks

3. Configuration Errors

   • Validate YAML syntax
   • Check required fields
   • Verify environment-specific settings

4. Integration Problems

   • Verify API compatibility
   • Check dependency versions
   • Review integration documentation

Getting Help

• Check official documentation
• Search GitHub issues
• Join community channels
• Review logs and metrics

Examples

Basic Configuration

# Basic configuration example
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{project_name}}-config
  namespace: default
data:
  # Configuration goes here
  config.yaml: |
    # Base configuration
    # Add your settings here

Kubernetes Deployment

# Kubernetes deployment for {{project_name}}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{project_name}}
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: {{project_name}}
  template:
    metadata:
      labels:
        app: {{project_name}}
    spec:
      containers:
      - name: {{project_name}}
        image: {{project_name}}:latest
        ports:
        - containerPort: 8080
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"

Kubernetes Service

# Kubernetes service for {{project_name}}
apiVersion: v1
kind: Service
metadata:
  name: {{project_name}}
  namespace: default
spec:
  selector:
    app: {{project_name}}
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  type: ClusterIP

When to Use

Use this skill when:

• Integrating a CNCF project into Kubernetes infrastructure — You need to configure, deploy, or troubleshoot a cloud-native tool within a cluster
• Designing cloud-native architecture — You are selecting and integrating CNCF tools to solve specific infrastructure challenges
• Resolving operational issues — A CNCF component is misbehaving, underperforming, or needs configuration changes

Core Workflow

1. Assess Requirements — Understand the use case, scale, integration needs, and existing infrastructure. Checkpoint: Document requirements, constraints, and success criteria.

2. Design Architecture — Plan component interactions, data flow, and deployment strategy using cloud-native best practices. Checkpoint: Verify the architecture addresses all requirements and follows CNCF conventions.

3. Implement & Configure — Create manifests, configurations, and deployment scripts. Include resource limits, health checks, and observability hooks. Checkpoint: Validate all YAML against schema and test in a staging environment.

4. Deploy & Monitor — Apply manifests to the cluster, verify component health, and confirm observability is working. Checkpoint: Confirm all pods/services are running, probes passing, and metrics/alerts configured.

Constraints

MUST DO

• Include at least one complete working YAML manifest example
• Note when content is auto-generated vs. manually verified
• Reference relevant CNCF project documentation

MUST NOT DO

• Deploy manifests without testing in a staging environment first
• Use deprecated API versions (e.g., apps/v1beta1)
• Omit resource limits and requests in Kubernetes manifests