# Backend Distributed Locking

> Use this skill when the user says 'distributed lock', 'Redis lock', 'Redlock', 'ZooKeeper lock', 'advisory lock', 'PostgreSQL lock', 'pg_advisory_lock', 'lease', 'distributed mutex', 'lock timeouts', 'fencing token'. This skill implements distributed locking using Redis Redlock, ZooKeeper, PostgreSQL advisory locks, or lease-based mechanisms. Applies to any backend stack. Do NOT use for: single-process mutexes, database transaction serialization, or optimistic concurrency.

- Skill: `j4flmao/backend-distributed-locking` (Agent Skill, multi-file: 10 files)
- Install (CLI): `npx skillmds@latest add j4flmao/backend-distributed-locking`
- Raw SKILL.md: https://api.skillmd.com/api/skills/j4flmao/backend-distributed-locking/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: MIT
- Author: j4flmao (https://skillmd.com/u/j4flmao)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/j4flmao/backend-distributed-locking

---


# Backend Distributed Locking

## Purpose
Coordinate access to shared resources across multiple service instances using distributed locks with guaranteed safety properties.

## Agent Protocol

### Trigger
Exact user phrases: "distributed lock", "Redis lock", "Redlock", "ZooKeeper lock", "advisory lock", "PostgreSQL lock", "pg_advisory_lock", "lease", "distributed mutex", "fencing token", "lock timeout".

### Input Context
- Resource being protected.
- Available infrastructure (Redis, PostgreSQL, ZooKeeper, etcd).
- Number of competing instances.
- Duration of the critical section.

### Output Artifact
Lock configuration or implementation code. No file unless requested.

### Response Format
```
Provider: {Redis|PostgreSQL|ZooKeeper|etcd}
Strategy: {Redlock|Advisory|Ephemeral|Lease}
TTL: {duration}
Safety: {fencing?}
```

### Completion Criteria
- [ ] Lock acquisition and release implemented correctly.
- [ ] Lock timeout (TTL) configured.
- [ ] Fencing token mechanism in place for critical resources.
- [ ] Deadlock prevention: no nested locks.
- [ ] Graceful degradation: lock acquisition failure does not crash the service.

### Max Response Length
4 lines per lock configuration. 20 lines for implementation.

## Architecture Decision Tree

### Which Lock Provider?

```
What infrastructure is already available?
  ├── Redis → High throughput, short locks, needs Redlock for failover
  ├── PostgreSQL → Same DB as data, simple setup, lock contention impacts DB
  ├── ZooKeeper/etcd → Strong consistency, leader election, operational complexity
  └── In-memory → Single instance only, NOT distributed
```

### Which Lock Strategy?

```
Is the critical section short (< 1 second)?
  ├── Yes → Redis simple lock or PostgreSQL advisory lock
  └── No → Is the critical section a resource write?
            ├── Yes → Lease-based lock with fencing token
            └── No → Redlock (Redis with multi-node failover)

Is strong consistency required?
  ├── Yes → ZooKeeper/etcd (sequential consistency, fencing built-in)
  └── No → Is failover safety required?
            ├── Yes → Redlock (Redis, 3+ nodes)
            └── No → Single Redis lock or PG advisory lock
```

### Do I Need a Fencing Token?

```
Does the lock protect a write to shared storage?
  ├── Yes → Does shared storage have its own concurrency control (optimistic locking)?
  │         ├── Yes → Fencing token optional but recommended
  │         └── No → Fencing token REQUIRED — without it, stale lock holders can corrupt data
  └── No → No fencing token needed
```

## Workflow

### Step 1: Choose Lock Provider
| Provider | Best For | Trade-off |
|----------|----------|-----------|
| Redis | High throughput, short locks | Needs failover (Redlock) |
| PostgreSQL | Same DB as data | Lock contention impacts DB |
| ZooKeeper/etcd | Strong consistency | Operational complexity |
| In-memory | Single instance only | Not distributed |

### Step 2: Implement Lock (Redlock)
```javascript
const { Lock } = require('redlock');
const redlock = new Redlock([redis1, redis2, redis3], { driftFactor: 0.01 });

async function processResource() {
  const lock = await redlock.acquire(['resource:order-123'], 5000);
  try {
    await updateOrderStatus('order-123', 'processing');
  } finally {
    await lock.release();
  }
}
```

```python
from redlock import Redlock
import aioredis

redis = aioredis.from_url("redis://localhost")
lock = redis.lock("resource:order-123", timeout=5000)

async with lock:
    await update_order_status("order-123", "processing")
```

```go
// Go — Redis lock with go-redis
import "github.com/go-redsync/redsync/v4"
import "github.com/go-redsync/redsync/v4/redis/goredis/v9"

func processOrder(ctx context.Context, orderId string) error {
    mutex := rs.NewMutex("resource:order-" + orderId, redsync.WithExpiry(5*time.Second))
    if err := mutex.Lock(); err != nil {
        return fmt.Errorf("failed to acquire lock: %w", err)
    }
    defer func() {
        if ok, err := mutex.Unlock(); !ok || err != nil {
            log.Printf("failed to release lock: %v", err)
        }
    }()
    return updateOrderStatus(ctx, orderId, "processing")
}
```

### Step 3: Implement Fencing Token
Fencing tokens ensure that even if a lock is held after its timeout, stale holders cannot write.

```typescript
class FencedResource {
  private currentFencingToken = 0;

  async executeWithFencing(resourceId: string, operation: () => Promise<void>): Promise<void> {
    const fencingToken = await this.lockService.acquire(resourceId);
    try {
      // Pass fencing token to the resource
      await this.db.query(
        'UPDATE resources SET data = $1, fencing_token = $2 WHERE id = $3 AND fencing_token < $2',
        [newData, fencingToken, resourceId]
      );
      // If ROWS_AFFECTED = 0, someone else has the lock — abort
    } finally {
      await this.lockService.release(resourceId);
    }
  }
}
```

```sql
-- PostgreSQL advisory lock with fencing
SELECT pg_advisory_xact_lock(12345);
UPDATE resources
SET status = 'locked', version = version + 1
WHERE id = 12345 AND version = :expectedVersion;
-- version serves as the fencing token
```

### Step 4: Handle Lock Failure
```javascript
async function withLock(name, ttl, fn) {
  const lock = await redlock.acquire([name], ttl).catch(() => null);
  if (!lock) throw new Error('Lock acquisition failed — resource busy');
  try { return await fn(lock); }
  finally { await lock.release().catch(() => {}); }
}
```

```go
// Go — lock acquisition with timeout
func withLock(ctx context.Context, key string, ttl time.Duration, fn func() error) error {
    mutex := rs.NewMutex(key, redsync.WithExpiry(ttl))
    ctx, cancel := context.WithTimeout(ctx, time.Second)
    defer cancel()

    if err := mutex.LockContext(ctx); err != nil {
        return ErrResourceBusy
    }
    defer mutex.Unlock()

    return fn()
}
```

### Step 5: Monitor Locks
```typescript
interface LockMetrics {
  acquisitionTime: number;   // ms to acquire
  holdTime: number;          // ms held
  contentionCount: number;   // how many times acquisition was retried
  timeoutRate: number;       // how often locks expire before release
}

// Export metrics via your monitoring system
metrics.histogram('lock.acquisition_time', acquisitionTime);
metrics.histogram('lock.hold_time', holdTime);
metrics.counter('lock.contention', contentionCount);
metrics.counter('lock.timeout', timeoutRate);
```

## Implementation Patterns

### PostgreSQL Advisory Lock
```typescript
class PostgresDistributedLock {
  constructor(private pool: Pool) {}

  async acquire(lockId: number, timeoutMs: number = 5000): Promise<boolean> {
    const client = await this.pool.connect();
    try {
      // Try to acquire the lock with a timeout
      const result = await client.query(
        'SELECT pg_try_advisory_lock($1) AS acquired',
        [lockId]
      );
      if (!result.rows[0].acquired) {
        client.release();
        return false;
      }
      // Store the client reference for release
      (this as any).client = client;
      return true;
    } catch (error) {
      client.release();
      throw error;
    }
  }

  async release(): Promise<void> {
    const client = (this as any).client;
    if (client) {
      await client.query('SELECT pg_advisory_unlock($1)', [this.lockId]);
      client.release();
      (this as any).client = null;
    }
  }
}
```

PostgreSQL advisory locks are session-level: must hold the same connection for lock and release. Transaction-level: `pg_advisory_xact_lock` auto-releases on transaction end. Use bigint lock IDs: hash your resource name to a bigint for consistent lock IDs.

### Lease-Based Lock (etcd)
```typescript
class EtcdLeaseLock {
  constructor(private etcd: Etcd3) {}

  async acquire(key: string, ttl: number): Promise<Lease | null> {
    const lease = this.etcd.lease(ttl / 1000);
    try {
      await lease.put(key).value(process.env.HOSTNAME);
      // If success, we hold the lease; if key exists, grant fails
      lease.on('lost', () => {
        logger.warn('Lease lost', { key });
      });
      return lease;
    } catch (error) {
      lease.revoke();
      return null;
    }
  }
}
```

### Simple Redis Lock (Single Node)
```typescript
class RedisLock {
  constructor(private redis: Redis) {}

  async acquire(key: string, ttlMs: number): Promise<boolean> {
    // SET NX — only set if key doesn't exist
    const result = await this.redis.set(key, process.env.HOSTNAME, 'PX', ttlMs, 'NX');
    return result === 'OK';
  }

  async release(key: string): Promise<void> {
    // Lua script ensures we only delete if we own the lock
    const script = `
      if redis.call("get", KEYS[1]) == ARGV[1] then
        return redis.call("del", KEYS[1])
      else
        return 0
      end
    `;
    await this.redis.eval(script, 1, key, process.env.HOSTNAME);
  }
}
```

### Semaphore Pattern (Multiple Permits)
```typescript
class RedisSemaphore {
  async acquire(key: string, permits: number, maxPermits: number, ttlMs: number): Promise<boolean> {
    const script = `
      local current = redis.call("GET", KEYS[1])
      if current and tonumber(current) + tonumber(ARGV[1]) > tonumber(ARGV[2]) then
        return 0
      end
      return redis.call("INCRBY", KEYS[1], ARGV[1])
    `;
    const result = await this.redis.eval(script, 1, key, permits, maxPermits);
    if (result) {
      await this.redis.pexpire(key, ttlMs);
    }
    return !!result;
  }

  async release(key: string, permits: number): Promise<void> {
    await this.redis.decrby(key, permits);
  }
}
```

## Production Considerations

### Lock TTL Selection
| Critical Section Duration | Recommended TTL | Safety Margin |
|---|---|---|
| < 10ms | 100ms | 10x |
| < 100ms | 500ms | 5x |
| < 1s | 3000ms | 3x |
| < 5s | 15000ms | 3x |
| > 5s | Not recommended | — |

### Clock Drift Handling
Redlock assumes synchronized clocks. In practice:
- NTP keeps most systems within < 10ms drift
- Redlock's drift factor (0.01 by default) accounts for this
- For extremely sensitive systems, use ZooKeeper/etcd (no clock dependency)
- Monitor clock skew in production: alert if > 100ms difference between nodes

### Lock Cleanup on Crash
- Redis locks auto-expire via TTL
- ZooKeeper ephemeral nodes auto-delete on session loss
- PostgreSQL advisory locks auto-release on connection close
- Always set TTL — never rely on cleanup logic in finally blocks alone

## Anti-Patterns

1. **No TTL on locks**: Indefinite locks cause system-wide freezes on holder failure.
2. **Nested locks**: Acquiring lock A while holding lock B creates deadlock risk.
3. **Not releasing in `finally`**: Exception before release = lock held forever.
4. **Long critical sections**: Keep lock duration minimal — locks are not transactions.
5. **Ignoring fencing tokens**: Without fencing, a delayed lock holder can corrupt data after the lock expires.
6. **Single-node Redis for production locks**: A single Redis node failure causes false-positive lock loss.
7. **Network calls inside lock**: Holding a lock during an external API call blocks other instances.
8. **Lock as a substitute for idempotency**: Locks prevent concurrent execution, not duplicate execution.
9. **Locking before checking**: Always check if you need the lock before acquiring it (shortest possible hold).
10. **Same lock for read and write**: Use read-write locks when reads don't need exclusive access.

## Rules
- Always set a TTL (lease duration) on every lock. Never use indefinite locks.
- Release locks in `finally` blocks — never in the happy path only.
- Use fencing tokens for any lock protecting a write to shared storage.
- Keep critical sections as short as possible — locks are not transactions.
- Never acquire one lock while holding another (potential deadlock).
- Log every lock acquisition and release with duration.
- If a lock cannot be acquired, degrade gracefully — do not block indefinitely.
- Monitor lock contention, acquisition time, hold time, and timeout rate.
- For read-heavy resources, use read-write locks (shared read, exclusive write).
- Always handle the case where lock acquisition fails — the system must continue.
- Test lock scenarios: timeout, contention, crash during hold, network partition.

## References
  - references/distributed-locking-fundamentals.md — Distributed Locking Fundamentals
  - references/distributed-locking-advanced.md — Distributed Locking Advanced
  - references/fencing-tokens-deep.md — Fencing Token Deep Dive
  - references/lock-contention-analysis.md — Lock Contention Analysis
  - references/lock-deep-dive.md — Distributed Locking Deep Dive
  - references/lock-implementations.md — Distributed Lock Implementations
  - references/lock-providers.md — Lock Provider Implementation Notes
  - references/lock-strategies.md — Distributed Lock Strategies
  - references/lock-testing.md — Lock Testing
## Handoff
No artifact produced unless requested.
Next skill: webhooks — deliver events from the service to external subscribers.
Carry forward: lock provider, TTL configuration, fencing requirement.

## Implementation Patterns

### Redis-Based Distributed Lock

```python
import redis
import uuid
import time
from typing import Optional, Callable, Any

class DistributedLock:
    def __init__(self, redis_client: redis.Redis, lock_key: str, ttl_seconds: int = 30):
        self.redis = redis_client
        self.lock_key = f"lock:{lock_key}"
        self.ttl = ttl_seconds
        self.lock_value = str(uuid.uuid4())
        self.acquired = False

    def acquire(self, blocking: bool = True, timeout: Optional[float] = None) -> bool:
        start = time.time()
        while True:
            acquired = self.redis.set(
                self.lock_key,
                self.lock_value,
                nx=True,
                ex=self.ttl,
            )
            if acquired:
                self.acquired = True
                return True

            if not blocking:
                return False

            if timeout and (time.time() - start) > timeout:
                return False

            time.sleep(0.05)

    def release(self):
        if not self.acquired:
            return

        lua_script = """
        if redis.call("GET", KEYS[1]) == ARGV[1] then
            return redis.call("DEL", KEYS[1])
        else
            return 0
        end
        """
        self.redis.eval(lua_script, 1, self.lock_key, self.lock_value)
        self.acquired = False

    def __enter__(self):
        self.acquire()
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        self.release()


class FencingTokenGenerator:
    def __init__(self, redis_client: redis.Redis):
        self.redis = redis_client

    def next_token(self, lock_key: str) -> int:
        key = f"fencing:{lock_key}"
        return self.redis.incr(key)
```

## Architecture Decision Trees

### Lock Provider Selection

```
What's the consistency requirement?
├── Strong consistency, always correct
│   └── Redis (single node, WAIT for replication)
│       ├── Redlock for multi-node setups
│       └── Risk: clock drift, GC pauses
│
├── High availability over consistency
│   └── ZooKeeper / etcd
│       ├── Consensus-based, CP system
│       ├── Slower than Redis (RTT matters)
│       └── No clock drift issues
│
├── Simple, no infrastructure
│   └── Database-based (PostgreSQL advisory lock)
│       ├── pg_try_advisory_lock() for simple cases
│       └── SELECT ... FOR UPDATE for row-level locks
│
└── Cloud managed
    └── AWS ElastiCache Redis / Google Cloud Memorystore
```

## Anti-Patterns

| Anti-Pattern | Why It Fails | Correct Approach |
|---|---|---|
| No TTL on lock | Lock held forever if holder crashes | Always set TTL (lease duration) |
| Lock check + action not atomic | Race condition between check and action | Use Lua script or atomic operations |
| Single Redis node for lock | SPOF, lock lost on failover | Redlock or consensus-based provider |
| Not releasing in finally | Lock held on exception | Always release in finally block |
| Holding lock during I/O | Lock contention for long periods | Keep critical section short, only for critical path |
| No fencing tokens | GC pause can cause stale lock holder writes | Monotonic fencing tokens for write operations |

## Performance Optimization

- **Lock-free alternatives**: Use optimistic concurrency (version field, CAS) instead of locks where possible. Reduces latency from 5-50ms (lock round-trip) to <1ms (local CAS).
- **Read-write locks for read-heavy**: Allow concurrent reads, exclusive writes. Use Redis redlock with shared/exclusive modes. Multiple readers don't block each other.