Slack Engineer

Mission: "Make work life simpler, more pleasant, and more productive." — Stewart Butterfield, 2013

Engineering Philosophy: "We don't sell saddles here." — Stewart Butterfield, 2013

Product Ethos: *"Utility curves" — Build things people actually want to use, not things that sound impressive. — Stewart Butterfield

§1 · Quick Start

§1.1 · One-Minute Setup

Activate this skill for Slack-style engineering:

# Add to CLAUDE.md
echo "Apply slack-engineer: Real-time messaging, API-first architecture, Bolt SDK patterns, Block Kit UI." >> CLAUDE.md

§1.2 · Essential Context

§1.3 · Core Capabilities

1. Real-Time Messaging Architecture — WebSocket-based pub/sub with Channel/Gateway/Presence Servers
2. Vitess MySQL Sharding — Horizontal scaling with channel-id as shard key
3. Bolt SDK Development — API-first bot building with Node.js/Python
4. Block Kit UI Design — Rich interactive components for Slack apps
5. API-First Integration — 2,600+ app directory, 650,000+ custom apps

§2 · Slack Engineering Culture

§2.1 · The Flickr Pivot & Stewart's Philosophy

From Glitch to Slack (2011-2013) Stewart Butterfield's game studio Tiny Speck was building "Glitch," an MMO that failed. The internal communication tool they built became Slack. Launched publicly February 2014:

• Day 1: 8,000 signups
• Week 2: 15,000 users
• Year 1: 285,000 daily active users
• IPO 2019: $19.5B valuation (direct listing)
• Acquisition 2021: $27.7B by Salesforce

"We Don't Sell Saddles Here" (2013)

"We're not selling a group chat system. We're selling organizational transformation." — Stewart Butterfield

§2.2 · The Three Cultural Pillars

1. Work Hard and Go Home

• Headquarters empty by 6:30 PM
• No hero culture around overtime
• Recharge to do better work tomorrow

2. 1% Increments

• Small daily improvements compound
• Special emoji celebrates micro-wins
• Continuous improvement over big bangs

3. Default to Open

• CEO avoids DMs, uses public channels
• #beef-tweets channel for product criticism
• Psychological safety through transparency

§2.3 · Hyper-Realistic Work-Like Activity

Stewart Butterfield's framework for avoiding "fake work":

Examples to Eliminate:

• Pre-meetings about meetings
• Slide decks that could be docs
• Status updates no one reads
• "Quick sync" calls that aren't

§3 · Technical Architecture

§3.1 · Real-Time Messaging at Scale

Core Services (All Java-based):

Message Journey:

Client A (send)
    ↓
Webapp API (Hack/PHP)
    ↓
Admin Server → Channel Server (consistent hash)
    ↓
All Gateway Servers (global)
    ↓
Client B, C, D... (receive in <500ms)

§3.2 · Consistent Hashing & CHARM

Channel Server Mapping:

• Channel ID hashed to specific CS host
• Consistent hash ring minimizes rebalancing
• CHARM (Consistent Hash Ring Manager) replaces failed CS in <20 seconds

Consistent Hashing:
  algorithm: rendezvous_hashing
  ring_manager: CHARM
  failover_time: <20_seconds
  channel_distribution: uniform_across_all_hosts

§3.3 · Vitess MySQL Sharding

Why Vitess:

• YouTube-origin sharding solution for MySQL
• Enables horizontal scaling without application changes
• Single master per shard with orchestrator failover

Sharding Strategy:

VtGate Query Flow:

Application → VtGate → Routes to correct shard(s)
                    ↓
         Scatter-Gather if needed
                    ↓
         Returns consolidated results

§3.4 · WebSocket Connection Model

Client Boot Sequence:

1. Fetch user token + WebSocket info from Webapp
2. Connect to nearest edge region (Envoy proxy)
3. Gateway Server authenticates via Webapp
4. GS subscribes to all CS hosts holding user's channels
5. Ready for real-time bidirectional communication

Transient Events (Typing Indicators):

• Not persisted to database
• Fast-path through GS → CS → All GS → Clients
• Lower latency than chat messages

§4 · Slack API & Bot Platform

§4.1 · API Foundations

Two API Types:

Cursor-Based Pagination:

# Instead of offset/limit (slow at scale)
response = client.conversations_history(
    channel="C123456",
    cursor="dXNlcjpVMDYxTkZUVDI=",  # Opaque pointer
    limit=100
)
# Next page: response['response_metadata']['next_cursor']

§4.2 · Bolt Framework

Bolt for JavaScript:

const { App } = require('@slack/bolt');

const app = new App({
  token: process.env.SLACK_BOT_TOKEN,
  socketMode: true,
  appToken: process.env.SLACK_APP_TOKEN
});

// Listen for messages
app.message('hello', async ({ message, say }) => {
  await say(`Hey there <@${message.user}>!`);
});

// Handle slash commands
app.command('/ticket', async ({ command, ack, respond }) => {
  await ack();
  await respond({
    blocks: [/* Block Kit UI */]
  });
});

Bolt for Python:

from slack_bolt import App
from slack_bolt.adapter.socket_mode import SocketModeHandler

app = App(token=os.environ["SLACK_BOT_TOKEN"])

@app.event("app_mention")
def handle_mention(event, say):
    say(f"Hello <@{event['user']}>")

if __name__ == "__main__":
    SocketModeHandler(app, os.environ["SLACK_APP_TOKEN"]).start()

§4.3 · Block Kit UI Framework

Core Principles:

• JSON-based declarative UI
• Up to 50 blocks per message
• Interactive components (buttons, select menus, date pickers)
• Modal support for complex workflows

Block Types:

Example Block Kit Message:

{
  "blocks": [
    {
      "type": "header",
      "text": {
        "type": "plain_text",
        "text": "New Ticket Created"
      }
    },
    {
      "type": "section",
      "fields": [
        {"type": "mrkdwn", "text": "*Priority:*\nHigh"},
        {"type": "mrkdwn", "text": "*Assignee:*\n<@U123>"}
      ]
    },
    {
      "type": "actions",
      "elements": [
        {
          "type": "button",
          "text": {"type": "plain_text", "text": "View Ticket"},
          "action_id": "view_ticket",
          "value": "TICKET-123"
        }
      ]
    }
  ]
}

§5 · Example Scenarios

§5.1 · Real-Time Message Broadcasting

Context: Design the message flow for a channel with 10,000 online users.

Slack-Engineer Approach:

Phase 1: Connection Establishment

Client Boot Sequence:
  1. Fetch: User token + WebSocket endpoint from Webapp (Hack/PHP)
  2. Connect: WebSocket to nearest edge region
  3. Gateway Server (GS): 
     - Authenticates user
     - Loads user's channel list from Webapp
     - Subscribes to all Channel Servers (CS) via consistent hash
  4. Ready: Bidirectional WebSocket established

Phase 2: Message Send Flow

# Client A sends message to channel C123
message_journey = {
    'client_a': {
        'action': 'send_message',
        'channel': 'C123',
        'content': 'Hello team!',
        'api': 'Webapp REST API'
    },
    'admin_server': {
        'action': 'route_to_cs',
        'cs_discovery': 'consistent_hash(C123)',
        'target_cs': 'cs-host-42.internal'
    },
    'channel_server': {
        'action': 'broadcast',
        'subscribers': 'all_gateway_servers_subscribed_to_C123',
        'distribution': 'global_fanout'
    },
    'gateway_servers': {
        'action': 'deliver_to_clients',
        'filter': 'clients_online_in_C123',
        'latency_target': '<500ms_global'
    }
}

Phase 3: Scaling Considerations

Load Characteristics:
  channel_size: 10_000_users
  online_rate: 0.6  # 60% online during work hours
  delivery_fanout: 6_000_clients
  
Resource Estimates:
  channel_servers: 1  # Handles all channels mapped via hash
  gateway_servers: 3  # Multi-region distribution
  bandwidth_per_message: "~100KB total (6K clients × ~16 bytes)"

§5.2 · Slack Bot Development with Bolt

Context: Build an incident management bot with Block Kit UI.

Slack-Engineer Approach:

Architecture:

┌─────────────┐     ┌──────────────┐     ┌─────────────┐
│   Slack     │◄───►│  Bolt App    │◄───►│  Incident   │
│   Client    │     │  (Node.js)   │     │  Database   │
└─────────────┘     └──────────────┘     └─────────────┘
                           │
                    ┌──────┴──────┐
                    ▼             ▼
              ┌─────────┐   ┌──────────┐
              │ PagerDuty│   │  Jira    │
              │   API    │   │   API    │
              └─────────┘   └──────────┘

Implementation:

const { App } = require('@slack/bolt');

const app = new App({
  token: process.env.SLACK_BOT_TOKEN,
  signingSecret: process.env.SLACK_SIGNING_SECRET
});

// Slash command to create incident
app.command('/incident', async ({ command, ack, client }) => {
  await ack();
  
  // Open modal with Block Kit
  await client.views.open({
    trigger_id: command.trigger_id,
    view: {
      type: 'modal',
      callback_id: 'incident_modal',
      title: { type: 'plain_text', text: 'Declare Incident' },
      blocks: [
        {
          type: 'input',
          block_id: 'title_block',
          element: {
            type: 'plain_text_input',
            action_id: 'title_input'
          },
          label: { type: 'plain_text', text: 'Incident Title' }
        },
        {
          type: 'input',
          block_id: 'severity_block',
          element: {
            type: 'static_select',
            action_id: 'severity_input',
            options: [
              { text: { type: 'plain_text', text: 'P0 - Critical' }, value: 'p0' },
              { text: { type: 'plain_text', text: 'P1 - High' }, value: 'p1' },
              { text: { type: 'plain_text', text: 'P2 - Medium' }, value: 'p2' }
            ]
          },
          label: { type: 'plain_text', text: 'Severity' }
        }
      ],
      submit: { type: 'plain_text', text: 'Create' }
    }
  });
});

// Handle modal submission
app.view('incident_modal', async ({ ack, body, view, client }) => {
  await ack();
  
  const values = view.state.values;
  const title = values.title_block.title_input.value;
  const severity = values.severity_block.severity_input.selected_option.value;
  
  // Create incident
  const incident = await createIncident({ title, severity });
  
  // Post to incident channel with rich UI
  await client.chat.postMessage({
    channel: '#incidents',
    blocks: [
      {
        type: 'header',
        text: { type: 'plain_text', text: `🚨 ${severity.toUpperCase()} Incident Declared` }
      },
      {
        type: 'section',
        fields: [
          { type: 'mrkdwn', text: `*ID:*\n${incident.id}` },
          { type: 'mrkdwn', text: `*Severity:*\n${severity.toUpperCase()}` }
        ]
      },
      {
        type: 'section',
        text: { type: 'mrkdwn', text: `*Title:* ${title}` }
      },
      {
        type: 'actions',
        elements: [
          {
            type: 'button',
            text: { type: 'plain_text', text: 'Join War Room' },
            style: 'primary',
            action_id: 'join_war_room',
            value: incident.id
          },
          {
            type: 'button',
            text: { type: 'plain_text', text: 'Update Status' },
            action_id: 'update_status',
            value: incident.id
          }
        ]
      }
    ],
    text: `Incident ${incident.id} declared: ${title}`  // Fallback
  });
});

Best Practices Applied:

• Modal for complex input (better UX than slash command args)
• Block Kit for rich formatting
• Action buttons for common workflows
• Fallback text for notifications

§5.3 · Vitess Sharding Migration

Context: Migrate a workspace-scaled database to channel-scaled Vitess sharding.

Slack-Engineer Approach:

Problem Analysis:

Before: Workspace-Scoped Sharding
  Issue: Large enterprises funnel all activity through one shard
  Hotspot: Single workspace = single database load
  Limitation: Can't scale individual large customers

After: Vitess Channel Sharding
  Shard Key: channel_id
  Benefit: Even distribution regardless of workspace size
  Scale: Can handle 16M+ channels per host

Migration Strategy:

migration_phases = {
    'phase_1_read_replicas': {
        'action': 'Set up Vitess alongside existing system',
        'verification': 'Dual-read consistency checks',
        'duration': '2 weeks'
    },
    'phase_2_write_shadowing': {
        'action': 'Write to both systems, read from old',
        'verification': 'Data divergence monitoring',
        'duration': '2 weeks'
    },
    'phase_3_read_cutover': {
        'action': 'Read from Vitess, write to both',
        'verification': 'Query latency monitoring',
        'rollback': 'Instant via config flag'
    },
    'phase_4_write_cutover': {
        'action': 'Write to Vitess only',
        'verification': '24-hour stability period'
    }
}

VtGate Configuration:

# Vitess topology
keyspaces:
  slack_messages:
    sharded: true
    shard_column: channel_id
    shards:
      - name: "-40"
        target: "replica"
      - name: "40-80"
        target: "replica"
      - name: "80-"
        target: "replica"

routing:
  # Single shard queries (most common)
  exact_match: "SELECT * FROM messages WHERE channel_id = ?"
  
  # Multi-shard scatter-gather
  in_query: "SELECT * FROM messages WHERE channel_id IN (?, ?, ?)"

§5.4 · API Integration Pattern

Context: Build a third-party integration that posts structured updates to Slack.

Slack-Engineer Approach:

Webhook Architecture:

External System → Incoming Webhook → Slack Channel
                        ↓
              Rate Limiting (1 msg/sec)
                        ↓
              Block Kit Formatting
                        ↓
              Channel Delivery

Implementation:

import requests
from datetime import datetime

def post_deployment_status(deployment):
    """Post deployment status to Slack via Incoming Webhook"""
    
    webhook_url = "https://hooks.slack.com/services/T000/B000/XXXX"
    
    payload = {
        "blocks": [
            {
                "type": "header",
                "text": {
                    "type": "plain_text",
                    "text": f"🚀 Deployment {deployment['status'].title()}"
                }
            },
            {
                "type": "section",
                "fields": [
                    {
                        "type": "mrkdwn",
                        "text": f"*Service:*\n{deployment['service']}"
                    },
                    {
                        "type": "mrkdwn",
                        "text": f"*Version:*\n`{deployment['version']}`"
                    },
                    {
                        "type": "mrkdwn",
                        "text": f"*Environment:*\n{deployment['env']}"
                    },
                    {
                        "type": "mrkdwn",
                        "text": f"*Duration:*\n{deployment['duration']}s"
                    }
                ]
            },
            {
                "type": "context",
                "elements": [
                    {
                        "type": "mrkdwn",
                        "text": f"By <@{deployment['deployer']}> at {datetime.now().isoformat()}"
                    }
                ]
            }
        ]
    }
    
    # Add conditional blocks based on status
    if deployment['status'] == 'failed':
        payload["blocks"].append({
            "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": f"*Error:*\n```{deployment['error'][:500]}```"
            }
        })
    
    response = requests.post(webhook_url, json=payload)
    response.raise_for_status()
    return response

Rate Limiting & Retry Logic:

import time
from functools import wraps

def slack_rate_limit(max_retries=3):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(max_retries):
                try:
                    return func(*args, **kwargs)
                except SlackApiError as e:
                    if e.response['error'] == 'rate_limited':
                        retry_after = int(e.response.headers.get('Retry-After', 1))
                        if attempt < max_retries - 1:
                            time.sleep(retry_after)
                            continue
                    raise
            return None
        return wrapper
    return decorator

§5.5 · Presence & Typing Indicators

Context: Implement real-time typing indicators at scale.

Slack-Engineer Approach:

Transient Event Flow:

Typing Event Flow:
  1. Client A: User starts typing
  2. WebSocket: Send 'user_typing' event to Gateway Server
  3. Gateway Server: Route to Channel Server (consistent hash)
  4. Channel Server: Fanout to all Gateway Servers subscribed to channel
  5. Gateway Servers: Push to all connected clients in channel (except sender)
  
Characteristics:
  persistence: none  # Not stored to database
  latency_target: <100ms
  rate_limit: "one event per 3 seconds per user"

Presence Server Architecture:

presence_system = {
    'servers': 'Presence Servers (PS) - in-memory',
    'sharding': 'user_id hash',
    'client_query': {
        'method': 'WebSocket via Gateway proxy',
        'batching': 'Only query visible users (viewport optimization)',
        'subscription': 'Real-time updates for queried users'
    },
    'status_types': ['active', 'away', 'dnd', 'offline'],
    'green_dot_logic': {
        'active': 'Currently using Slack',
        'away': 'Inactive for 10+ minutes',
        'update_frequency': 'Real-time via WebSocket'
    }
}

Client-Side Optimization:

// Viewport-aware presence subscription
const visibleUserIds = getVisibleUsersInChannel();

// Subscribe to presence updates only for visible users
client.subscribeToPresence({
  users: visibleUserIds,  // Not all 10,000 channel members
  onUpdate: (userId, status) => {
    updatePresenceIndicator(userId, status);
  }
});

// Debounced typing indicator
const sendTyping = debounce(() => {
  client.sendTyping(channelId);
}, 3000, { leading: true, trailing: false });

§6 · Tool Reference

§6.1 · Slack Developer Tools

§6.2 · Internal ↔ Open Source Mapping

§7 · Quality Checklist

§7.1 · Bot Development Review

• Token Security — Never commit tokens, use environment variables
• Rate Limiting — Handle 429s with exponential backoff
• Error Handling — Graceful degradation for API failures
• Block Kit Validation — Test in Block Kit Builder first
• Mobile Compatibility — Test on iOS/Android Slack apps
• Accessibility — Emoji have text alternatives

§7.2 · Real-Time System Review

• WebSocket Resilience — Auto-reconnect with backoff
• State Synchronization — Handle missed events on reconnect
• Latency Monitoring — Track P50/P95/P99 delivery times
• Fanout Limits — Consider large channel impact (>50K users)
• Regional Failover — Test edge region failover behavior

§7.3 · API Integration Review

• Scope Minimization — Request only needed OAuth scopes
• Token Rotation — Implement refresh token flow
• Payload Size — Keep messages under 40KB
• Retry Logic — Exponential backoff with jitter
• Idempotency — Prevent duplicate operations

§8 · Risk Framework

§8.1 · Common Pitfalls

§8.2 · Scaling Risks

WebSocket_Connections:
  risk: "Millions of concurrent connections"
  mitigation: "Multi-region Gateway Servers, connection pooling"
  
Message_Fanout:
  risk: "1 message → 10K+ deliveries"
  mitigation: "Consistent hashing, efficient pub/sub"
  
Database_Hotspots:
  risk: "Popular channels concentrate load"
  mitigation: "Vitess channel sharding, read replicas"

§9 · Learning Resources

§9.1 · Essential Reading

§9.2 · Stewart Butterfield's Key Insights

"What percentage of the intelligence, creativity, talent and experience of all the employees is actually being effectively applied towards the accomplishment of goals? At most companies it's 10%. I think 15-20% is possible."

"If you could get rid of a third of meetings, collectively you're talking about billions of hours of people's time per year."

"Two things that have served me well: really deep thinking about what is going on in the mind of the other person and what kind of incentives have been created."

§10 · Quick Reference Cards

§10.1 · Block Kit Template

{
  "blocks": [
    {
      "type": "header",
      "text": {"type": "plain_text", "text": "Title"}
    },
    {
      "type": "section",
      "text": {"type": "mrkdwn", "text": "*Bold* and ~strikethrough~"},
      "accessory": {
        "type": "button",
        "text": {"type": "plain_text", "text": "Click"},
        "action_id": "button_click"
      }
    },
    {
      "type": "divider"
    },
    {
      "type": "context",
      "elements": [
        {"type": "mrkdwn", "text": "Context footer"}
      ]
    }
  ]
}

§10.2 · Message Delivery SLA

Target Latency:
  Same region:     <50ms
  Cross-region:    <200ms
  Global worst:    <500ms
  
Availability:
  Core messaging:  99.99%
  File uploads:    99.9%
  Search:          99.5%

§10.3 · Bolt Quick Start

# Setup
npm install @slack/bolt
export SLACK_BOT_TOKEN=xoxb-...
export SLACK_SIGNING_SECRET=...

# Basic app
const { App } = require('@slack/bolt');
const app = new App({ token, signingSecret });
app.message('hello', ({ say }) => say('Hello!'));
(async () => { await app.start(3000); })();

End of Skill Document

"Becoming isn't about arriving somewhere. It's not about achieving an aim. It's not about selling saddles. It's about the journey." — Stewart Butterfield (adapted)

Workflow

Phase 1: Assessment

| Done | Phase completed | | Fail | Criteria not met |

• Gather requirements

| Done | All tasks completed | | Fail | Tasks incomplete |

• Analyze current state

Phase 2: Planning

| Done | Phase completed | | Fail | Criteria not met |

• Develop approach

| Done | All tasks completed | | Fail | Tasks incomplete |

• Set timeline

Phase 3: Execution

| Done | Phase completed | | Fail | Criteria not met |

• Implement solution

| Done | All tasks completed | | Fail | Tasks incomplete |

• Verify progress

Phase 4: Review

| Done | Phase completed | | Fail | Criteria not met |

• Validate outcomes

| Done | All tasks completed | | Fail | Tasks incomplete |

• Document lessons

Examples

Example 1: Standard Scenario

Input: Design and implement a slack engineer solution for a production system Output: Requirements Analysis → Architecture Design → Implementation → Testing → Deployment → Monitoring

Key considerations for slack-engineer:

• Scalability requirements
• Performance benchmarks
• Error handling and recovery
• Security considerations

Example 2: Edge Case

Input: Optimize existing slack engineer implementation to improve performance by 40% Output: Current State Analysis:

• Profiling results identifying bottlenecks
• Baseline metrics documented

Optimization Plan:

1. Algorithm improvement
2. Caching strategy
3. Parallelization

Expected improvement: 40-60% performance gain

§ 1.2 · Decision Framework — Weighted Criteria (0-100)

Composite Decision Rule:

• Score ≥85: Proceed
• Score 70-84: Conditional with monitoring
• Score <70: Stop and address issues

§ 1.3 · Thinking Patterns — Mental Models

Domain Benchmarks

Done Criteria

• All tasks completed per specification
• Quality standards met
• Stakeholder approval received

Fail Criteria

• Quality defects detected
• Requirements not met
• Timeline/budget overrun