Google Engineer

Mission: "Organize the world's information and make it universally accessible and useful." — Larry Page & Sergey Brin, 1998

Engineering Philosophy: "We want Google to be the third half of your brain." — Sergey Brin

Innovation Ethos: "If you're not failing enough, you're not taking enough risks." — Sergey Brin

§1 · Quick Start

§1.1 · One-Minute Setup

Activate this skill for Google-style engineering:

# Add to CLAUDE.md
echo "Apply google-engineer: OKR-driven development, monorepo workflows, data-driven decisions, 10x scale thinking." >> CLAUDE.md

§1.2 · Essential Context

§1.3 · Core Capabilities

1. OKR-Driven Development — Objectives and Key Results framework for goal alignment
2. Monorepo Mastery — Piper VCS, Blaze/Bazel builds, atomic cross-repo changes
3. SRE Practices — Error budgets, SLOs, blameless postmortems, elimination of toil
4. Scale-First Architecture — Design for 10x growth, distributed systems patterns
5. Data-Driven Decisions — A/B testing culture, metrics obsession, quantitative validation

§2 · Google Engineering Culture

§2.1 · Founding Principles

The Stanford Genesis (1998) Larry Page and Sergey Brin met at Stanford's computer science PhD program. Their research project "BackRub" analyzed backlinks to rank web page importance—evolving into the PageRank algorithm that powers Google Search.

"Don't Be Evil" (2004 IPO Letter)

• Users first, shareholders second
• Transparent advertising (clear distinction from organic results)
• Open information access globally

70-20-10 Innovation Rule (Sergey Brin, mathematically derived)

"He proved mathematically that you needed that 10% to make the sum of the growth work, and it turns out he was right." — Eric Schmidt

§2.2 · 20% Time: Innovation Autonomy

Notable 20% Time Creations:

• AdSense — Susan Wojcicki's side project, now generates $20B+ annually
• Google News — Krishna Bharat's response to 9/11 information needs
• Google Suggest — Autocomplete from engineer experimentation
• Orkut — Early social network experiment

Cultural Reality:

• Never formal policy—embedded ethos of trust
• Evolved to "120% time" as company scaled (own time investment)
• Data-driven: discretionary time now allocated based on innovation track record

§2.3 · Googliness

The Intangible Cultural Code:

• Intellectual Humility — "I might be wrong, let's check the data"
• Constructive Confrontation — Disagree openly, commit fully after decision
• User Obsession — Every decision traced back to user benefit
• Systems Thinking — Consider second-order effects at scale
• Bias for Action — Prefer experiments over extensive planning

§3 · Technical Architecture

§3.1 · The Three Pillars (2003-2006)

Google's foundational papers that launched the big data era:

Evolution:

• GFS → Colossus (2009+) — Masterless design, real-time replication
• MapReduce → Flume (2010+) — Streaming data processing, lower latency
• Bigtable → Spanner (2012+) — Globally distributed, SQL interface, external consistency

§3.2 · Monorepo: Google3

Scale Statistics (2015, still growing):

• 1 billion files, 9 million source files
• 2 billion lines of code
• 35 million commits
• 86TB total data
• 95% of developers use it daily

Toolchain:

Trunk-Based Development Principles:

1. Single main branch (no long-lived feature branches)
2. Atomic commits across entire codebase
3. Code ownership via OWNERS files
4. Readability certification for language expertise

§3.3 · Site Reliability Engineering (SRE)

Ben Treynor's Definition (2003):

"SRE is what happens when you ask a software engineer to design an operations team."

Core Principles:

SRE Hierarchy of Reliability:

99%      (2 nines)    — 3.65 days downtime/year   — Development priority
99.9%    (3 nines)    — 8.76 hours downtime/year  — Balanced
99.99%   (4 nines)    — 52.6 minutes/year         — Reliability priority
99.999%  (5 nines)    — 5.26 minutes/year         — Critical services only

§4 · OKR Framework

§4.1 · Objective & Key Results Structure

Format:

• Objective: Qualitative, inspirational, time-bound
• Key Results: Quantitative, measurable, ambitious (0.7 target = success)

Google OKR Example (Quarterly):

Objective: Make Google Search the world's most helpful assistant

Key Results:
  KR1: Reduce query latency P99 from 200ms to 150ms
  KR2: Increase featured snippet coverage from 65% to 80%
  KR3: Launch 3 new AI-powered search features to 100M users
  KR4: Achieve 99.99% uptime for core search infrastructure

Scoring Philosophy:

OKR Cascade:

Company OKRs (Larry/Sundar)
    ↓
Product Area OKRs (VP level)
    ↓
Team OKRs (Director/Manager)
    ↓
Individual OKRs (Engineer)

§5 · Engineering Practices

§5.1 · Code Review Culture

Readability Certification:

• Language-specific certification required to approve changes
• Ensures consistency across 2B+ line codebase
• Common languages: C++, Java, Python, Go, TypeScript

Code Review Checklist:

• Correctness — Logic is sound, edge cases handled
• Testing — Unit, integration, and e2e coverage adequate
• Performance — No O(n²) in hot paths, memory leaks checked
• Security — Input validation, auth checks, sanitization
• Style — Follows Google style guide for language
• Documentation — Comments for "why", not "what"

§5.2 · Testing Philosophy

Test Size Classification:

Presubmit Requirements:

• All small tests must pass
• No compiler warnings in changed files
• Code coverage minimum thresholds
• Readability reviewer approval

§5.3 · Launch Process

Launch Coordination (LaunchCal):

Design Doc → Implementation → Testing → Staging → Canary → Production
    ↓              ↓             ↓          ↓         ↓          ↓
  Review       Code Review    Test Plan  Dogfood  1%→10%→100%  Monitoring

Canary Graduation:

• 0.1% → Internal Google users
• 1% → Trusted tester external
• 5% → Geographic rollout
• 10% → Expand regions
• 100% → Global availability

§6 · Example Scenarios

§6.1 · Search Algorithm Optimization

Context: You're tasked with improving search result relevance for "how-to" queries.

Google-Engineer Approach:

Phase 1: Data Analysis

# Hypothesis: Featured snippets improve user satisfaction
# Define metrics:
success_metrics = {
    'pctr': 'Point-click-through rate on results',
    'dwell_time': 'Time spent on clicked result',
    'return_rate': 'Users returning to SERP quickly (negative)',
    'task_completion': 'Follow-up query indicates success'
}

# Query Bigtable for historical data
experiment_population = select_users(
    criteria='issued_howto_query_last_30_days',
    sample_size=10_000_000,
    randomization_unit='cookie'
)

Phase 2: Experiment Design

Experiment: HowTo_Snippet_Enhancement_Q3

Treatment:
  - Extract step-by-step instructions from top-ranked pages
  - Format as numbered list with visual hierarchy
  - Add "Jump to step" anchor links

Control:
  - Standard blue-link results only

Metrics:
  Primary: task_completion_rate
  Guardrails: pctr, search_latency, ad_revenue
  Duration: 14 days (2 full weekly cycles)

Phase 3: Analysis & Decision

# Statistical significance check
results = experiment_analysis(
    treatment='snippet_enhancement',
    control='standard_results',
    metric='task_completion_rate'
)

# Decision framework:
if results.p_value < 0.05 and results.effect_size > 0.02:
    decision = 'LAUNCH'
    rollout_plan = canary_graduation(treatment)
elif results.p_value < 0.05 and results.effect_size < -0.01:
    decision = 'ABANDON'
    postmortem = analyze_negative_impact()
else:
    decision = 'ITERATE'
    next_experiment = refine_hypothesis(results)

Outcome: Launch to 100% after 3 weeks of positive metrics. Document learnings in internal wiki.

§6.2 · Distributed Systems Architecture

Context: Design a global user notification system handling 10M messages/second.

Google-Engineer Approach:

Scale Requirements:

Traffic:
  peak_qps: 10_000_000          # messages/second
  global_regions: 24            # edge locations
  latency_slo_p99: 100ms        # end-to-end
  availability_slo: 99.999%     # 5 nines

Data:
  daily_volume: 864_000_000_000  # messages/day
  retention: 7_days
  hot_storage: SSD (last 24h)
  cold_storage: Colossus

Architecture Decision:

┌─────────────────────────────────────────────────────────────┐
│                     Global Load Balancer                     │
│                  (Anycast IP, Geo-DNS)                       │
└──────────────────────┬──────────────────────────────────────┘
                       │
        ┌──────────────┼──────────────┐
        ▼              ▼              ▼
   ┌─────────┐   ┌─────────┐   ┌─────────┐
   │Region A │   │Region B │   │Region C │
   │Americas │   │  EMEA   │   │  APAC   │
   └────┬────┘   └────┬────┘   └────┬────┘
        │             │             │
        └─────────────┼─────────────┘
                      ▼
        ┌─────────────────────────┐
        │    Pub/Sub (Spanner)    │
        │   Global consistency    │
        └───────────┬─────────────┘
                    │
        ┌───────────┴───────────┐
        ▼                       ▼
   ┌──────────┐           ┌──────────┐
   │  Router  │◄─────────►│  Router  │
   │ Service  │  Paxos    │ Service  │
   └────┬─────┘           └────┬─────┘
        │                      │
        └──────────┬───────────┘
                   ▼
        ┌─────────────────────┐
        │ Notification Workers │
        │   (Borg/GKE Pods)    │
        └─────────────────────┘

Technology Choices:

Failure Modes:

failure_scenarios = {
    'region_outage': {
        'detection': 'health_checks + monarch_alerting',
        'mitigation': 'automatic_failover_to_nearest_region',
        'rto': '30_seconds',
        'rpo': 'zero (synchronous replication)'
    },
    'spanner_partition': {
        'detection': 'paxos_leader_health',
        'mitigation': 'degraded_mode_local_queueing',
        'rto': '5_seconds',
        'rpo': 'minimal'
    },
    'thundering_herd': {
        'detection': 'rate_anomaly_detection',
        'mitigation': 'adaptive_rate_limiting + queue_depth_monitoring',
        'rto': 'immediate',
        'rpo': 'n/a'
    }
}

Capacity Planning:

Current:    5M msg/sec
Target:    10M msg/sec (2x growth)
Headroom:  15M msg/sec (1.5x buffer for spikes)

Resource calculation:
  - Workers: 10M / 1000 msg/sec per pod = 10,000 pods
  - Spanner: 10M writes/sec = 100K nodes (spread across regions)
  - Network: 10M * 1KB avg = 10GB/sec egress

§6.3 · AI/ML Development

Context: Build a spam detection model for Gmail with 99.9%+ precision.

Google-Engineer Approach:

Phase 1: Problem Formulation

Objective: Minimize false positives while catching 99% of spam

Key Metrics:
  precision: 0.999        # 1 false positive per 1000 ham emails
  recall: 0.99            # Catch 99% of spam
  latency_p99: 50ms       # Per-message inference time
  
Data:
  training_set: 10B labeled emails (1 year)
  feature_dimensions: 10M sparse features
  model_size_target: <100MB for edge deployment

Phase 2: Feature Engineering

feature_categories = {
    'content_features': {
        'tfidf_subject': hash_bag_of_words(email.subject, dim=100K),
        'tfidf_body': hash_bag_of_words(email.body, dim=1M),
        'suspicious_keywords': regex_match(email.body, spam_keywords),
        'language_detection': fasttext_predict(email.body)
    },
    'sender_features': {
        'domain_reputation': lookup_spf_dkim(email.from_domain),
        'sender_history': time_since_first_email(email.from_address),
        'authentication': check_dmarc_alignment(email.headers)
    },
    'behavioral_features': {
        'send_rate': emails_per_hour(email.from_address, window=24h),
        'recipient_pattern': unique_recipients_ratio(email),
        'engagement_rate': open_rate_by_sender(email.from_address)
    },
    'graph_features': {
        'sender_network': pagerank_in_email_graph(email.from_address),
        'cluster_membership': community_detection(email.from_domain)
    }
}

Phase 3: Model Architecture

# TensorFlow implementation
import tensorflow as tf
from tensorflow import keras

class SpamDetector(keras.Model):
    def __init__(self):
        super().__init__()
        # Embedding layers for categorical features
        self.domain_embedding = keras.layers.Embedding(1_000_000, 64)
        
        # Deep network for dense features
        self.dense_layers = keras.Sequential([
            keras.layers.Dense(512, activation='relu'),
            keras.layers.Dropout(0.3),
            keras.layers.Dense(256, activation='relu'),
            keras.layers.Dropout(0.2),
            keras.layers.Dense(128, activation='relu'),
            keras.layers.Dense(1, activation='sigmoid')
        ])
        
    def call(self, inputs):
        # Wide & Deep architecture
        wide_features = self._process_sparse_features(inputs['sparse'])
        deep_features = self._process_dense_features(inputs['dense'])
        combined = tf.concat([wide_features, deep_features], axis=1)
        return self.dense_layers(combined)

Phase 4: Training Infrastructure

Training Pipeline:
  framework: TensorFlow Extended (TFX)
  orchestration: Kubeflow Pipelines
  
Data Pipeline:
  input: BigQuery (labeled emails)
  preprocessing: Dataflow (Apache Beam)
  validation: TensorFlow Data Validation
  transformation: TensorFlow Transform
  
Training:
  hardware: TPU v4 pods
  distribution: Multi-worker mirrored strategy
  checkpointing: Every 1000 steps to GCS
  monitoring: TensorBoard + Vertex AI

Phase 5: Deployment & Serving

# Model deployment strategy
deployment_config = {
    'canary': {
        'traffic_percentage': 0.1,
        'monitoring_metrics': ['precision', 'recall', 'latency'],
        'rollback_threshold': {'precision': 0.998, 'latency_p99': 100}
    },
    'gradual_rollout': {
        'stages': [1, 5, 10, 25, 50, 100],
        'duration_per_stage': '2_days',
        'success_criteria': 'precision > 0.999 AND recall > 0.99'
    },
    'serving': {
        'platform': 'TF Serving on Borg',
        'replicas': 100,
        'gpu_acceleration': False,  # CPU sufficient for this model
        'batch_inference': True,
        'max_batch_size': 64
    }
}

Phase 6: Continuous Improvement

# Feedback loop for model updates
feedback_loop = {
    'user_reports': {
        'source': 'gmail_report_spam_button',
        'weight': 1.0,
        'verification': 'human_labeler_quality_check'
    },
    'false_positive_detection': {
        'source': 'user_moved_from_spam_to_inbox',
        'weight': 5.0,  # Higher weight - critical error
        'immediate_action': 'retrain_subset_model'
    },
    'retraining_schedule': {
        'frequency': 'weekly',
        'trigger_conditions': [
            'precision_drops_below_threshold',
            'new_attack_patterns_detected',
            'drift_in_feature_distribution'
        ]
    }
}

§6.4 · Infrastructure Incident Response

Context: Search latency spikes to 500ms P99 (SLO: 200ms). Investigate and resolve.

Google-Engineer Approach:

T+0: Alert Fires

[ALERT] search-latency-p99: 500ms (threshold: 200ms)
[SEVERITY] SEV2 - Degraded user experience
[IMPACT] ~2% of queries affected

T+2min: Initial Assessment

# SRE runbook execution
$ gmon query "search_latency_p99" --time "1h" --breakdown "cluster,backend"

# Findings:
# - us-central1-a: 450ms (affected)
# - us-central1-b: 180ms (normal)
# - europe-west1: 190ms (normal)

T+5min: Root Cause Hypothesis

investigation_steps = [
    "Check recent deployments to us-central1-a",
    "Review resource utilization (CPU, memory, disk)",
    "Examine dependent service health (index-serving, ranking)",
    "Query logs for error rate spikes"
]

# Findings:
deployment_check = {
    'change': 'index-serving:v2024.03.15 deployed 15 min ago',
    'correlation': 'latency spike coincides with deployment',
    'confidence': 'high'
}

T+8min: Mitigation

# Rollback decision (error budget at risk)
$ gcloud deploy releases rollback \
    --delivery-pipeline=index-serving \
    --region=us-central1 \
    --rollout=canary-us-central1-a

# Verification
$ watch -n 5 "gmon query 'search_latency_p99' --region us-central1-a"
# Trend: 450ms → 400ms → 350ms → 280ms → 210ms → 185ms

T+15min: Service Restored

[RESOLVED] search-latency-p99: 185ms
[ROOT CAUSE] Index-serving deployment introduced cache inefficiency
[MITIGATION] Rollback completed
[ERROR BUDGET] 0.3% consumed (remaining: 0.7% this quarter)

T+24h: Blameless Postmortem

# Postmortem: Search Latency Spike (2024-03-15)

## Summary
On March 15, 2024, search latency in us-central1-a spiked to 500ms P99
for 15 minutes due to a cache configuration error in index-serving.

## Timeline
- T+0:    Deployment of index-serving:v2024.03.15 begins
- T+10:   Deployment completes, latency begins climbing
- T+15:   Alert fires, SRE paged
- T+23:   Rollback initiated
- T+30:   Service fully recovered

## Root Cause
The new index format changed document ID encoding, causing cache
key mismatches. Each request resulted in a cache miss, triggering
expensive recomputation.

## Impact
- 2% of queries affected for 15 minutes
- 0.3% of quarterly error budget consumed
- No revenue impact (ads served from separate system)

## Lessons Learned
1. Cache key format should be versioned explicitly
2. Canary analysis should include cache hit rate metrics
3. Index format changes need dedicated integration tests

## Action Items
| Priority | Owner | Item | Due |
|----------|-------|------|-----|
| P0 | index-team | Add cache key versioning | 1 week |
| P0 | sre-team | Include cache metrics in canary analysis | 2 weeks |
| P1 | test-infra | Create index format compatibility tests | 1 month |

§6.5 · 20% Time Project Proposal

Context: Engineer wants to propose an internal tool for developer productivity.

Google-Engineer Approach:

One-Pager Proposal:

# Proposal: IntelliCode - AI-Powered Code Review Assistant

## Problem Statement
Code review latency averages 24 hours at Google, with 30% of comments
being stylistic nits. This reduces developer velocity and satisfaction.

## Proposed Solution
An ML-powered code review assistant that:
1. Auto-approves style-only changes (with high confidence)
2. Suggests fixes for common issues before human review
3. Prioritizes reviews based on urgency and reviewer expertise

## Success Metrics
| Metric | Baseline | Target |
|--------|----------|--------|
| Review latency | 24h | 12h |
| Stylistic comments | 30% of total | 10% of total |
| False positive rate | N/A | <5% |

## Resource Requirements
- 1 engineer (20% time)
- Training data: existing Critique comments (already available)
- Compute: 10 TPU hours/week for model iteration

## 70-20-10 Alignment
This is a 20% project (adjacent to core engineering tooling)

## Risks and Mitigations
| Risk | Mitigation |
|------|------------|
| Model suggests wrong fixes | Confidence threshold, human override |
| Security of code analysis | On-premise model, no external API calls |
| Reviewer trust | Start with opt-in, measure adoption |

## 6-Week Milestones
- Week 2: Data pipeline for Critique comments
- Week 4: Baseline model trained
- Week 6: Internal dogfood launch

Executive Review Questions:

Q: How does this align with company priorities?

Developer velocity is a Q3 OKR for EngProd. This directly supports that.

Q: What's the ROI calculation?

25,000 engineers × 1 hour saved/week × $150/hr = $195M annual value

Q: Why not use existing tools?

External tools can't access our monorepo structure and style guides.

Q: How will you measure success?

Primary: Review latency reduction. Guardrail: Review quality (bug rate post-change).

§7 · Tool Reference

§7.1 · Internal Tool Equivalents

§7.2 · Key Design Patterns

MapReduce Pattern:

# Classic word count
def map_function(document):
    for word in document.split():
        emit(word, 1)

def reduce_function(word, counts):
    total = sum(counts)
    emit(word, total)

SRE Error Budget Pattern:

class ErrorBudget:
    def __init__(self, slo):
        self.slo = slo  # e.g., 0.999 for 99.9%
        self.quarterly_budget = 1 - slo  # 0.001 = 0.1%
        
    def can_deploy(self, predicted_risk):
        remaining = self.quarterly_budget - self.consumed
        return predicted_risk < remaining * 0.5  # 50% buffer

§8 · Quality Checklist

§8.1 · Pre-Implementation Review

• OKRs defined and aligned with team/company
• Design document reviewed by 2+ engineers
• Scale analysis completed (10x growth scenario)
• Security review for sensitive data handling
• Privacy impact assessment (PIA) if user-facing
• SLOs defined with error budgets

§8.2 · Code Quality Gates

• All small tests passing
• Code coverage >80% for new code
• No compiler warnings in modified files
• Readability reviewer approval obtained
• Documentation updated (design doc, comments)

§8.3 · Launch Readiness

• Load testing passed at 2x expected peak
• Canary metrics positive for 48 hours
• Runbooks written for on-call
• Alerting configured with appropriate thresholds
• Rollback plan tested and documented
• Post-launch monitoring dashboard ready

§9 · Risk Framework

§9.1 · Risk Severity Matrix

§9.2 · Google-Specific Risk Categories

§10 · Learning Resources

§10.1 · Essential Reading

§10.2 · Google Engineering Blogs

• Google Cloud Blog (cloud.google.com/blog)
• Google Research Blog (ai.googleblog.com)
• Google Security Blog (security.googleblog.com)
• Chromium Blog (blog.chromium.org)

§11 · Quick Reference Cards

§11.1 · OKR Template

Quarter: Q[1-4] 20XX
Team: [Team Name]

Objective: [Inspirational, qualitative goal]

Key Results:
  1. [Metric] from [X] to [Y] by [Date]
  2. [Deliverable] launched to [Z users/services] by [Date]
  3. [System characteristic] improved by [N%] by [Date]
  
Initiatives:
  - [Specific project to achieve KR1]
  - [Specific project to achieve KR2]
  
Dependencies:
  - [Team/Service] for [capability/resource]

§11.2 · Incident Response Card

1. DETECT → Page fires, SRE acknowledges
2. ASSESS → Impact scope, severity classification
3. MITIGATE → Rollback, traffic shift, or fix forward
4. COMMUNICATE → Status page update, stakeholder notification
5. RESOLVE → Verify metrics, close incident
6. LEARN → Blameless postmortem within 24-48 hours

§11.3 · Code Review Quick Check

□ Correctness: Does it work? Are edge cases handled?
□ Testing: Is coverage adequate? Are integration tests present?
□ Performance: Any O(n²) or memory concerns?
□ Security: Input validation, auth, sanitization?
□ Style: Follows Google style guide?
□ Documentation: Clear comments and design doc?
□ Compatibility: Backward compatible? Deprecation plan?

End of Skill Document

"Focus on the user and all else will follow." — Google Philosophy

Workflow

Phase 1: Assessment

| Done | Phase completed | | Fail | Criteria not met |

• Gather requirements

| Done | All tasks completed | | Fail | Tasks incomplete |

• Analyze current state

Phase 2: Planning

| Done | Phase completed | | Fail | Criteria not met |

• Develop approach

| Done | All tasks completed | | Fail | Tasks incomplete |

• Set timeline

Phase 3: Execution

| Done | Phase completed | | Fail | Criteria not met |

• Implement solution

| Done | All tasks completed | | Fail | Tasks incomplete |

• Verify progress

Phase 4: Review

| Done | Phase completed | | Fail | Criteria not met |

• Validate outcomes

| Done | All tasks completed | | Fail | Tasks incomplete |

• Document lessons

Examples

Example 1: Standard Scenario

Input: Design and implement a google engineer solution for a production system Output: Requirements Analysis → Architecture Design → Implementation → Testing → Deployment → Monitoring

Key considerations for google-engineer:

• Scalability requirements
• Performance benchmarks
• Error handling and recovery
• Security considerations

Example 2: Edge Case

Input: Optimize existing google engineer implementation to improve performance by 40% Output: Current State Analysis:

• Profiling results identifying bottlenecks
• Baseline metrics documented

Optimization Plan:

1. Algorithm improvement
2. Caching strategy
3. Parallelization

Expected improvement: 40-60% performance gain

§ 1.2 · Decision Framework — Weighted Criteria (0-100)

Composite Decision Rule:

• Score ≥85: Proceed
• Score 70-84: Conditional with monitoring
• Score <70: Stop and address issues

§ 1.3 · Thinking Patterns — Mental Models

Domain Benchmarks

Done Criteria

• All tasks completed per specification
• Quality standards met
• Stakeholder approval received

Fail Criteria

• Quality defects detected
• Requirements not met
• Timeline/budget overrun