Results for “distributed-compute”

Scale pandas and NumPy workflows to larger-than-memory datasets using parallel and distributed computing.

Framework for scaling Python applications from a laptop to a cluster. Includes Ray Core for distributed computing, Ray Serve for model serving, Ray Tune for hyperparameter optimization, and Ray Data for distributed data processing.

Designs and implements big data architectures, processes large-scale datasets with distributed systems, and optimizes data pipelines for throughput using Hadoop, Spark, and cloud platforms.

Distributed training orchestration across clusters. Scales PyTorch/TensorFlow/HuggingFace from laptop to 1000s of nodes. Built-in hyperparameter tuning with Ray Tune, fault tolerance, elastic scaling. Use when training massive models across multiple machines or running distributed hyperparameter sweeps.

Scales AI and Python applications across clusters with distributed computing primitives for ML workloads.

Write, optimize, and debug Apache Spark jobs for high-performance distributed data processing, ETL pipelines, and big data workloads.

Expert guidance for distributed NoSQL databases (Cassandra, DynamoDB). Focuses on mental models, query-first modeling, single-table design, and avoiding hot partitions in high-scale systems.

Orchestrate token-heavy research, coding, and testing by delegating bounded tasks to cheaper subagents while reserving Claude Fable for architecture, synthesis, and final review.

DSPy: declarative LM programs, auto-optimize prompts, RAG.

Distributed training orchestration across clusters. Scales PyTorch/TensorFlow/HuggingFace from laptop to 1000s of nodes. Built-in hyperparameter tuning with Ray Tune, fault tolerance, elastic scaling. Use when training massive models across multiple machines or running distributed hyperparameter sweeps.

Add distributed training support to any PyTorch script with minimal code changes using a unified API for DDP, DeepSpeed, FSDP, and mixed precision.

Process large-scale ML datasets with distributed streaming execution across CPU/GPU, supporting Parquet, CSV, JSON, images, and integration with PyTorch, TensorFlow, and Ray Train.

Scales Python ML workloads across clusters using Ray's distributed tasks, actors, data, and serving capabilities.

DSPy: declarative LM programs, auto-optimize prompts, RAG.

Master enterprise-grade Scala development with functional programming, distributed systems, and big data processing. Expert in Apache Pekko, Akka, Spark, ZIO/Cats Effect, and reactive architectures. Use PROACTIVELY for Scala system design, performance optimization, or enterprise integration.

Provides expert guidance for distributed training with DeepSpeed, covering ZeRO optimization stages, pipeline parallelism, FP16/BF16/FP8, 1-bit Adam, and sparse attention.

Connects agents to the GSTD Grid via MCP tools for autonomous payments, task discovery, and shared knowledge storage.

Systems ML and hardware-aware optimization specialist. Use when the user asks to talk to Killjoy, requests the systems researcher, or needs compute-aware architecture tradeoffs, distributed training, and inference optimization.