Results for “sharding”

Optimizes Qdrant query performance for large limits across multiple shards by using Poisson-distributed subsampling to reduce inter-shard data transfer.

Provides expert guidance on PyTorch Fully Sharded Data Parallel (FSDP) training, covering parameter sharding, mixed precision, CPU offloading, and FSDP2.

Load sharded datasets (npy, Parquet, HDF5, raw binary) into distributed cuPyNumeric arrays using manual partitioning and Legate task launches.

Build and debug Retrieval-Augmented Generation pipelines — chunking, embedding, retrieval, reranking

Use this skill for search indexing, embeddings, RAG chunking, freshness, retrieval evaluation, source citations. Trigger when the task involves ai engineering work related to Search Indexing RAG, implementation, audits, debugging, strategy, or validation.

Design a worked example fading sequence from fully worked examples through to independent practice. Use when teaching procedures, algorithms, or multi-step processes to novice learners.