Results for “deduplication”

GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.

Merges agent or review outputs into one deduplicated, evidence-linked, severity-ranked actionable report.

Two-party working posture — user as director, agent as executor. Every fork, tradeoff, and taste choice is surfaced via batched AskUserQuestion with structural framing, a recommended default, and concrete previews when comparison is visual, so the human steers direction while the agent handles implementation. Eliminates the review-bottleneck (no giant diff to approve at the end — review is distributed across picks) and prevents codebase-understanding debt (the user remembers the architecture because they picked it). Use whenever the user invokes /duet, or says "work with me", "ask before", "check with me", "I want to decide", "don't assume", "human-in-the-loop", "co-author", "pair with me", "duet", or whenever a task clearly involves aesthetic, architectural, or irreversible strategic decisions — even without those exact words. Pair with the Duet output style to minimize cognitive load between picks.

Run a reproduce → isolate → verify debugging workflow for concrete bugs, regressions, flaky failures, and environment-specific behavior. Use when the user already has a failing command, test, request, UI flow, or narrowed symptom and needs root-cause diagnosis or fix verification rather than raw log-line selection, broad test-policy design, PR review, or generic performance tuning.

Discover, compare, audit, repair, adapt, craft, run, debrief, and publish repeatable AI-agent loops for engineering workflows.

Build and debug Retrieval-Augmented Generation pipelines — chunking, embedding, retrieval, reranking

Use the @steipete/oracle CLI to bundle a prompt plus the right files and get a second-model review (API or browser) for debugging, refactors, design checks, or cross-validation.

Curate Claude Code's auto-memory into durable project knowledge. Analyze MEMORY.md for patterns, promote proven learnings to CLAUDE.md and .claude/rules/, extract recurring solutions into reusable skills. Use when: (1) reviewing what Claude has learned about your project, (2) graduating a pattern from notes to enforced rules, (3) turning a debugging solution into a skill, (4) checking memory health and capacity.

Data Quality Specialist IA — Expert en qualité des données (profiling, cleaning, déduplication, validation de schéma, Great Expectations)

Fine-tune, evaluate, run inference, and export Cosmos-Embed1 video-text embedding models for tasks like text-to-video retrieval and semantic deduplication.

Paper Retrieval Agent - Multi-database paper fetching from Semantic Scholar, OpenAlex, arXiv Handles rate limiting, deduplication, and PDF URL extraction Use when: fetching papers, searching databases, paper retrieval Triggers: fetch papers, retrieve papers, database search, Semantic Scholar, OpenAlex, arXiv