Results for “deduplication”

GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.

GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.

Data Quality Specialist IA — Expert en qualité des données (profiling, cleaning, déduplication, validation de schéma, Great Expectations)

This skill should be used when a SKILL.md file needs compression, deduplication, or token reduction. It provides an embedding-based compression pipeline that detects and removes redundant chunks within SKILL.md files using local embeddings (all-MiniLM-L6-v2). Two-pass approach: (1) free intra-skill deduplication via cosine similarity clustering, (2) optional LLM-judged graded eval to detect pretraining overlap. Typical result: 25-46% token reduction with zero quality loss. This skill is not intended for editing skill content, creating new skills, routing optimization, or cross-skill deduplication.

Daily B2B prospect search via Apollo against the brand's ICP, with deduplication, scoring, and Notion CRM dropoff. Runs daily on cron schedule.

Deduplicating Training Data Makes Language Models Better

Extract, score, and backlog skill idea candidates from Claude Code session logs for a weekly skill generation pipeline.

Deploy DefectDojo as a centralized vulnerability management dashboard with scanner integrations, deduplication, metrics tracking, and Jira ticketing workflows.

Use to find duplicate or repeated support conversations and produce a reviewable merge plan. Trigger for "find duplicate tickets", "customers contacting us twice about the same thing", deduplicating a helpdesk, cleaning up ticket volume, or measuring how much of your contact volume is the same issue submitted more than once.

Refactors duplicated code by extracting helper methods, shared abstractions, or reusable structures, with automated testing and commits after each extraction.

Systematic debugging — reproduce, read the error, isolate, hypothesize, fix, verify, document. Use when tracking down a bug, unexpected behavior, or test failure — including phrases like "this doesn't work", "why is this failing", or "help me debug".

Aggregate RSS feeds from YouTube, Vimeo, Behance, Twitter/X, Bilibili, Weibo, Douyin, Xiaohongshu, and Zhihu with incremental updates, deduplication, and AI classification.

4-phase systematic debugging methodology with root cause analysis and evidence-based verification. Use when debugging complex issues.

Investigate a problem, log it to the debug changelog, cross-check past issues, and suggest a non-duplicate fix.

Detect and resolve naming collisions, overlapping intent triggers, and insufficient intent diversity across the skill library. Load when a new skill is created (called by universal-skill-creator), during improvement passes (called by improve-skills), or when the user asks to deconflict skills, check for naming collisions, audit trigger overlap, review intent coverage, find duplicate triggers, check skill names, or says "are any skills too similar", "which skills overlap", "deconflict the library", "check for confusing skill names", "intent audit".

Merges agent or review outputs into one deduplicated, evidence-linked, severity-ranked actionable report.

Wire ETL, ingestion, cron, edge-function, and queue jobs correctly. Use for "build a pipeline", "sync X into Y", "nightly aggregation", "cron double-counts", "dedupe", "backfill", "the numbers are wrong after a retry". Bakes in idempotency, atomic writes, data contracts, dead-letter, and observability.

Use when specialized expertise is needed to unblock delivery on this task.

Reproduces failures, gathers evidence, tests hypotheses, fixes root causes, and adds regression coverage.

Provides a structured debugging workflow for non-technical founders, covering reproduction, diagnostic gathering, and AI-assisted fixes.

You are the debugger. You follow the scientific method: observe, hypothesize, test, conclude. You never guess-and-patch.

Two-party working posture — user as director, agent as executor. Every fork, tradeoff, and taste choice is surfaced via batched AskUserQuestion with structural framing, a recommended default, and concrete previews when comparison is visual, so the human steers direction while the agent handles implementation. Eliminates the review-bottleneck (no giant diff to approve at the end — review is distributed across picks) and prevents codebase-understanding debt (the user remembers the architecture because they picked it). Use whenever the user invokes /duet, or says "work with me", "ask before", "check with me", "I want to decide", "don't assume", "human-in-the-loop", "co-author", "pair with me", "duet", or whenever a task clearly involves aesthetic, architectural, or irreversible strategic decisions — even without those exact words. Pair with the Duet output style to minimize cognitive load between picks.

Complete WhatsApp management for OpenClaw agents: per-conversation memory (groups + DMs), unanswered message tracking, loop prevention, and multi-PA coordination. Use when: tracking conversation context, recalling past decisions, finding unanswered messages, or preventing echo/duplicate message loops.

Runs a three-layer verification pipeline on AI-generated output: extracts verifiable claims, finds supporting or contradicting sources via web search, and produces a structured verification report with source links for human review.

Analyzes .NET test suites for duplicated boilerplate, copy-paste tests, and structural maintainability issues, producing a report with concrete before/after refactoring suggestions.

Debug Python: pdb REPL + debugpy remote (DAP).

Run a reproduce → isolate → verify debugging workflow for concrete bugs, regressions, flaky failures, and environment-specific behavior. Use when the user already has a failing command, test, request, UI flow, or narrowed symptom and needs root-cause diagnosis or fix verification rather than raw log-line selection, broad test-policy design, PR review, or generic performance tuning.

Use this skill for removing accidental complexity, reducing duplication, clarifying boundaries, preserving behavior. Trigger when the task involves programming work related to Code Simplification, production implementation, audits, debugging, strategy, or validation.

Step-by-step debugging workflow: reproduce the bug, isolate the failing component, trace to root cause, apply a targeted fix, and verify the fix resolves the issue without regressions. Use when you encounter a bug, error, exception, crash, or unexpected behavior that needs troubleshooting.

Three-layer verification pipeline for AI output. Extracts verifiable claims, finds supporting or contradicting sources via web search, runs adversarial review for hallucination patterns, and produces a structured verification report with source links for human review.