Embedding Analysis

Compute and analyze embeddings for dataset quality, distribution comparison, semantic deduplication, diversity measurement, and similarity-based filtering. Covers sentence-transformers, embedding space diagnostics, and 2025-2026 literature techniques (NeMo Curator semantic dedup, embedding similarity metrics for data selection, DataRater-style quality scoring).

mkurman Updated 327 repo stars
Picture from the README

File contents

mkurman/zorai commit b62b62ff1c

Frequently asked questions

npx get-skill add mkurman/embedding-analysis-2