Minicoil Training

miniCOIL Training Reference

Complete reference for training miniCOIL sparse neural retrieval models. Covers the full pipeline from word vocabulary construction through evaluation.

What is miniCOIL?

miniCOIL is a lightweight sparse neural retrieval model created by Qdrant. It adds contextual semantic awareness to BM25-style keyword matching while preserving inverted-index compatibility.

The core problem: The same word has different meanings in different contexts ("fruit bat" vs. "baseball bat"). BM25 is blind to this. miniCOIL trains a tiny per-word linear layer that projects sentence embeddings into a 4D meaning space, letting the inverted index distinguish word senses.

Scoring Formula

score(D, Q) = SUM_{i: q_i in D}  IDF(q_i) * dot(layer_i(emb_Q), layer_i(emb_D))

Where layer_i is the per-word Linear(512, 4) + tanh projection for word i.

The sum runs only over words present in both Q and D. When a word is not in the vocabulary, it is invisible to miniCOIL (Qdrant can combine with BM25 in hybrid search to cover these).

Comparison with Alternatives

*SPLADE uses document expansion which reduces sparsity and increases index size.

Why not SPLADE? Requires relevance-labeled training data, making it domain-dependent. Document expansion increases index size and computational cost.

Why not COIL? Uses subword tokenization and end-to-end relevance training. Domain-dependent and incompatible with standard inverted indexes.

Why miniCOIL? Self-supervised (no labeled data), domain-independent, inverted-index compatible, and lightweight enough to train on a laptop.

Architecture

• Input encoder: jina-embeddings-v2-small-en (512D, 33M params)
• Mining encoder: mxbai-embed-large-v1 (1024D)
• Vocabulary: 30,000 most common English words (cleaned, stemmed, >3 chars)
• Layer: Linear(512, 4) + tanh per word
• Training data: 40M sentences from OpenWebText
• Training time: ~50 seconds per word on single CPU
• Training samples: ~8,000 sentences per word

Key Design Decisions

Why per-word? Each word gets its own tiny linear layer. This allows the model to learn distinct 4D representations for different contextual usages of the same word.

Why 4D output? Empirically found to be the sweet spot between expressiveness and sparsity. Each word occupies exactly 4 cells in the sparse vector. More dimensions = larger index, diminishing returns on quality.

Why tanh activation? Bounds output to [-1, 1], preventing any single word from dominating the sparse dot product. Allows negative values, which encode "this context is unlike the typical usage of this word."

Why triplet loss (not contrastive)? Triplet loss with semi-hard mining works well for learning relative similarity without needing explicit labels. The margin parameter directly controls how far apart different senses need to be in the 4D space.

Full Training Pipeline

Step 1: Build Word Vocabulary

Build a vocabulary of the 30,000 most common English words from frequency lists. Words are cleaned, stemmed, and filtered to keep only words with >3 characters.

Output: word_vocabulary.json (30,000 English words)

Step 2: Extract Training Sentences

Extract sentences from OpenWebText, match against the word vocabulary, and store in a training database.

Process:

1. Stream OpenWebText data (40 million sentences)
2. Split into sentences at sentence-ending punctuation
3. Filter: keep sentences with 5-100 words
4. Tokenize to lowercase, match against word vocabulary
5. Store with cap per word to avoid topical bias
6. Target: ~8,000 sentences per word

Step 3: Per-Word Training

The core training loop. For each word:

1. Fetch sentences containing this word from the training database
2. Encode with input encoder (jina-embeddings-v2-small-en, 512D)
3. Encode with mining encoder (mxbai-embed-large-v1, 1024D) for harder triplet mining
4. Mine triplets (see Triplet Mining section below)
5. Train Linear(512, 4) + tanh with TripletMarginLoss(margin=0.3), SGD
6. Save trained weight and bias

Training loop per word (pseudocode):

layer = nn.Linear(512, 4)
optimizer = SGD(layer.parameters())
loss_fn = TripletMarginLoss(margin=0.3)

for epoch in range(num_epochs):
    a_out = tanh(layer(embs[anchor_idx]))
    p_out = tanh(layer(embs[pos_idx]))
    n_out = tanh(layer(embs[neg_idx]))
    loss = loss_fn(a_out, p_out, n_out)
    loss.backward()
    optimizer.step()

Output: word_layers.pt -- dict of {word: {"weight": Tensor[4,512], "bias": Tensor[4]}}.

Step 4: Sparse Encoding

Converts text to sparse vectors at inference time:

1. Tokenize input text -> extract lowercase word tokens
2. Look up word indices in the vocabulary
3. Encode full text with jina-embeddings-v2-small-en -> 512D dense embedding
4. For each matched word: output = tanh(W @ emb + b) -> 4D meaning vector
5. Build sparse vector: word_index * 4 + offset -> value
6. Filter near-zero values (|v| < 1e-6)

Sparse index formula: sparse_index = word_index * OUTPUT_DIM + dim_offset where word_index is the word's position in the vocabulary (0-29999) and dim_offset is 0-3.

from minicoil.encoder import MiniCoilEncoder

encoder = MiniCoilEncoder("data/")

# Single text
sparse = encoder.encode_sparse("The cat sat on the mat")
# Returns: {168: 0.42, 169: -0.31, 170: 0.85, 171: -0.12, ...}

# Batch encoding
results = encoder.encode_batch_sparse(
    ["doc one", "doc two"],
    batch_size=64,
)

Step 5: Evaluation

Evaluates on BEIR retrieval benchmarks using IDF-weighted sparse dot product.

Pipeline:

1. Download BEIR dataset (zip from TU Darmstadt)
2. Encode corpus -> scipy CSR sparse matrix (cached to disk for reuse)
3. Compute IDF: log(1 + (N - df + 0.5) / (df + 0.5)) -- same formula as Qdrant's Modifier.IDF
4. Encode queries -> sparse vectors
5. IDF-weighted sparse dot product: score = q_sparse_idf @ corpus_csr.T
6. Compute nDCG@10

# English monolingual (BEIR)
minicoil evaluate --datasets nq quora fiqa hotpotqa msmarco

Benchmarks:

Triplet Mining Algorithm

The triplet mining is the heart of miniCOIL's training quality.

For each triplet to mine:

1. Select anchor: Random sentence index from the word's sentence set
2. Compute similarities: Cosine similarity between anchor and all other sentences using the mining encoder embeddings (mxbai-embed-large-v1)
3. Select positive (same-sense, high similarity):
   • Get top-20 most similar sentences (TRIPLET_TOPK=20)
   • Random selection from top-20
4. Select negative (different-sense, semi-hard):
   • Semi-hard criterion: neg_sim < pos_sim AND neg_sim > NEG_SIM_FLOOR (floor=-1.5)
   • Choose the closest negative (highest similarity that is still below positive) -- this is the "hardest" semi-hard negative
   • Fallback: if no semi-hard candidates exist, pick the least similar sentence overall

Adaptive triplet count

Actual triplets per word: min(num_triplets, num_sentences * TRIPLETS_PER_SENTENCE_MULTIPLIER). With TRIPLETS_PER_SENTENCE_MULTIPLIER=5, a word with 100 sentences gets min(5000, 500) = 500 triplets. This prevents over-mining words with few sentences.

Sparse Encoding Format

Vector Layout

Each word occupies 4 consecutive cells in the sparse vector:

Word 0:     indices 0, 1, 2, 3
Word 1:     indices 4, 5, 6, 7
Word 42:    indices 168, 169, 170, 171
...
Word 29999: indices 119996, 119997, 119998, 119999

Total sparse dimension: num_words * 4 = 30,000 * 4 = 120,000

Qdrant Integration

miniCOIL sparse vectors are directly compatible with Qdrant's sparse vector feature:

from qdrant_client import models

# Collection with IDF modifier (critical for miniCOIL)
client.create_collection(
    collection_name="minicoil",
    vectors_config={},
    sparse_vectors_config={
        "text": models.SparseVectorParams(
            modifier=models.Modifier.IDF,  # Qdrant computes IDF at query time
        )
    },
)

# Convert miniCOIL output to Qdrant format
sparse_dict = encoder.encode_sparse("The cat sat on the mat")
sv = models.SparseVector(
    indices=list(sparse_dict.keys()),
    values=list(sparse_dict.values()),
)

For hybrid search (dense + sparse), the dense embedding can be reused from the same jina-embeddings-v2-small-en encoder.

Hyperparameter Guidance

When to Change What

Default Parameters

Common Failure Modes and Debugging

Training

Problem: Training loss oscillates or NaN Cause: Learning rate too high or degenerate triplets Fix: Lower learning rate. Check that words have enough sentences (>= 10).

Problem: Training takes too long Cause: Large vocabulary (30k words) Fix: ~50 seconds per word on CPU. Total ~17 hours for full vocabulary on single CPU. Can be parallelized.

Evaluation

Problem: All nDCG@10 scores are 0.0 Cause: Sparse vectors are empty (no word matches in corpus) Fix: Check that the word vocabulary matches the trained model. Verify tokenization is consistent.

Problem: Eval scores lower than expected Cause: Wrong IDF weighting or tokenization mismatch Fix: Ensure IDF formula matches Qdrant's Modifier.IDF. Check tokenization produces same tokens at train and eval time.

Key References

• miniCOIL article (Qdrant)
• GitHub - qdrant/miniCOIL
• FastEmbed integration (v0.7.0+)
• HuggingFace model (Qdrant/minicoil-v1)
• BEIR benchmark

See resources/architecture.md for detailed architecture diagrams and data flow.