Matrix Dimensionality Reduction

/matrix-dimensionality-reduction

When to Use

• Compress 1536-dim embeddings to 64-128 dims for faster [[in-memory-vector-storage]]
• PCA projection to reduce memory footprint by 10-20× at small accuracy cost
• Whitening embeddings before cosine similarity
• Matrix operations without Python/WASM dependency

Do NOT use for

• Production-scale PCA (> 100k vectors — use FAISS or scikit-learn)
• Neural network matrix ops (use TensorFlow.js or ONNX)

Matrix operations (plain TypeScript)

type Matrix = number[][]

// Matrix multiply A × B
function matMul(A: Matrix, B: Matrix): Matrix {
  const rows = A.length, cols = B[0].length, inner = B.length
  return Array.from({ length: rows }, (_, i) =>
    Array.from({ length: cols }, (_, j) =>
      A[i].reduce((sum, _, k) => sum + A[i][k] * B[k][j], 0)
    )
  )
}

// Transpose
function transpose(A: Matrix): Matrix {
  return A[0].map((_, j) => A.map(row => row[j]))
}

// Matrix-vector multiply
function matVecMul(A: Matrix, v: number[]): number[] {
  return A.map(row => row.reduce((sum, val, j) => sum + val * v[j], 0))
}

PCA projection (offline — fit on corpus, transform on query)

function meanCenter(X: Matrix): { centered: Matrix; mean: number[] } {
  const mean = X[0].map((_, j) => X.reduce((s, row) => s + row[j], 0) / X.length)
  const centered = X.map(row => row.map((v, j) => v - mean[j]))
  return { centered, mean }
}

// Covariance matrix (for small D)
function covariance(X: Matrix): Matrix {
  const n = X.length
  const XT = transpose(X)
  return matMul(XT, X).map(row => row.map(v => v / (n - 1)))
}

// Project embeddings to lower dimension using pre-computed projection matrix
function projectEmbeddings(embeddings: number[][], projMatrix: Matrix): number[][] {
  return embeddings.map(vec => matVecMul(projMatrix, vec))
}

// Usage: fit offline, use projMatrix in production
// const { centered, mean } = meanCenter(corpus)
// const projMatrix = computePCA(covariance(centered), dims=64)
// const compressed = projectEmbeddings(newEmbeddings, projMatrix)

L2 normalization (unit sphere projection)

function normalize(v: number[]): number[] {
  const norm = Math.sqrt(v.reduce((s, x) => s + x * x, 0))
  return norm > 0 ? v.map(x => x / norm) : v
}

// Normalize batch of embeddings
const normalized = embeddings.map(normalize)
// After normalization: cosine similarity = dot product (faster computation)

Dimension reduction ratio guide

Original dim  → Compressed  → Storage reduction
─────────────────────────────────────────────────
1536 (ada-002) → 128         → 12×
1536          → 64          → 24×
768 (MiniLM)  → 64          → 12×
Rule of thumb: keep 90% explained variance → ~20% of original dims

Anti-Fake-Pass Checklist

❌ PCA projection matrix computed on small sample → poor generalization
❌ Mean not subtracted before PCA → first principal component = mean direction
❌ matMul O(n³) — fine for dim < 1000, catastrophic for large matrices
❌ Projection matrix not persisted → must recompute from same corpus on restart
❌ Normalized embeddings compared with Euclidean distance → use dot product
❌ No dimension mismatch check → wrong projMatrix silently corrupts results