Profile My skills Saved My plugins Edit profile Submit a skill

Mtp General

Implement, update, or review a general multi-token prediction (MTP) scheme for transformer LMs, including multi-head token-shift inputs, per-head targets, fusion blocks (norm + concat + projector + transformer layer), and inference-time multi-token decoding.

Happenmass Updated
AI & ML

File contents

General MTP (Multi-Token Prediction)

Use this skill when designing or debugging a general MTP implementation (not tied to a specific project).

Workflow

  • Read references/mtp-general.md for the current design, inputs/targets, and inference flow.
  • Keep training and inference consistent: head0 uses the base LLM output; subsequent heads use MTP fusion blocks with shifted speech tokens and eos alignment.
  • When changing the MTP structure, update both prepare_lm_input_target and inference logic together.
  • If integrating into a project, ensure config/registry points to the new MTP class and model selection logic supports it.

Guardrails

  • Preserve token alignment rules: each extra head removes one speech token and appends one eos token to keep lengths aligned.
  • Do not add extra eos to head0 inputs (targets already include eos).
  • Keep module grouping: MTP fusion block encapsulates norm, projector, and decoder layer.

Happenmass/ML-awesome-skills/tree/main/.codex/skills/mtp-general commit 45d4fc54c3

Browse category

Frequently asked questions

npx skillmds@latest add happenmass/mtp-general