Extend Language Support

This skill builds on configure-selective-highlighting. Read it first for explicit registration and optional embedding.

Extend Language Support

Setup

Define an isolated tokenizer that returns semantic source ranges:

import {
  createHighlighter,
  defineLanguage,
  type TokenRange,
} from '@tanstack/highlight/core'

const taskList = defineLanguage({
  name: 'task-list',
  aliases: ['tasks'],
  tokenize(code) {
    const ranges: Array<TokenRange> = []
    const pattern = /\b(?:TODO|DONE|BLOCKED)\b/g
    let match: RegExpExecArray | null

    while ((match = pattern.exec(code))) {
      ranges.push({
        start: match.index,
        end: match.index + match[0].length,
        className: match[0] === 'DONE' ? 'inserted' : 'keyword',
      })
    }

    return ranges
  },
})

export const highlighter = createHighlighter({
  languages: [taskList],
})

Gaps remain plain text automatically. Ranges must be non-overlapping, zero-based, end-exclusive UTF-16 offsets.

Core Patterns

Preserve exact source through tokenization

import { highlighter } from './task-list-highlighter'

const code = 'TODO write docs\nDONE add tests'
const result = highlighter.tokenize(code, { lang: 'tasks' })

export const sourceWasPreserved =
  result.tokens.map((token) => token.value).join('') === code

Source reconstruction is the primary tokenizer invariant.

Delegate embedded content through the registry

import {
  createHighlighter,
  defineLanguage,
  type TokenRange,
} from '@tanstack/highlight/core'
import { js } from '@tanstack/highlight/languages/js'

const scriptDocument = defineLanguage({
  name: 'script-document',
  tokenize(code, context) {
    const ranges: Array<TokenRange> = []
    const pattern = /<script>([\s\S]*?)<\/script>/g
    let match: RegExpExecArray | null

    while ((match = pattern.exec(code))) {
      const openStart = match.index
      const bodyStart = openStart + '<script>'.length
      const bodyEnd = bodyStart + match[1].length
      const closeEnd = bodyEnd + '</script>'.length

      ranges.push({ start: openStart, end: bodyStart, className: 'tag' })

      if (context.hasLanguage('js')) {
        ranges.push(
          ...context.tokenize(match[1], 'js').map((range) => ({
            ...range,
            start: range.start + bodyStart,
            end: range.end + bodyStart,
          })),
        )
      }

      ranges.push({ start: bodyEnd, end: closeEnd, className: 'tag' })
    }

    return ranges
  },
})

export const highlighter = createHighlighter({
  languages: [scriptDocument, js],
})

The outer definition remains independent of JavaScript; the application decides whether delegation is available.

Prefer unclassified text over incorrect classes

import { defineLanguage, type TokenRange } from '@tanstack/highlight/core'

export const identifiers = defineLanguage({
  name: 'identifiers',
  tokenize(code) {
    const ranges: Array<TokenRange> = []
    const pattern = /\b[A-Z][A-Za-z0-9]*\b/g
    let match: RegExpExecArray | null

    while ((match = pattern.exec(code))) {
      ranges.push({
        start: match.index,
        end: match.index + match[0].length,
        className: 'type',
      })
    }

    return ranges
  },
})

Do not classify ambiguous gaps merely to increase token coverage.

Common Mistakes

CRITICAL Returning inclusive range ends

Wrong:

import type { TokenRange } from '@tanstack/highlight/core'

export function keywordRange(start: number, value: string): TokenRange {
  return {
    start,
    end: start + value.length - 1,
    className: 'keyword',
  }
}

Correct:

import type { TokenRange } from '@tanstack/highlight/core'

export function keywordRange(start: number, value: string): TokenRange {
  return {
    start,
    end: start + value.length,
    className: 'keyword',
  }
}

Range ends match String.prototype.slice() and exclude the end offset.

Source: docs/guides/custom-languages.md

HIGH Returning overlapping token ranges

Wrong:

import type { TokenRange } from '@tanstack/highlight/core'

export function tokenize(code: string): Array<TokenRange> {
  return [
    { start: 0, end: code.length, className: 'string' },
    { start: 6, end: 10, className: 'keyword' },
  ]
}

Correct:

import type { TokenRange } from '@tanstack/highlight/core'

export function tokenize(code: string): Array<TokenRange> {
  const keywordStart = code.indexOf('TODO')

  return keywordStart < 0
    ? []
    : [
        {
          start: keywordStart,
          end: keywordStart + 'TODO'.length,
          className: 'keyword',
        },
      ]
}

The core sorts by source position and drops later overlaps, so an earlier broad range can suppress a specific range.

Source: src/core.ts

CRITICAL Importing an embedded definition into a language

Wrong:

import { defineLanguage } from '@tanstack/highlight/core'
import { js } from '@tanstack/highlight/languages/js'

export const scriptDocument = defineLanguage({
  name: 'script-document',
  tokenize(code, context) {
    return js.tokenize(code, context)
  },
})

Correct:

import { defineLanguage } from '@tanstack/highlight/core'

export const scriptDocument = defineLanguage({
  name: 'script-document',
  tokenize(code, context) {
    return context.hasLanguage('js') ? context.tokenize(code, 'js') : []
  },
})

Direct definition imports make an embedded dependency part of every outer-language bundle.

Source: docs/guides/custom-languages.md

HIGH Inventing language-specific token classes

Wrong:

import type { TokenRange } from '@tanstack/highlight/core'

export const range = {
  start: 0,
  end: 4,
  className: 'tsx-generic',
} as unknown as TokenRange

Correct:

import type { TokenRange } from '@tanstack/highlight/core'

export const range = {
  start: 0,
  end: 4,
  className: 'type',
} satisfies TokenRange

The fixed semantic union keeps every theme complete and language-independent.

Source: src/core.ts

CRITICAL Shipping without tokenizer quality gates

Wrong:

import { defineLanguage } from '@tanstack/highlight/core'

export const taskList = defineLanguage({
  name: 'task-list',
  tokenize() {
    return []
  },
})

Correct:

import { expect, test } from 'vitest'
import { createHighlighter } from '@tanstack/highlight/core'
import { taskList } from '../src/languages/task-list'

test('task-list reconstructs valid source', () => {
  const highlighter = createHighlighter({ languages: [taskList] })
  const code = 'TODO write docs'
  const result = highlighter.tokenize(code, { lang: 'task-list' })

  expect(result.tokens.map((token) => token.value).join('')).toBe(code)
  expect(result.tokens.some((token) => token.className === 'keyword')).toBe(
    true,
  )
})

Shipped languages require valid-code fixtures, source reconstruction, focused regressions, and explicit-import size evaluation.

Source: docs/guides/custom-languages.md

HIGH Omitting embedded tokenizer registration

Wrong:

import { createHighlighter } from '@tanstack/highlight/core'
import { scriptDocument } from './script-document'

export const highlighter = createHighlighter({
  languages: [scriptDocument],
})

Correct:

import { createHighlighter } from '@tanstack/highlight/core'
import { js } from '@tanstack/highlight/languages/js'
import { scriptDocument } from './script-document'

export const highlighter = createHighlighter({
  languages: [scriptDocument, js],
})

Delegation remains plain when the target definition is absent from the same registry.

Source: docs/guides/embedded-languages.md

HIGH Tension: language isolation versus embedding depth

Use TokenizerContext for optional delegation and let the application register targets. Importing target definitions inside an outer tokenizer hides bundle cost.

See also: configure-selective-highlighting/SKILL.md - registration controls delegated language availability.

References

• Tokenizer contracts and repository implementation patterns

See also: configure-selective-highlighting/SKILL.md - aliases, fallbacks, and embedded targets resolve through the registry.