Token Budget

Token Budget Skill

Purpose

Help plan, implement, and monitor token budgets in LLM-based applications. Token budgets come in three flavors — often used in combination:

1. Context Budget (Quality) — How much fits into the context window?
2. Cost Budget (Cost) — How much are we spending on tokens?
3. Quota Budget (Rate) — How fast are we consuming tokens?

Each flavor addresses a different concern but they share infrastructure (token counting, tracking, configuration). This skill guides the user through an interview to understand their needs, then generates the appropriate Quarkus/LangChain4j code.

Interview

Before generating code, ask these questions to understand the scope. Skip questions that are already answered from context.

Questions

1. Which use case? Context / Cost / Quota / Combination?

   • Local LLM (Ollama, LM Studio) → usually only Context is relevant
   • Cloud API (OpenAI, Anthropic) → all three may be relevant

2. Which LLM provider? Ollama / LM Studio / OpenAI / Anthropic / Azure OpenAI / other?

   • This determines whether cost tracking makes sense and which token counting approach to use

3. How many services/agents make LLM calls?

   • Single service → simpler tracking
   • Multiple services → need per-service budgets and fair scheduling

4. Is there a monthly monetary budget?

   • If yes → Cost Budget with alerting thresholds
   • If no → may still want usage visibility

5. Which framework? Quarkus + LangChain4j (primary) / Spring + LangChain4j / other?

6. Should token usage be persisted?

   • In-memory → fine for dev, resets on restart
   • Database → production-grade, historical data, dashboards

After the interview, determine which budget types to generate and read the corresponding reference file(s) from references/ for implementation details.

Implementation Strategy

Central Hook: ChatModelListener

All three budget types share a common entry point — the ChatModelListener from LangChain4j. This CDI bean intercepts every LLM call and provides access to input/output token counts.

@ApplicationScoped
public class TokenTrackingListener implements ChatModelListener {

    @Override
    public void onResponse(ChatModelResponseContext context) {
        var usage = context.response().tokenUsage();
        int inputTokens = usage.inputTokenCount();
        int outputTokens = usage.outputTokenCount();
        // Route to the appropriate budget handler(s)
    }
}

This listener is the foundation. Each budget type adds its own logic on top.

Token Estimation (for pre-call budget checks)

Before sending a request, you often need to estimate how many tokens the content will use. The skill generates a TokenEstimator utility:

• Default heuristic: 1 token ≈ 4 characters (English text), configurable
• Word-based: 1 token ≈ 0.75 words
• Provider-specific: Optional integration with tiktoken (OpenAI) or provider tokenizer APIs
• The estimator is used by Context Budget to decide what fits, and by Quota Budget for pre-flight checks

What Gets Generated

Based on the interview answers, read the appropriate reference file(s) and generate code:

Technical Constraints

• Thread-safe: Use AtomicLong / ConcurrentHashMap for in-memory tracking
• No vendor lock-in: Works with OpenAI, Ollama, LM Studio, Anthropic — the ChatModelListener abstracts the provider
• Metrics: Compatible with Micrometer/Prometheus via quarkus-micrometer-registry-prometheus
• Configuration: All budgets configurable via application.properties with sensible defaults
• Framework: Quarkus + LangChain4j as primary stack, but patterns are transferable

Output Conventions

Generated code follows these patterns:

• Package: {project.package}.token (or {module}.control if using BCE)
• CDI: @ApplicationScoped for all services
• Config: @ConfigProperty with prefix token-budget
• Logging: Log.infof() for budget consumption, Log.warnf() for threshold alerts
• Tests: @QuarkusTest with REST Assured for endpoint testing