Gnosis LLM Self Awareness

Overview

Gnosis is a lightweight self-awareness mechanism enabling frozen LLMs to perform intrinsic self-verification by inspecting internal computation states. It addresses a critical problem: LLMs generate confident but incorrect outputs, and detecting failures typically requires expensive external judges or multiple inference passes.

Core Innovation: Rather than relying on external verification, extract correctness prediction signals directly from the LLM's internal states—the hidden representations and attention patterns that encode the model's confidence and reasoning process.

Architecture

State Extraction Layer

Capture internal states at each layer of the transformer:

Hidden States:

• Extract final token's hidden representations from each transformer layer
• Retain information about accumulated computation and reasoning

Attention Patterns:

• Track attention weights across heads and layers
• Reveal what model attends to when reasoning
• Indicate confidence through attention concentration

Multi-Layer Analysis:

• Early layers encode syntactic/surface information
• Middle layers perform semantic reasoning
• Later layers prepare for output generation
• Different layers provide complementary correctness signals

State Compression

Compress extracted states into fixed-budget descriptors:

def compress_internal_states(hidden_states, attention_patterns):
    """Compress multi-layer states to fixed-size descriptor."""

    # Extract key statistics from each layer
    descriptors = []
    for layer_idx, (hidden, attn) in enumerate(
        zip(hidden_states, attention_patterns)
    ):
        # Hidden state statistics
        mean_activation = hidden.mean(dim=-1)
        activation_variance = hidden.var(dim=-1)

        # Attention statistics
        attention_entropy = calculate_entropy(attn)
        attention_concentration = attn.max()

        # Temporal statistics (change across tokens)
        activation_change = (hidden[:, -1] - hidden[:, -2]).norm()

        layer_descriptor = {
            "layer": layer_idx,
            "mean_activation": mean_activation,
            "activation_variance": activation_variance,
            "attention_entropy": attention_entropy,
            "attention_concentration": attention_concentration,
            "activation_change": activation_change,
        }
        descriptors.append(layer_descriptor)

    return descriptors

Correctness Prediction Head

Lightweight neural network predicting correctness from compressed states:

class CorrectnessPredictor(nn.Module):
    def __init__(self, hidden_size=256):
        super().__init__()
        # Small linear layers for prediction
        self.layers = nn.Sequential(
            nn.Linear(descriptor_size, hidden_size),
            nn.ReLU(),
            nn.Linear(hidden_size, hidden_size // 2),
            nn.ReLU(),
            nn.Linear(hidden_size // 2, 1),
            nn.Sigmoid(),  # Output: correctness probability [0, 1]
        )

    def forward(self, descriptors):
        """Predict correctness probability."""
        return self.layers(descriptors)

Parameter Count: ~5M parameters (negligible vs. LLM size)

Performance Across Domains

Mathematical Reasoning (MATH, AMC datasets):

• Accurately predicts correctness on challenging math problems
• Outperforms external judges in both accuracy and calibration
• Works across problem types (algebra, geometry, combinatorics)

Open-Domain Question Answering (Natural Questions, TriviaQA):

• Detects hallucinations and factual errors
• Better calibration than text-based confidence measures
• Generalizes to diverse question types

Academic Knowledge (CollegeBench, STEM Q&A):

• Identifies knowledge gaps and uncertain reasoning
• Useful for educational applications
• Robust across knowledge domains

Backbone Model Range: 1.7B to 20B parameters

• Consistent effectiveness across model scales
• No need for model-specific training

Key Advantages

vs. External Judges:

• No additional inference passes needed
• Negligible computational overhead
• Works with API-only models (observe states, pass to Gnosis)
• Better calibration on most benchmarks

vs. Text-Based Self-Critique:

• Observes actual computation, not post-hoc reasoning
• Lower token cost (no generating explanations)
• More reliable on tasks where reasoning is implicit

vs. Ensemble Methods:

• Single inference pass (just access internal states)
• Consistent performance across ensemble sizes
• Deterministic verification (not sampling-based)

Zero-Shot Generalization

A powerful capability: Gnosis trained on one task type generalizes zero-shot to partial generations:

Partial Generation Detection:

• Predict correctness before full response is generated
• Enable early stopping when model is failing
• Reduce wasted computation on doomed attempts

Implementation:

def early_stopping_inference(prompt, model, gnosis, threshold=0.3):
    """Generate tokens with early stopping on low correctness signal."""

    generated = []
    for step in range(max_steps):
        # Generate next token
        next_token, hidden_states = model.forward_with_states(
            generated, return_internal_states=True
        )
        generated.append(next_token)

        # Predict correctness after K tokens
        if len(generated) > min_generation_length:
            descriptor = compress_states(hidden_states)
            correctness = gnosis(descriptor)

            if correctness < threshold:
                # Restart with different strategy
                return "RESTART"

    return "".join(generated)

Benefit: Compute-aware control—allocate more computation to promising solution attempts.

When to Use Gnosis

Use when:

• Building reliable LLM applications requiring confidence estimates
• Verifying LLM outputs without external services
• Need to detect hallucinations and failures in closed-book settings
• Optimizing inference compute through early stopping
• Fine-tuning models and needing in-domain correctness prediction

When NOT to use:

• Tasks with very simple, deterministic outputs (direct evaluation sufficient)
• Scenarios where external judge cost is acceptable
• Applications where inference overhead is critical (5-10% latency increase)
• Domains with extremely different task distributions than training

Implementation Considerations

Training Data Requirements:

• Pairs of (generation, correctness) labels
• Domain-specific training for best accuracy
• Data from same task distribution as deployment
• Binary correctness labels sufficient (yes/no, correct/incorrect)

Inference Setup:

• Requires access to model's internal states
• Straightforward with HuggingFace models (register hooks)
• Works with frozen model weights (no fine-tuning needed)
• ~5-10% inference latency overhead

Calibration:

• Probability outputs well-calibrated across domains
• Can adjust threshold for precision/recall tradeoff
• Temperature scaling if additional calibration needed

Research Contributions

• Internal State Interpretation: Correctness signals are intrinsic to generation
• Efficient Detection: Lightweight mechanism (5M parameters) for self-awareness
• Zero-Shot Generalization: Transfer to partial generations and new domains
• Comprehensive Evaluation: Testing across 7+ benchmarks and 4 model scales

Related Work

External Verification:

• Self-Critique, Reflexion (external judges)
• Multi-sample consistency (expensive inference)

Internal Inspection:

• Activation analysis and circuit discovery
• Probe-based interpretation (Gnosis is lightweight alternative)

Code Availability

Full implementation and models available at repository.

Evaluation Benchmarks:

• MATH, AMC (mathematical reasoning)
• Natural Questions, TriviaQA (open-domain QA)
• CollegeBench, STEM datasets (academic knowledge)

References

• Gnosis achieves consistent improvements across 7+ benchmarks
• Works with frozen LLMs (1.7B-20B parameters)
• 5M-parameter overhead with reliable correctness prediction
• Superior calibration vs. external judges and self-critique methods