Consistencychecker Eval

consistencychecker-eval

ConsistencyChecker: Tree-based Evaluation of LLM Generalization Capabilities — Hong et al. (2025) (arXiv:2506.12376, 2025)

What this evaluates

Evaluates LLM generalization and functional consistency by measuring how well models preserve core functionality after iterative, reversible transformations. It probes cumulative error and path-specific divergence across multi-step transformation sequences without relying on static benchmarks.

Datasets

• ConsistencyChecker (Dynamic) — total ?; splits: (unstated); repo https://github.com/ulab-uiuc/consistencychecker

Metrics

• forest-level consistency score (C3(F)) (primary) — range: [0, 1]
  • Average of path-level consistency scores across all trees in a forest at depth n=3. Path consistency C(P) is the similarity (e.g., cosine similarity or BLEU) between the initial node output and the final node output after n reversible transformations. C3(F) = (1/M) * sum_{m=1 to M} C_3(T_m).

Input / output format

Input: A root node containing an initial text or code instance, a set of test inputs, and a sequence of reversible transformation operations (e.g., translate-then-translate-back, add-logging-then-remove-logging).

Output: The model generates transformed versions of the input at each tree depth. The evaluation computes execution outputs or semantic embeddings for each node to calculate similarity scores.

Scoring recipe

def compute_c3_forest(forest_trees, depth=3):
    forest_scores = []
    for tree in forest_trees:
        paths = get_all_paths(tree, depth)
        path_scores = []
        for path in paths:
            initial_out = exec_or_embed(path[0].content)
            final_out = exec_or_embed(path[-1].content)
            path_scores.append(similarity(initial_out, final_out))
        tree_score = mean(path_scores)
        forest_scores.append(tree_score)
    return mean(forest_scores)

Common pitfalls

• Assuming the evaluation relies on static external benchmarks; the framework is benchmark-free and dynamically generates transformation trees.
• Treating the similarity metric as fixed; the paper explicitly allows cosine similarity or BLEU, which produce different absolute scales.
• Varying the depth n arbitrarily; the final reported metric strictly uses n=3 to balance computational cost and error accumulation.

Evidence (verbatim from paper)

To balance these trade-offs, we select n=3 and use the forest-level consistency score C3(F) as the final metric in our ConsistencyChecker framework. C_n(F) := (1/M) \sum_{m=1}^{M} C_n(T_m), where C_n(T) denotes the consistency at path length n of tree T.

Citation

@misc{hong2025consistencychecker,
  title={ConsistencyChecker: Tree-based Evaluation of LLM Generalization Capabilities},
  author={Hong et al. (2025)},
  year={2025},
  note={arXiv:2506.12376}
}

• arXiv: 2506.12376