Cypherbench Eval

cypherbench-eval

CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era — Feng et al. (2024) (arXiv:2412.18702, 2024)

What this evaluates

Evaluates LLMs' ability to generate precise Cypher queries from natural language questions over large-scale property graphs. It probes complex graph retrieval capabilities including multi-hop reasoning, temporal constraints, aggregations, and strict schema adherence.

Datasets

• CypherBench — total 10000; splits: test (-1), train (-1); repo https://github.com/megagonlabs/cypherbench

Metrics

• EX (primary) — range: percent
  • Execution Accuracy: the percentage of predicted Cypher queries that execute successfully on Neo4j and return exactly the same result set as the ground-truth query.
• PSJS — range: percent
  • Provenance Subgraph Jaccard Similarity: the Jaccard index between the set of nodes and edges used in the predicted query versus the ground-truth query, measuring graph matching quality independent of the RETURN clause.
• Exec. — range: percent
  • Executable Percentage: the percentage of predicted queries that run without syntax or runtime errors within a 120-second timeout.

Input / output format

Input: Natural language question, the full graph schema, and a brief instruction prompt.

Output: A single Cypher query string.

Scoring recipe

def score(predictions, golds, neo4j_client):
    ex, psjs, exec_scores = [], [], []
    for pred, gold in zip(predictions, golds):
        try:
            pred_res = neo4j_client.execute(pred, timeout=120)
            is_exec = True
        except Exception:
            is_exec = False
            pred_res = None
        gold_res = neo4j_client.execute(gold, timeout=30)
        ex.append(1.0 if (is_exec and pred_res == gold_res) else 0.0)
        pred_sub = extract_provenance(pred)
        gold_sub = extract_provenance(gold)
        psjs.append(jaccard(pred_sub, gold_sub))
        exec_scores.append(1.0 if is_exec else 0.0)
    return {'EX': mean(ex), 'PSJS': mean(psjs), 'Exec.': mean(exec_scores)}

Common pitfalls

• Incorrect deduplication: Models often merge distinct entities that share the same name, causing false positives in result sets.
• RETURN clause mismatches: Including extra columns in the RETURN statement causes zero EX despite perfect graph matching (PSJS = 1.0).
• Schema violations: Predicted queries may use invalid relation directions or node/relationship types not present in the schema.

Evidence (verbatim from paper)

Finally, the predicted Cypher queries were executed on Neo4j using 8-thread parallelization with a 120-second timeout (4x the maximum execution time of the ground-truth Cypher) to compute the metrics. Table 3: Zero-shot execution accuracy (EX), provenance subgraph jaccard similarity (PSJS) and executable percentage (Exec.) on the CypherBench test set.

Citation

@misc{feng2024cypherbench,
  title={CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era},
  author={Feng et al. (2024)},
  year={2024},
  note={arXiv:2412.18702}
}

• arXiv: 2412.18702