Nl2gql Eval

nl2gql-eval

$R^3$-NL2GQL: A Model Coordination and Knowledge Graph Alignment Approach for NL2GQL — Zhou et al. (2023) (arXiv:2311.01862, 2023)

What this evaluates

Evaluates a model's capability to translate natural language queries into Graph Query Language (GQL) by measuring syntactic correctness, semantic comprehension, and execution fidelity against a knowledge graph schema.

Datasets

• NL2GQL dataset — total ?; splits: train (-1), test (-1)

Metrics

• Syntax Accuracy — range: percent
  • Percentage of generated queries that are syntactically valid GQL strings, regardless of execution results.
• Comprehension Accuracy — range: percent
  • Percentage of queries where the generated GQL correctly captures the semantic intent of the natural language prompt.
• Execution Accuracy (primary) — range: percent
  • Percentage of generated queries that produce exactly the same execution results as the ground truth query when run against the target knowledge graph.
• Intra Execution Accuracy — range: percent
  • Percentage of queries where individual sub-components or steps of the generated query execute correctly, even if the full query fails.

Input / output format

Input: Natural language query text, optionally accompanied by a few-shot example from the training set, and a serialized text representation of the target knowledge graph schema.

Output: Generated nGQL query string.

Scoring recipe

def compute_metrics(predictions, golds):
    sa, ca, ea, iea = 0, 0, 0, 0
    for pred, gold in zip(predictions, golds):
        sa += 1 if is_valid_gql_syntax(pred) else 0
        ca += 1 if matches_semantic_intent(pred, gold) else 0
        ea += 1 if execute_query(pred) == execute_query(gold) else 0
        iea += 1 if execute_subcomponents(pred) == execute_subcomponents(gold) else 0
    n = len(golds)
    return {
        'Syntax Accuracy': (sa / n) * 100,
        'Comprehension Accuracy': (ca / n) * 100,
        'Execution Accuracy': (ea / n) * 100,
        'Intra Execution Accuracy': (iea / n) * 100
    }

Common pitfalls

• Random selection of few-shot examples introduces sampling variability; the protocol requires repeating experiments multiple times (e.g., three) and averaging results to ensure stability.
• Syntax Accuracy (SA) only measures GQL grammatical validity and can be high even if the query fails to execute on the specific schema, so it should not be conflated with Execution Accuracy.
• Smaller foundation models exhibit very low generalization and syntax learning capabilities for GQL without coordination from larger models, making direct comparison with vanilla prompts misleading.

Evidence (verbatim from paper)

Table 2 showcases the comparative performance between our $R^3$-NL2GQL framework and leading GPT series models across Zero-Shot, One-Shot, and Few-Shot scenarios. ... Table 2: Comparison of the four metrics (%) among $R^3$-NL2GQL and the GPT family models. The bold numbers denote the best results and the underlined ones are the second-best performance. Model | Syntax Accuracy | Comprehension Accuracy | Execution Accuracy | Intra Execution Accuracy

Citation

@misc{zhou2023r3nl2gql,
  title={$R^3$-NL2GQL: A Model Coordination and Knowledge Graph Alignment Approach for NL2GQL},
  author={Zhou et al. (2023)},
  year={2023},
  note={arXiv:2311.01862}
}

• arXiv: 2311.01862