Kg Benchmark Eval

kg-benchmark-eval

Scaling Knowledge Graphs for Automating AI of Digital Twins — Ploennigs et al. (2022) (arXiv:2210.14596, 2022)

What this evaluates

Evaluates the query execution performance and scalability of various knowledge graph systems across e-commerce, academic, and digital twin domains. It measures how efficiently triple stores and property graphs handle subsumption, recursive queries, and large-scale RDF datasets under realistic workload conditions.

Datasets

• BSBM — total ?; splits: S2 (377241), S5 (1711567), S10 (3738188), S20 (7749994), S50 (17571059), S100 (35159904)
• LUBM — total ?; splits: S2 (288894), S5 (781694), S10 (1591694), S20 (3360686), S50 (8317905), S100 (16753468)
• DTBM — total ?; splits: S2 (114177), S5 (283785), S10 (570232), S20 (1136634), S50 (2844499), S100 (5693601)

Metrics

• average response time (seconds) (primary) — range: other
  • Mean execution time in seconds across multiple query runs. For BSBM, measured via the default test driver excluding deserialization time; for LUBM and DTBM, measured via a custom Python executor over 500 sequential runs including deserialization, with results capped at 1,000 records.

Input / output format

Input: RDF datasets generated by BSBM, LUBM, or DTBM benchmarks, loaded into graph database systems. Queries (SPARQL, Cypher, or YAML) parameterized with random values from the dataset.

Output: Average query response time in seconds.

Scoring recipe

def compute_avg_response_time(queries, system, runs=500, limit=1000):
    total_time = 0.0
    for q in queries:
        for _ in range(runs):
            start = time.time()
            system.execute(q, limit=limit)
            total_time += time.time() - start
    return total_time / (len(queries) * runs)

Common pitfalls

• BSBM uses 16 parallel queries while LUBM/DTBM use sequential execution, making direct cross-benchmark time comparisons invalid without normalization.
• BSBM excludes deserialization time from measurements, whereas LUBM/DTBM include it, conflating execution and serialization costs.
• Neo4j requires manually written Cypher queries to emulate RDFS subsumption, which is not scalable in production and artificially boosts its performance relative to native RDF stores.

Evidence (verbatim from paper)

In LUBM and DTBM, we use our own query executor in Python to evaluate the performance of the various systems. We execute each query 500 times and we report the average response time including the deserialization of the response. We also place a constraint to the result size limiting the result to 1,000 records to put the focus on the query execution time and not the serialization time.

Citation

@misc{ploennigs2022scaling,
  title={Scaling Knowledge Graphs for Automating AI of Digital Twins},
  author={Ploennigs et al. (2022)},
  year={2022},
  note={arXiv:2210.14596}
}

• arXiv: 2210.14596