Realtime Simulation Checking Eval

realtime-simulation-checking-eval

Simulation-Checking of Real-Time Systems with Fairness Assumptions — Wang (2010) (arXiv:1007.0523, 2010)

What this evaluates

Evaluates the scalability and performance of a simulation-checking algorithm for timed automata under fairness assumptions. It measures how efficiently the algorithm verifies liveness properties and handles state-space explosion across parameterized real-time system benchmarks.

Datasets

• Fischer's timed mutual exclusion algorithm — total ?; splits: test (-1)
• CSMA/CD — total ?; splits: test (-1)
• Timed consumer/producer — total ?; splits: test (-1)
• Network of TAs — total ?; splits: test (-1)

Metrics

• CPU time (primary) — range: seconds
  • Wall-clock execution time in seconds required to complete the simulation check.
• memory consumption — range: kilobytes
  • Total memory in kilobytes or megabytes consumed by the data-structures during state-space representation.

Input / output format

Input: Timed automata models and specifications parameterized by the number of processes (m) and environment topology (linear, binary-tree, irregular).

Output: Boolean simulation check result, along with recorded CPU time (seconds) and memory consumption (KB/MB).

Scoring recipe

def evaluate(model, spec, fairness):
    start = time.time()
    result = run_simulation_check(model, spec, fairness)
    end = time.time()
    cpu_time = end - start
    memory = get_data_structure_memory()
    return {'check_result': result, 'cpu_time': cpu_time, 'memory': memory}

Common pitfalls

• Benchmarks are parameterized by the number of processes (m), so performance scales non-linearly with system size.
• Memory reported is specifically for data-structures in state-space representations, not total system RAM.
• Values marked 'N/A' indicate the algorithm exceeded the 1800-second timeout or ran out of memory.

Evidence (verbatim from paper)

The CPU time used and the total memory consumption for the data-structures in state-space representations are reported. As can be seen, the performance of our new simulation against a common environment is significantly better than the classic one.

Citation

@misc{wang2010simulation,
  title={Simulation-Checking of Real-Time Systems with Fairness Assumptions},
  author={Wang (2010)},
  year={2010},
  note={arXiv:1007.0523}
}

• arXiv: 1007.0523