Trace Encoding Benchmark Eval

trace-encoding-benchmark-eval

Trace Encoding in Process Mining: a survey and benchmarking — Barbon Jr. et al. (2023) (arXiv:2301.02167, 2023)

What this evaluates

Evaluates the quality and efficiency of trace encoding methods for process mining event logs. It probes how well encodings preserve trace similarities (expressivity), their computational cost as data scales (scalability), and their suitability for downstream process mining tasks.

Datasets

• Process Mining Event Log Scenarios (1-5) — total ?; splits: test (-1); repo https://github.com/gbrltv/business_process_encoding

Metrics

• T4 (primary) — range: [0, 1]
  • Proportion of relevant dimensions used by the encoding vector to map the event log, ranging from 0 to 1. Lower values indicate better expressivity (less redundancy/complexity needed).
• Encoding Time (s) — range: other
  • Total seconds consumed during the encoding process across the entire event log.
• Encoding Memory (KB) — range: other
  • Total kilobytes of memory consumed during the encoding process.

Input / output format

Input: Raw event logs (sequences of activities/events) representing business process traces, tested at sizes of 1k, 5k, and 10k traces.

Output: Numerical feature vectors representing the encoded traces (dimensionality varies by method).

Scoring recipe

def evaluate_encoding(event_log, encoding_method):
    # 1. Compute encoding vectors
    vectors = encoding_method.encode(event_log)
    
    # 2. Calculate T4 (Expressivity)
    # Proportion of relevant dimensions via PCA criterion (0 to 1)
    t4 = compute_proportion_relevant_dimensions(vectors)
    
    # 3. Measure Scalability
    time_cost = measure_execution_time_seconds(encoding_method.encode, event_log)
    memory_cost = measure_peak_memory_kb(encoding_method.encode, event_log)
    
    # 4. Qualitative PCA Expressivity (optional/visual)
    pca_2d = PCA(vectors, n_components=2)
    expressivity_level = assess_cluster_separation(pca_2d, scenario_labels)
    
    return {'T4': t4, 'Time(s)': time_cost, 'Memory(KB)': memory_cost}

Common pitfalls

• Misinterpreting T4 values: lower T4 indicates better expressivity, not higher.
• Evaluating scalability on a single dataset size instead of testing across multiple scales (1k, 5k, 10k traces).
• Relying solely on visual PCA inspection without quantitative validation like T4.

Evidence (verbatim from paper)

T4 gives a rough measure, from 0 to 1, of the proportion of relevant dimensions used by the encoding vector to map the event log. A higher T4 value indicates a more complex relationship between the input variables, indicating a larger number of original features are required to describe the data variability. ... In our experiments, we considered the time (seconds) and memory (KB) consumption accumulated during the whole encoding.

Citation

@misc{barbon2023traceencoding,
  title={Trace Encoding in Process Mining: a survey and benchmarking},
  author={Barbon Jr. et al. (2023)},
  year={2023},
  note={arXiv:2301.02167}
}

• arXiv: 2301.02167