Iccma17 Eval

iccma17-eval

Design and Results of the Second International Competition on Computational Models of Argumentation — Gaggl et al. (2019) (arXiv:1909.00621, 2019)

What this evaluates

Evaluates computational argumentation solvers on their ability to correctly compute extensions (e.g., semi-stable, stage, ideal) across diverse argumentation frameworks ranging from random graphs to application-derived structures.

Datasets

• ICCMA'17 Benchmark Suite — total 3990; splits: test (3990)

Metrics

• exact-match accuracy (primary) — range: [0, 1]
  • 1 if the solver's predicted extension exactly matches the ground-truth extension for the given semantics, 0 otherwise. Averaged across all instances in the benchmark suite.

Input / output format

Input: Argumentation Framework (AF) represented as a set of arguments and a binary attack relation between them.

Output: Computed extension (set of arguments) or complete labeling according to a specified semantics.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if set(p) == set(g))
    return correct / len(gold)

Common pitfalls

• Different semantics (semi-stable, stage, ideal) have distinct extension definitions; evaluating across them requires separate ground-truth labels.
• Instance generation parameters (e.g., n, probAttacks) vary widely and must be fixed to ensure reproducibility across runs.
• Runtime/efficiency is often tracked in competitions but is distinct from correctness accuracy.

Evidence (verbatim from paper)

The second International Competition on Computational Models of Argumentation (ICCMA'17) introduces three new semantics—semi-stable, stage, and ideal—and implements a novel scoring scheme, instance selection process, and a 'Dung's Triathlon' track that evaluates solvers across multiple complexity layers. Thus, the benchmark suite of ICCMA'17 is finally composed of 3990 instances over 11 domains.

Citation

@misc{gaggl2019iccma17,
  title={Design and Results of the Second International Competition on Computational Models of Argumentation},
  author={Gaggl et al. (2019)},
  year={2019},
  note={arXiv:1909.00621}
}

• arXiv: 1909.00621