Iirc Eval

iirc-eval

IIRC: Incremental Implicitly-Refined Classification — Abdelsalam et al. (2020) (arXiv:2012.12477, 2020)

What this evaluates

Evaluates lifelong learning algorithms on incremental label refinement, requiring models to predict both coarse (superclass) and fine-grained (subclass) labels over time without forgetting prior knowledge, while operating under incomplete information constraints.

Datasets

• IIRC-CIFAR — total ?; splits: eval (-1)
• IIRC-ImageNet — total ?; splits: eval (-1)

Metrics

• pw-JS (primary) — range: [0, 1]
  • Per-window Jaccard Similarity. Computes the Jaccard index between the predicted set of labels and the ground truth set of labels for each sample, then averages across all samples and tasks. Requires predicting all correct coarse and fine-grained labels for a given sample.
• JS — range: [0, 1]
  • Standard Jaccard Similarity metric used for comparison, though pw-JS is the primary evaluation metric for the incremental setup.

Input / output format

Input: Image input (CIFAR-100 or ImageNet images)

Output: Set of predicted labels per sample, including both coarse (superclass) and fine-grained (subclass) labels.

Scoring recipe

def compute_pw_js(predictions, gold):
    jaccard_scores = []
    for pred_set, gold_set in zip(predictions, gold):
        intersection = len(pred_set & gold_set)
        union = len(pred_set | gold_set)
        jaccard_scores.append(intersection / union if union > 0 else 0.0)
    return sum(jaccard_scores) / len(jaccard_scores)

Common pitfalls

• Models tend to over-predict fine-grained labels, which heavily penalizes the pw-JS score due to the union term in the Jaccard calculation.
• Conflicting supervision signals arise when the same subclass is presented with a superclass label in the replay buffer and a subclass label in the current task, causing models to fail to connect the two pieces of information.
• Memory constraints alone are insufficient; models must also handle incomplete information where not all label hierarchies are visible at once.

Evidence (verbatim from paper)

Specifically, as the model finishes training on the $j^{th}$ task, we report the average performance $R_{j}$, as measured by the pw-JS metric using Equation[2], over the evaluation set of all the tasks the model has seen so far. Recall that when computing $R_{j}$, the model has to predict all the correct labels for a given sample, even if the labels were seen across different tasks.

Citation

@misc{abdelsalam2020iirc,
  title={IIRC: Incremental Implicitly-Refined Classification},
  author={Abdelsalam et al. (2020)},
  year={2020},
  note={arXiv:2012.12477}
}

• arXiv: 2012.12477