Social Chem 101 Eval

social-chem-101-eval

Social Chemistry 101: Learning to Reason about Social and Moral Norms — Forbes et al. (2020) (arXiv:2011.00620, 2020)

What this evaluates

Evaluates a model's ability to reason about and generate rules-of-thumb (RoTs) that capture social and moral norms across 12 distinct dimensions of judgment, such as cultural pressure, legality, and moral foundations. It probes whether neural models can produce attribute-aware, context-sensitive normative judgments for unseen social scenarios.

Datasets

• Social-Chem-101 — total 292000; splits: train (-1), val (-1), test (-1)

Metrics

• micro-F1 (primary) — range: [0, 1]
  • Micro-averaged F1 score computed across all 12 annotation dimensions. True positives, false positives, and false negatives are aggregated globally across all instances and dimensions before calculating precision and recall.

Input / output format

Input: A described social situation or scenario.

Output: A set of rules-of-thumb (RoTs) specifying the behavior/action and its acceptability judgment, or a classification label across the 12 social/moral dimensions.

Scoring recipe

def compute_micro_f1(predictions, golds):
    tp = fp = fn = 0
    for pred, gold in zip(predictions, golds):
        for dim in all_dimensions:
            p = pred.get(dim)
            g = gold.get(dim)
            if p == g == 1: tp += 1
            elif p == 1 and g == 0: fp += 1
            elif p == 0 and g == 1: fn += 1
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Performance varies drastically across dimensions (micro-F1 ranges from 0.28 to 0.91), indicating models struggle with nuanced moral/cultural sensitivities.
• The dataset is culturally bounded to English-speaking North American norms, limiting generalizability to other cultures.
• Multiple RoTs can apply to a single situation, requiring models to handle variable-length outputs or multi-label classification rather than single-label prediction.

Evidence (verbatim from paper)

The framework demonstrates that neural models can generate attribute-aware, context-sensitive RoTs, though performance varies across dimensions (micro-F1: 0.28–0.91), highlighting gaps in current models' ability to capture nuanced cultural and moral sensitivities.

Citation

@misc{forbes2020socialchemistry101,
  title={Social Chemistry 101: Learning to Reason about Social and Moral Norms},
  author={Forbes et al. (2020)},
  year={2020},
  note={arXiv:2011.00620}
}

• arXiv: 2011.00620