Identity Fraud Detection Eval

identity-fraud-detection-eval

Are You for Real? Detecting Identity Fraud via Dialogue Interactions — Wang et al. (2019) (arXiv:1908.06820, 2019)

What this evaluates

This evaluation probes a dialogue system's ability to dynamically generate derived questions and manage multi-turn interactions to accurately classify loan applicants as fraudulent or legitimate based on their knowledge of personal information triplets.

Datasets

• Applicant Personal Information Dataset — total 906; splits: train (706), dev (100), test (100)

Metrics

• recognition accuracy (primary) — range: [0, 1]
  • The proportion of correctly classified applicants (Fraud vs. Non-Fraud) out of the total test set.
• average turns — range: other
  • The mean number of dialogue turns required by the system to reach a final decision per applicant.

Input / output format

Input: Applicant's personal information represented as knowledge graph triplets, along with the current dialogue history and state.

Output: A sequence of derived questions generated during the dialogue, culminating in a binary final decision: 'Fraud' or 'Non-Fraud'.

Scoring recipe

def compute_metrics(predictions, gold_labels, turns_list):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
    accuracy = correct / len(gold_labels)
    avg_turns = sum(turns_list) / len(turns_list)
    return {'recognition accuracy': accuracy, 'average turns': avg_turns}

Common pitfalls

• Evaluation relies on a heuristic user simulator calibrated via search engine frequency and human experiments, which may not capture real-world fraudster behavior.
• The system is evaluated on a small test set (100 instances), which may lead to high variance in accuracy and average turns.
• The 'average turns' metric is reported but the model is primarily optimized for accuracy, meaning higher accuracy may come at the cost of more turns.

Evidence (verbatim from paper)

We compare Full-S with baselines in terms of two metrics: recognition accuracy and average turns. We collect 906 applicants' personal information, and randomly select 706 for training, 100 for dev, and 100 for test.

Citation

@misc{wang2019areyouforreal,
  title={Are You for Real? Detecting Identity Fraud via Dialogue Interactions},
  author={Wang et al. (2019)},
  year={2019},
  note={arXiv:1908.06820}
}

• arXiv: 1908.06820