Idnet Dataset Eval

idnet-dataset-eval

IDNet: A Novel Dataset for Identity Document Analysis and Fraud Detection — Guan et al. (2024) (arXiv:2408.01690, 2024)

What this evaluates

Evaluates the quality and utility of a large-scale synthetic identity document dataset for fraud detection. It measures metadata diversity, visual fidelity to real documents, stealthiness of forged modifications, and downstream model accuracy.

Datasets

• IDNet — total 837060; splits: train (-1), val (-1), test (-1)

Metrics

• SSIM (primary) — range: [0, 1]
  • Structural Similarity Index measuring perceptual image quality and structural similarity between two images. Calculated using luminance, contrast, and structure comparisons.
• Metadata Entropy — range: bits
  • Shannon entropy computed over document metadata fields to quantify diversity and information richness.
• Downstream Accuracy — range: percent
  • Classification accuracy of AI/ML models trained on the dataset for fraud detection tasks, compared against real-world baselines.

Input / output format

Input: Synthetic identity document images (authentic and six types of forged variants) paired with JSON metadata containing document identifiers, face IDs, personal details, and fraud parameters.

Output: Quality metrics (SSIM scores, entropy values) and downstream task performance metrics (fraud detection accuracy).

Scoring recipe

def compute_ssim(img1, img2):
    return structural_similarity(img1, img2, multichannel=True)

def compute_entropy(metadata_df):
    return entropy(metadata_df.apply(lambda x: x.value_counts(normalize=True).values))

def compute_accuracy(predictions, labels):
    return (predictions == labels).mean() * 100

Common pitfalls

• SSIM thresholds differ by comparison: >0.85 for synthetic vs real fidelity, >0.95 for authentic vs forged stealthiness.
• The dataset is entirely synthetic; evaluating it requires comparing against real-world benchmarks like MIDV-2020 rather than assuming real data splits.
• Utility is measured by downstream model accuracy, not dataset-level metrics, so model architecture and training protocols heavily influence results.

Evidence (verbatim from paper)

The specific aims include: Diversity. We aim for the dataset’s document metadata diversity (e.g., entropy) to meet or exceed that of other synthetic benchmarks like MIDV-2020. Fidelity. The visual similarity of IDNet’s document images to their real-world counterparts should be high, aiming for a Structural Similarity Index (SSIM) greater than 0.85. Stealthiness. To maintain the stealthiness of the fraudulent modifications, the structural similarity between documents with and without fraudulent modifications should be nearly indistinguishable, with an SSIM greater than 0.95. Utility. The dataset should enable AI/ML models to achieve accuracy levels comparable to those attained with real-world datasets.

Citation

@misc{guan2024idnet,
  title={IDNet: A Novel Dataset for Identity Document Analysis and Fraud Detection},
  author={Guan et al. (2024)},
  year={2024},
  note={arXiv:2408.01690}
}

• arXiv: 2408.01690