Servimage Eval

servimage-eval

ServImage: An Image Generation and Editing Benchmark from Real-world Commercial Imaging Services — Ji et al. (2026) (arXiv:2604.24023, 2026)

What this evaluates

Evaluates the commercial viability and economic performance of text-to-image and image editing models in real-world design workflows. It measures how well generated images meet baseline requirements, visual quality standards, and commercial intent, linking outputs directly to human payment decisions and platform revenue.

Datasets

• ServImage — total ?; splits: test (-1); repo https://github.com/FengxianJi/ServImage

Metrics

• Task Acceptance (%) (primary) — range: percent
  • Proportion of tasks where the model's output is approved according to human payment decisions. Calculated as the number of accepted tasks divided by the total number of tasks.
• ServImageScore (BRF/VEQ/CNS) — range: [0, 1]
  • Composite score evaluating Baseline Requirements Fulfilment (BRF), Visual Execution Quality (VEQ), and Commercial Necessity Satisfaction (CNS). Computed automatically using OpenAI’s GPT-5-mini as a judge.
• AUC / Spearman / PR-AUC — range: [0, 1]
  • Alignment metrics between automatic metric scores and binary human payment labels. AUC is the area under the ROC curve, Spearman is the rank correlation, and PR-AUC is the area under the precision–recall curve.

Input / output format

Input: Text prompts and/or source images for commercial design tasks across Portrait, Product, and Digital categories.

Output: Generated images (text-to-image or image-to-image edits).

Scoring recipe

def evaluate_model(model, test_set, human_labels):
    task_accepted = 0
    for task in test_set:
        output = model.generate(task.prompt, task.source_image)
        if human_labels[task.id][output] == 1:
            task_accepted += 1
    task_acc = task_accepted / len(test_set)
    
    # Automatic metric alignment
    for metric in [BRF, VEQ, CNS]:
        scores = [metric(model.generate(t.prompt, t.source_image)) for t in test_set]
        auc = compute_auc(scores, human_labels)
        spearman = compute_spearman(scores, human_labels)
        pr_auc = compute_pr_auc(scores, human_labels)
    return task_acc, auc, spearman, pr_auc

Common pitfalls

• Confusing standard perceptual metrics (e.g., FID, CLIP) with commercial acceptance; this benchmark explicitly prioritizes human payment decisions over technical quality scores.
• Assuming revenue/share metrics are intrinsic to the model; they are highly dependent on the specific settlement rules, task pricing, and competitive regime (standard vs. winner-takes-all) defined by the platform.
• Overlooking category-specific performance; models may dominate one commercial domain (e.g., Product) while failing in another (e.g., Portrait), masking overall commercial viability.

Evidence (verbatim from paper)

Table 2 uses human accept/reject labels as ground truth; consequently, Table 9 reports an automatic proxy estimated from ServImageModel-predicted payment probabilities. We examine whether automatic metrics can approximate human payment decisions. We apply each metric to the same subset and evaluate its alignment with human payment labels using three criteria: (i) AUC, the area under the ROC curve for binary payment outcomes; (ii) Spearman, the rank correlation between metric scores and payment labels; (iii) PR-AUC, the area under the precision–recall curve for payment prediction.

Citation

@misc{ji2026servimage,
  title={ServImage: An Image Generation and Editing Benchmark from Real-world Commercial Imaging Services},
  author={Ji et al. (2026)},
  year={2026},
  note={arXiv:2604.24023}
}

• arXiv: 2604.24023