System Loss

system-loss

To Save Mobile Crowdsourcing from Cheap-talk: A Game Theoretic Learning Approach — Hao et al. (2023) (arXiv:2306.06791, 2023)

What this evaluates

Evaluates the expected cost incurred by a crowdsourcing platform when inferring service states from biased, strategic user reviews under different baseline mechanisms.

Datasets

• Theoretical Crowdsourcing Model — total ?; splits: (unstated)

Metrics

• system loss (primary) — range: other
  • Expected cost given the platform's action and the true state distribution. For majority voting: L_1 = (μ_H - μ_L)^2 * (1/2^N) * (p_H * Σ_{k=0}^{⌈N/2⌉-1} C_N^k + (1-p_H) * Σ_{l=⌊N/2⌋+1}^{N} C_N^l + 1_{⌊N/2⌋=N/2} * p_H(1-p_H) * C_N^{N/2}). For blind abandoning: L_2 = p_H(1-p_H)(μ_H - μ_L)^2.

Input / output format

Input: Parameters: number of users N, prior probability of high-type users p_H, mean service states for high/low types μ_H and μ_L, and user messages.

Output: Platform's recommendation action a and the resulting expected system loss L.

Scoring recipe

def compute_system_loss(N, p_H, mu_H, mu_L, scheme='majority_voting'):
    if scheme == 'majority_voting':
        loss = (mu_H - mu_L)**2 * (1 / 2**N) * (
            p_H * sum(comb(N, k) for k in range(0, ceil(N/2))) +
            (1 - p_H) * sum(comb(N, l) for l in range(floor(N/2) + 1, N + 1)) +
            (1 if floor(N/2) == N/2 else 0) * p_H * (1 - p_H) * comb(N, N/2)
        )
    elif scheme == 'blind_abandoning':
        loss = p_H * (1 - p_H) * (mu_H - mu_L)**2
    return loss

Common pitfalls

• System loss scales quadratically with the difference between high and low type means (μ_H - μ_L), so it can be arbitrarily large.
• Benchmarks assume users blindly message their biased type rather than playing a strategic equilibrium, which may overstate their effectiveness.

Evidence (verbatim from paper)

The resultant system loss is ... which can be arbitrarily large as μ_H - μ_L → ∞.

Citation

@misc{hao2023save,
  title={To Save Mobile Crowdsourcing from Cheap-talk: A Game Theoretic Learning Approach},
  author={Hao et al. (2023)},
  year={2023},
  note={arXiv:2306.06791}
}

• arXiv: 2306.06791