# Kuaisim Eval

> kuaisim-eval

- Skill: `qhjqhj00/kuaisim-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/kuaisim-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/kuaisim-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/kuaisim-eval

---


# kuaisim-eval

> KuaiSim: A Comprehensive Simulator for Recommender Systems — Zhao et al. (2023) (arXiv:2309.12645, 2023)

## What this evaluates

Evaluates recommender system policies across three temporal granularities: request-level list-wise ranking, whole-session sequential recommendation under reinforcement learning, and cross-session user retention optimization. It measures how well simulated agents balance immediate engagement rewards with long-term user retention and list diversity.

## Datasets

- **KuaiRand** — total ?; splits: train (-1), test (-1)
- **ML-1m** — total ?; splits: train (-1), test (-1)

## Metrics

- `Average L-reward` **(primary)** — range: other
  - Average of item-wise immediate rewards across user requests in a mini-batch.
- `Coverage` — range: other
  - Number of distinct items exposed in a mini-batch.
- `ILD (Intra-list diversity)` — range: other
  - Estimates dissimilarity between items in each recommended list based on item embeddings.
- `Whole-session reward` — range: other
  - Average sum of immediate rewards for each session.
- `Return day` — range: other
  - Average time gap (in days) between the last request of one session and the first request of the next.
- `User retention` — range: percent
  - Average ratio of users visiting the system again after a session.

## Input / output format

**Input**: User interaction history or current session context, along with a candidate item pool.

**Output**: A ranked list of recommended items or a sequence of items for the session.

## Scoring recipe

```python
def evaluate(predictions, gold, embeddings, user_history):
    l_rewards = [sum(item_reward(i) for i in pred) for pred in predictions]
    avg_l = sum(l_rewards) / len(l_rewards)
    max_l = max(l_rewards)
    coverage = len(set(i for pred in predictions for i in pred))
    ild = diversity_score(predictions, embeddings)
    session_rewards = [sum(r for r in sess) for sess in sessions]
    whole_sess = sum(session_rewards) / len(session_rewards)
    depth = max(len(sess) for sess in sessions)
    return_day = avg_gap_days(user_history)
    retention = returning_users / total_users
    return avg_l, max_l, coverage, ild, whole_sess, depth, return_day, retention
```

## Common pitfalls

- Metrics are defined at different granularities (request-level vs session-level vs cross-session), making direct comparison across tasks difficult.
- Reward signals are generated by pretrained simulation models rather than real user clicks, which may introduce distributional shift.
- ILD relies on item embeddings that may not align with human-perceived diversity or business objectives.

## Evidence (verbatim from paper)

> List-wise reward (L-reward) is the average of item-wise immediate reward. We use both the average L-reward and the max L-reward across user requests in a mini-batch. Coverage describes the number of distinct items exposed in a mini-batch. Intra-list diversity (ILD) estimates the dissimilarity between items in each recommended list, based on item embedding.

## Citation

```bibtex
@misc{zhao2023kuaisim,
  title={KuaiSim: A Comprehensive Simulator for Recommender Systems},
  author={Zhao et al. (2023)},
  year={2023},
  note={arXiv:2309.12645}
}
```

- arXiv: 2309.12645