# Stella Living Lab Eval

> Evaluates academic search and recommendation systems in live production environments using A/B testing and user interaction logs, bridging the gap between offline test collections and real-world performance. Use when the user wants to benchmark on LIVIVO, GESIS Search, or asks about evaluating this task. Reports click-paths.

- Skill: `qhjqhj00/stella-living-lab-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/stella-living-lab-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/stella-living-lab-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/stella-living-lab-eval

---


# stella-living-lab-eval

> Overview of LiLAS 2020 -- Living Labs for Academic Search — Philipp Schaer, Johann Schaible, Leyla Jael Garcia Castro (2023) (arXiv:2310.20387, 2023)

## What this evaluates

Evaluates academic search and recommendation systems in live production environments using A/B testing and user interaction logs, bridging the gap between offline test collections and real-world performance.

## Datasets

- **LIVIVO** — total 25000000; splits: test (-1)
- **GESIS Search** — total ?; splits: test (-1)

## Metrics

- `click-paths` **(primary)** — range: count / percent
  - Implicit feedback metric tracking user click sequences on retrieved documents during live A/B testing. Used to infer relevance and system preference.

## Input / output format

**Input**: User search queries routed through STELLA's API to Dockerized experimental retrieval systems.

**Output**: Ranked result sets from experimental vs. production systems, mixed via A/B testing or interleaving for live user interaction.

## Scoring recipe

```python
def evaluate_ab_test(experimental_results, production_results, user_interactions):
    # Route users to experimental or production system via A/B split
    # Log click-paths and interaction events per query
    # Aggregate click counts and dwell times
    # Compare usage levels between experimental and production
    return aggregate_interaction_stats
```

## Common pitfalls

- Metrics rely on implicit user feedback (clicks) rather than explicit relevance judgments, introducing potential bias.
- Evaluation requires live production traffic and Docker-based deployment; cannot be replicated offline with static logs.
- Results are highly dependent on the specific domain (Life Sciences vs. Social Sciences) and user demographics of the host platforms.

## Evidence (verbatim from paper)

> While this is still work in progress, the GESIS Search data and possible relevance indicators, such as click-paths, can be obtained.

## Citation

```bibtex
@misc{schaer2023overview,
  title={Overview of LiLAS 2020 -- Living Labs for Academic Search},
  author={Philipp Schaer, Johann Schaible, Leyla Jael Garcia Castro (2023)},
  year={2023},
  note={arXiv:2310.20387}
}
```

- arXiv: 2310.20387