Stella Living Lab Eval

stella-living-lab-eval

Overview of LiLAS 2020 -- Living Labs for Academic Search — Philipp Schaer, Johann Schaible, Leyla Jael Garcia Castro (2023) (arXiv:2310.20387, 2023)

What this evaluates

Evaluates academic search and recommendation systems in live production environments using A/B testing and user interaction logs, bridging the gap between offline test collections and real-world performance.

Datasets

• LIVIVO — total 25000000; splits: test (-1)
• GESIS Search — total ?; splits: test (-1)

Metrics

• click-paths (primary) — range: count / percent
  • Implicit feedback metric tracking user click sequences on retrieved documents during live A/B testing. Used to infer relevance and system preference.

Input / output format

Input: User search queries routed through STELLA's API to Dockerized experimental retrieval systems.

Output: Ranked result sets from experimental vs. production systems, mixed via A/B testing or interleaving for live user interaction.

Scoring recipe

def evaluate_ab_test(experimental_results, production_results, user_interactions):
    # Route users to experimental or production system via A/B split
    # Log click-paths and interaction events per query
    # Aggregate click counts and dwell times
    # Compare usage levels between experimental and production
    return aggregate_interaction_stats

Common pitfalls

• Metrics rely on implicit user feedback (clicks) rather than explicit relevance judgments, introducing potential bias.
• Evaluation requires live production traffic and Docker-based deployment; cannot be replicated offline with static logs.
• Results are highly dependent on the specific domain (Life Sciences vs. Social Sciences) and user demographics of the host platforms.

Evidence (verbatim from paper)

While this is still work in progress, the GESIS Search data and possible relevance indicators, such as click-paths, can be obtained.

Citation

@misc{schaer2023overview,
  title={Overview of LiLAS 2020 -- Living Labs for Academic Search},
  author={Philipp Schaer, Johann Schaible, Leyla Jael Garcia Castro (2023)},
  year={2023},
  note={arXiv:2310.20387}
}

• arXiv: 2310.20387