# Bis Eval

> Benchmarks energy-function-based safe control algorithms on the BIS (Benchmark of Interactive Safety) dataset, scoring safety, efficiency, and hybrid performance in human-robot and robot co-working scenarios.

- Skill: `qhjqhj00/bis-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/bis-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/bis-eval/raw
- Safety review: PASS (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML, Research & Search, Model Training & Fine-tuning
- Tags: Benchmark, Bis, Energy Functions, Evaluation, Human Robot Interaction, Metrics, Robot Co Working, Safe Control
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-08-22
- Page: https://skillmd.com/skills/qhjqhj00/bis-eval

---


# bis-eval

> Safe Control Algorithms Using Energy Functions: A Unified Framework, Benchmark, and New Directions — Wei et al. (2019) (arXiv:1908.01883, 2019)

## What this evaluates

Evaluates the trade-off between safety and efficiency of energy-function-based safe control algorithms in human-robot interaction and robot co-working scenarios. It tests how well different controllers navigate toward goals while avoiding collisions with human or robot agents across varying dynamic models.

## Datasets

- **BIS (Benchmark of Interactive Safety)** — total 40; splits: test (40); repo https://github.com/intelligent-control-lab/BIS

## Metrics

- `efficiency_score` **(primary)** — range: other
  - Average number of goals achieved by the robot within the 30-second simulation period.
- `safety_score` — range: other
  - safety = -∑_{0}^{T} min(0, log(d/d_s)) * ḋ, where d is distance to obstacle, d_s is a threshold, and ḋ is relative velocity. Penalizes rapid approach to obstacles even without collision.
- `hybrid_score` — range: other
  - Maximum efficiency (goals achieved) achieved by the algorithm under the constraint of zero collisions. Used specifically for robot-robot collaboration scenarios.

## Input / output format

**Input**: Robot state, environment state, goal positions, and obstacle positions. The control module receives these to update state via Kalman filter, compute minimum distance/closest point, and generate control input.

**Output**: Control input (velocity/acceleration) applied to the robot simulator each frame at 20 fps.

## Scoring recipe

```python
def compute_metrics(goals_achieved, collisions, distances, velocities, T, d_s):
    efficiency = goals_achieved
    safety = 0.0
    for t in range(T):
        d = distances[t]
        dd = velocities[t]
        if d < d_s:
            safety += -min(0, math.log(d / d_s)) * dd
    hybrid = goals_achieved if collisions == 0 else 0
    return efficiency, safety, hybrid
```

## Common pitfalls

- The safety score includes a 'psychological safety' component that penalizes high relative velocity near obstacles even without physical collision, which differs from standard collision-rate metrics.
- The hybrid score is only applicable to robot-robot collaboration scenarios, not human-robot interactions, and strictly requires zero collisions to be valid.
- Performance is highly sensitive to algorithm-specific parameters (e.g., d_min, k, c1, c2, eta, lambda) which must be tuned per algorithm for fair comparison.

## Evidence (verbatim from paper)

> We use 40 pieces of 30 seconds long test scenarios to test different algorithms in the experiments. The frame rate is 20 fps. Three metrics are used to evaluate the performance of different algorithms: an efficiency score and a safety score for human-robot interactions, and a hybrid score for robot co-working. For all scores, the higher, the better. We use the average number of goals achieved in a given period by the robot as the efficiency score. Based on these considerations, the safety score is defined as safety = -∑_{0}^{T} min(0, log(d/d_s)) ḋ. We define the hybrid score as the maximum efficiency without collision to evaluate the performance of the algorithms in these situations.

## Citation

```bibtex
@misc{wei2019safecontrol,
  title={Safe Control Algorithms Using Energy Functions: A Unified Framework, Benchmark, and New Directions},
  author={Wei et al. (2019)},
  year={2019},
  note={arXiv:1908.01883}
}
```

- arXiv: 1908.01883