Libero Eval

libero-eval

LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning — Liu et al. (2023) (arXiv:2306.03310, 2023)

What this evaluates

Evaluates a robot policy's ability to sequentially learn multiple manipulation tasks while transferring knowledge and minimizing catastrophic forgetting. It measures forward transfer speed, backward transfer (forgetting), and overall performance across a curriculum of procedurally generated tasks.

Datasets

• LIBERO-LONG — total ?; splits: task_suite (-1)
• LIBERO-SPATIAL — total ?; splits: task_suite (-1)
• LIBERO-OBJECT — total ?; splits: task_suite (-1)
• LIBERO-GOAL — total ?; splits: task_suite (-1)

Metrics

• FWT (primary) — range: [0, 1]
  • Forward Transfer. FWT = (1/K) * sum_{k=1}^K FWT_k, where FWT_k = (1/11) * sum_{e in {0..50}} c_{k,k,e}. Measures the average success rate across training epochs for each task k. Higher is better.
• NBT — range: [0, 1]
  • Negative Backward Transfer. NBT = (1/K) * sum_{k=1}^K NBT_k, where NBT_k = (1/(K-k)) * sum_{tau=k+1}^K (c_{k,k} - c_{tau,k}). Measures average performance drop on task k after learning subsequent tasks tau. Lower is better.
• AUC — range: [0, 1]
  • Area Under the Success Rate Curve. AUC = (1/K) * sum_{k=1}^K AUC_k, where AUC_k = (1/(K-k+1)) * (FWT_k + sum_{tau=k+1}^K c_{tau,k}). Combines forward transfer and backward transfer to measure overall lifelong learning performance. Higher is better.

Input / output format

Input: RGB images of the robot's environment and language embeddings (e.g., BERT task descriptions) representing the current task.

Output: Robot action vector (continuous or discrete control commands for manipulation).

Scoring recipe

# c[i][j][e] = success rate on task j after learning i-1 tasks and e epochs on task i
# K = total tasks, epochs = [0, 5, ..., 50]
c_best = [max(c[i][i]) for i in range(K)]
e_star = [next(e for e in epochs if c[i][i][e] == c_best[i]) for i in range(K)]
c_clamped = [[c_best[i] if e >= e_star[i] else c[i][i][e] for e in epochs] for i in range(K)]
fwt_k = [sum(c_clamped[k][e] for e in epochs) / 11 for k in range(K)]
nbt_k = [(c_best[k] - sum(c[tau][k][e_star[tau]] for tau in range(k+1, K)) / (K - k)) for k in range(K)]
auc_k = [(fwt_k[k] + sum(c[tau][k][e_star[tau]] for tau in range(k+1, K))) / (K - k + 1) for k in range(K)]
return {"FWT": sum(fwt_k)/K, "NBT": sum(nbt_k)/K, "AUC": sum(auc_k)/K}

Common pitfalls

• NBT is Negative Backward Transfer, so lower values indicate less forgetting (better performance), contrary to typical accuracy metrics.
• Success rates are not evaluated continuously; they are sampled at epochs {0, 5, ..., 50} and clamped to the best achieved rate for all subsequent epochs.
• AUC here does not refer to ROC-AUC; it is a custom metric combining forward transfer and backward transfer over the task sequence.

Evidence (verbatim from paper)

We report three metrics: FWT (forward transfer) [20], NBT (negative backward transfer), and AUC (area under the success rate curve). All metrics are computed in terms of success rate, as previous literature has shown that the success rate is a more reliable metric than training loss for manipulation policies [42] (Detailed explanation in Appendix E.2). Lower NBT means a policy has better performance in the previously seen tasks, higher FWT means a policy learns faster on a new task, and higher AUC means an overall better performance considering both NBT and FWT. Specifically, denote $c_{i,j,e}$ as the agent's success rate on task $j$ when it learned over $i - 1$ previous tasks and has just learned $e$ epochs ( $e \in {0,5,\dots,50}$ ) on task $i$ . Let $c_{i,i}$ be the best success rate over all evaluated epochs $e$ for the current task $i$ (i.e., $c_{i,i} = \max_e c_{i,i,e}$ ). Then, we find the earliest epoch $e_i^$ in which the agent achieves the best performance on task $i$ (i.e., $e_i^ = \arg \min_e c_{i,i,e_i} = c_{i,i}$ ), and assume for all $e \geq e_i^*$ , $c_{i,i,e} = c_{i,i}$ . Given a different task $j \neq i$ , we define $c_{i,j} = c_{i,j,e_i

Citation

@misc{liu2023libero,
  title={LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning},
  author={Liu et al. (2023)},
  year={2023},
  note={arXiv:2306.03310}
}

• arXiv: 2306.03310