Thinkjepa Ego Dex Eval

thinkjepa-ego-dex-eval

ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model — Zhang et al. (2026) (arXiv:2603.22281, 2026)

What this evaluates

Evaluates a model's ability to forecast future 3D hand/joint trajectories and latent video representations from egocentric video inputs. It probes long-horizon temporal consistency and physical plausibility in dexterous manipulation scenarios.

Datasets

• EgoDex — total ?; splits: train (-1), val (-1), test (-1)
• EgoExo4D — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• ADE (primary) — range: other
  • Mean Euclidean distance between predicted and ground-truth 3D joint positions across all future frames and joints, averaged over the batch.
• FDE — range: other
  • Mean Euclidean distance on the final future frame, averaged over joints and batch.
• Accuracy — range: percent
  • Fraction of predicted joint positions where the Euclidean error is below 0.05 m, aggregated over time and joints.
• FD — range: other
  • Feature L2 distance between predicted and target V-JEPA latent representations.
• SL1 — range: other
  • SmoothL1 distance between predicted and target latents.
• CD — range: other
  • Cosine distance, defined as 1 - cosine similarity, between predicted and target latents.
• A@H / F@H — range: other
  • ADE@H and FDE@H computed after recursive autoregressive rollout to horizon H ∈ {4, 8, 16, 32}.

Input / output format

Input: Sequence of past egocentric video frames (typically 32 frames) used to predict future 3D hand/joint trajectories and latent video representations.

Output: Predicted 3D joint coordinates Y_hat ∈ R^(B x T_f x J x 3) for future frames, and predicted latent feature vectors.

Scoring recipe

def compute_metrics(pred_traj, gt_traj, pred_lat, gt_lat, threshold=0.05):
    dist = np.linalg.norm(pred_traj - gt_traj, axis=-1)
    ade = np.mean(dist)
    fde = np.mean(dist[:, -1, :])
    acc = np.mean(dist < threshold)
    fd = np.mean(np.linalg.norm(pred_lat - gt_lat, axis=-1))
    sl1 = np.mean(smooth_l1_loss(pred_lat, gt_lat))
    cd = np.mean(1 - cosine_similarity(pred_lat, gt_lat))
    return ade, fde, acc, fd, sl1, cd

Common pitfalls

• Latent metrics (FD/SL1/CD) are computed on V-JEPA feature space, not raw pixel space.
• Rollout metrics (A@H/F@H) require recursive autoregressive prediction, not single-step forecasting.
• Accuracy uses a strict 0.05 m Euclidean threshold per joint, not a global pose similarity metric.

Evidence (verbatim from paper)

We compute: ADE (Average Displacement Error): the mean Euclidean distance over all future frames and joints, averaged over the batch. FDE (Final Displacement Error): the mean Euclidean distance on the final future frame, averaged over joints and batch. Accuracy: the fraction of predicted joint positions with Euclidean error below 0.05 m, aggregated over time and joints.

Citation

@misc{zhang2026thinkjepa,
  title={ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2603.22281}
}

• arXiv: 2603.22281