Gibson Env Eval

gibson-env-eval

Gibson Env: Real-World Perception for Embodied Agents — Xia et al. (2018) (arXiv:1808.10654, 2018)

What this evaluates

Evaluates the geometric complexity, scene diversity, and sim-to-real transfer capability of the Gibson virtual environment. It benchmarks neural rendering pipelines and embodied agents on tasks like depth estimation, scene classification, and navigation.

Datasets

• Gibson — total 572; splits: full (572)

Metrics

• Real-World Transfer Error (primary) — range: meters
  • Average depth estimation error in meters when a network trained on synthetic database images is tested on real-world 2D-3D-S images.
• Depth Estimation Error — range: meters
  • Mean absolute difference between predicted and ground truth depth maps, measured in meters.
• Scene Class Accuracy — range: [0, 1]
  • Top-1 classification accuracy on scene categories using a pretrained network on 10k random images per database.
• SSIM — range: [0, 1]
  • Structural Similarity Index Measure quantifying perceptual image similarity between rendered and real images.
• L1 — range: [0, 1]
  • Mean absolute pixel-wise difference between rendered and real images.
• MMD — range: other
  • Maximum Mean Discrepancy computed on VGG16 features to measure distributional distance between synthetic and real domains.
• CORAL — range: other
  • Correlation Alignment distance computed on VGG16 features to measure distributional distance between synthetic and real domains.
• Average Reward — range: other
  • Mean cumulative reward over evaluation iterations for navigation tasks, incorporating distance decrease and collision penalties.

Input / output format

Input: RGB-D panorama images (1024x2048 or cropped 256x256) and agent state (position, orientation, target relative position).

Output: Predicted depth maps, scene class labels, action logits (forward/backward/left/right or continuous torques), and rendered images.

Scoring recipe

def compute_metrics(preds, gold):
    depth_err = np.mean(np.abs(preds['depth'] - gold['depth']))
    class_acc = np.mean(preds['class'] == gold['class'])
    ssim_val = ssim(preds['img'], gold['img'])
    l1_val = np.mean(np.abs(preds['img'] - gold['img']))
    mmd_val = max_mean_discrepancy(preds['features'], gold['features'])
    coral_val = correlation_alignment(preds['features'], gold['features'])
    avg_reward = np.mean(preds['rewards'])
    return {'depth_err': depth_err, 'class_acc': class_acc, 'ssim': ssim_val, 'l1': l1_val, 'mmd': mmd_val, 'coral': coral_val, 'avg_reward': avg_reward}

Common pitfalls

• Domain gap between synthetic and real images can cause depth estimation errors to inflate significantly if the Goggles domain-adaptive network is not applied.
• Navigation complexity metric depends on A* pathfinding on reconstructed meshes, which may vary with mesh quality and post-processing steps.
• Scene diversity entropy calculation relies on a fixed pretrained classifier, making it sensitive to the classifier's inherent bias and not a pure measure of environment variety.

Evidence (verbatim from paper)

Real-World Transfer Error: We train a neural network for depth estimation using the images of each database and test them on real images of 2D-3D-S dataset [10]. Training images of SUNCG and Matterport3D are rendered using MINOS [76] and our dataset is rendered using Gibson's engine. The training set of each database is 20k random RGB-depth image pairs with 90∘ field of view. The reported value is average depth estimation error in meters.

Citation

@misc{xia2018gibson,
  title={Gibson Env: Real-World Perception for Embodied Agents},
  author={Xia et al. (2018)},
  year={2018},
  note={arXiv:1808.10654}
}

• arXiv: 1808.10654