Garment Folding Eval

garment-folding-eval

Learning to Fold Real Garments with One Arm: A Case Study in Cloud-Based Robotics Research — Hoque et al. (2022) (arXiv:2204.10297, 2022)

What this evaluates

Evaluates a robot's ability to manipulate deformable fabric garments using a single arm. It probes two capabilities: flattening a crumpled T-shirt to maximize coverage, and folding a flattened T-shirt to match a goal configuration while minimizing wrinkles.

Datasets

• Google Reach T-shirt folding environment — total ?; splits: (unstated)

Metrics

• max_coverage_pct (primary) — range: percent
  • Percentage of the pixel coverage of a fully flattened shirt (47,000 pixels) achieved by the algorithm. Evaluated over 10 trials from a crumpled initial state, terminating early if a 96% threshold is reached.
• IoU — range: [0, 1]
  • Intersection over Union between the final shirt mask and a goal template, after optimal rotation and translation alignment.
• wrinkle_penalty — range: [0, 1]
  • Fraction of pixels in the interior of the final shirt mask detected as edges by the Canny edge detector.

Input / output format

Input: RGB/depth image observation of the garment state, goal template image (for folding), and current action history.

Output: Discrete action primitives: pick-and-place (p0, p1) or drop, executed via the PyReach Gym API.

Scoring recipe

def score_flattening(final_mask, goal_pixels=47000):
    coverage = count_nonzero(final_mask) / goal_pixels
    return coverage

def score_folding(final_mask, goal_template):
    best_iou = max_iou_over_transformations(final_mask, goal_template)
    edges = canny_edge_detector(final_mask)
    interior = interior_pixels(final_mask)
    wrinkle_penalty = count_nonzero(edges & interior) / count_nonzero(interior)
    return best_iou, wrinkle_penalty

Common pitfalls

• Flattening evaluation terminates early if coverage reaches 96% (45,000 pixels), so reported action counts may be truncated and not reflect total steps taken.
• IoU calculation requires optimal rotation and translation alignment of the goal template to the final state, which is non-trivial to implement without the paper's code.
• Wrinkle penalty uses Canny edge detection on the shirt mask interior, which can be sensitive to lighting conditions and camera calibration.

Evidence (verbatim from paper)

In Table II we report maximum coverage as a percentage of the pixel coverage of a fully flattened shirt, i.e. 47,000 pixels in the shirt mask. In Table III we report the number of actions and execution time per action, and we measure the quality of the final state against a goal configuration according to two metrics: (1) intersection over union (IoU) and (2) a penalty for edges and wrinkles. IoU is calculated between the shirt mask and the goal template, after rotating and translating the goal to best match the shirt mask. The wrinkle penalty calculates the fraction of pixels in the interior of the shirt mask detected as edges by the Canny edge detector.

Citation

@misc{hoque2022learning,
  title={Learning to Fold Real Garments with One Arm: A Case Study in Cloud-Based Robotics Research},
  author={Hoque et al. (2022)},
  year={2022},
  note={arXiv:2204.10297}
}

• arXiv: 2204.10297