Mixing Transforms

Mixing Transforms Policy

Apply this skill when implementing, reviewing, or using transforms that combine data from multiple images: Mosaic, CopyAndPaste, OverlayElements, HistogramMatching, PixelDistributionAdaptation, etc.

1. The caller owns the donor pool

The caller supplies donor records under metadata_key; a transform never accesses a dataset, loader, or global donor source. A normal Mosaic call supplies the needed donors, and every valid supplied donor is used.

Mosaic handles caller mistakes deterministically within that pool:

• if there are more valid donors than visible cells need, it samples only the surplus away with the invocation's SamplingContext RNG;
• if there are fewer, it fills remaining cells by replicating the primary item.

This preserves user control over the candidate set while keeping the public transform usable with an oversized or undersized list.

# CORRECT — caller provides candidate donor records
donors = [dataset[random.choice(indices)] for _ in range(n)]
result = transform(image=image, mosaic_metadata=donors)

# INCORRECT — transform reaches into a dataset on its own
result = MosaicWithSampling(dataset=dataset)(image=image)

2. Metadata format: list[dict]

All mixing transforms receive auxiliary data as list[dict] under a metadata_key. Each dict is one item (one full image for Mosaic, one object instance for CopyAndPaste). This is consistent across transforms.

mosaic_metadata = [
    {"image": img1, "mask": mask1, "bboxes": bboxes1, "bbox_labels": {...}},
    {"image": img2, ...},
]

copy_paste_metadata = [
    {"image": src_img, "mask": obj_mask, "bbox": [x1, y1, x2, y2], "bbox_labels": {"class_id": 3}},
    {"image": src_img, "mask": obj_mask2, "bbox_labels": {"class_id": 7}},
]

3. Label fields: bbox_labels and keypoint_labels (dicts)

All mixing transforms use the same wrapper dict convention for labels:

• bbox_labels: dict[str, Any] — maps each label field name (as declared in BboxParams.label_fields) to its value(s) for this item.
• keypoint_labels: dict[str, Any] — maps each label field name (as declared in KeypointParams.label_fields) to its value(s) for this item.

For CopyAndPaste (one object per dict), values are scalars (one bbox, one object):

{
    "image": src_image,
    "mask": obj_mask,
    "bbox": [10, 20, 50, 80],  # same coord_format as BboxParams
    "bbox_labels": {
        "class_id": 3,
        "is_crowd": 0,
    },
    "keypoints": [[25, 40]],  # same coord_format as KeypointParams
    "keypoint_labels": {
        "joint_name": "left_eye",
    },
}

For Mosaic (one full image per dict), values are lists — one entry per bbox/keypoint:

{
    "image": img,
    "bboxes": [[10, 20, 50, 80], [5, 5, 30, 30]],
    "bbox_labels": {
        "class_id": [3, 7],
        "is_crowd": [0, 1],
    },
    "keypoints": [[25, 40], [60, 70]],
    "keypoint_labels": {
        "joint_name": ["left_eye", "nose"],
    },
}

Key rule: the dict keys in bbox_labels / keypoint_labels must exactly match what is declared in BboxParams(label_fields=[...]) and KeypointParams(label_fields=[...]).

4. Coordinates use the same format as BboxParams / KeypointParams

Bboxes and keypoints in metadata dicts must use the same coord_format as declared in Compose. The processor's preprocess() converts them to the internal albumentations format — no manual conversion needed.

# BboxParams declared with coord_format='pascal_voc'
# → bboxes in metadata must also be pascal_voc [x_min, y_min, x_max, y_max]
copy_paste_metadata = [
    {"image": img, "mask": m, "bbox": [10, 20, 50, 80], "bbox_labels": {"class_id": 3}},
]

5. metadata_key pattern

Every mixing transform exposes metadata_key: str in its constructor and lists it in targets_as_params. This ensures Compose validates that the key is present.

@property
def targets_as_params(self) -> list[str]:
    return [self.metadata_key]

6. Empty metadata is transform-specific

Do not impose a universal no-op rule on mixing transforms. CopyAndPaste returns no-op parameters when it has no usable donor. Mosaic instead creates its remaining visible cells from replicated primary data, so empty or missing metadata can still produce a mosaic. State this behavior in the transform's public docstring and test it explicitly.

# CopyAndPaste: no donor means no change.
if not usable_donors:
    return self._no_op_params()

# Mosaic: remaining cells use copies of the primary item.
final_items = [primary, *usable_donors, *replicated_primary_items]