# Generalized Few Shot Point Cloud Segmentation > Segment novel 3D point cloud classes with few support samples by combining dense but noisy pseudo-labels from 3D vision-language models with precise sparse few-shot annotations. GFS-VL adapts to new classes while retaining base class performance, using prototype-guided filtering and adaptive infilling strategies ideal for applications with limited labeled training data. - Skill: `adu2021/generalized-few-shot-point-cloud-segmentation` (Agent Skill) - Install (CLI): `npx skillmds@latest add adu2021/generalized-few-shot-point-cloud-segmentation` - Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/generalized-few-shot-point-cloud-segmentation/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - License: MIT - Author: adu2021 (https://skillmd.com/u/adu2021) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/adu2021/generalized-few-shot-point-cloud-segmentation --- ## Core Concept Generalized few-shot 3D point cloud segmentation (GFS-PCS) solves a critical challenge: segmenting new object classes in 3D scenes using only a handful of labeled examples while maintaining performance on known classes. Existing methods rely on sparse few-shot knowledge, limiting generalization. GFS-VL synergizes two complementary sources: dense but noisy pseudo-labels from 3D vision-language models (which generalize broadly but contain errors) and precise yet sparse few-shot annotations (which are accurate but limited). This dual-source approach maximizes strengths while mitigating weaknesses—enabling practical deployment in scenarios where annotation budgets are tight but label quality matters. ## Architecture Overview The framework orchestrates five interconnected components: - **3D Vision-Language Model Backbone**: Leverages pretrained 3D VLMs to generate pseudo-labels across unlabeled regions with open-world generalization - **Prototype-Guided Pseudo-Label Selection**: Filters low-quality pseudo-labels using prototype matching to retain only reliable predictions - **Adaptive Infilling Strategy**: Combines pseudo-label contexts with few-shot samples to intelligently label previously unlabeled filtered regions - **Novel-Base Mix Strategy**: Embeds few-shot samples into training scenes preserving spatial context for improved learning - **Diverse Benchmarks**: Two challenging datasets with varied novel classes for comprehensive generalization evaluation ## Implementation Steps ### 1. 3D VLM Pseudo-Label Generation Initialize pretrained 3D vision-language models to generate initial pseudo-labels across scenes. These models generalize to novel classes but may contain noise: ```python import torch import numpy as np from typing import Tuple, Dict def generate_pseudo_labels_from_vlm(point_cloud: torch.Tensor, vlm_model, novel_class_descriptions: Dict[str, str] ) -> Tuple[torch.Tensor, torch.Tensor]: """ Generate pseudo-labels using pretrained 3D vision-language model. Handles dense but potentially noisy predictions across point cloud. """ # Point cloud shape: (N, 3) for coordinates num_points = point_cloud.shape[0] # Encode class descriptions using VLM text encoder class_names = list(novel_class_descriptions.keys()) text_features = vlm_model.encode_text(class_names) # (num_classes, dim) # Extract geometric features from point cloud point_features = vlm_model.encode_geometry(point_cloud) # (N, dim) # Compute similarities for pseudo-label prediction similarities = torch.matmul(point_features, text_features.t()) # (N, num_classes) pseudo_labels = torch.argmax(similarities, dim=1) # (N,) confidence_scores = torch.max(torch.softmax(similarities, dim=1), dim=1)[0] # (N,) return pseudo_labels, confidence_scores ``` ### 2. Prototype-Guided Pseudo-Label Selection Filter pseudo-labels by comparing point features to class prototypes derived from few-shot samples. Only retain high-confidence regions: ```python def prototype_guided_selection(point_cloud: torch.Tensor, pseudo_labels: torch.Tensor, confidence_scores: torch.Tensor, few_shot_samples: Dict[int, torch.Tensor], threshold: float = 0.7) -> Tuple[torch.Tensor, torch.Tensor]: """ Select reliable pseudo-labels by comparing to few-shot prototypes. Filters low-quality regions to prevent error propagation during training. """ num_points = point_cloud.shape[0] num_classes = len(few_shot_samples) # Compute class prototypes from few-shot samples prototypes = {} for class_id, samples in few_shot_samples.items(): prototypes[class_id] = torch.mean(samples, dim=0) # Average feature vector # Compare each point to prototypes selected_mask = torch.zeros(num_points, dtype=torch.bool) for idx in range(num_points): point_feat = point_cloud[idx] pred_class = pseudo_labels[idx] # Distance to predicted class prototype if pred_class in prototypes: dist_to_prototype = torch.norm(point_feat - prototypes[pred_class]) # Accept if confidence high and close to prototype if confidence_scores[idx] > threshold and dist_to_prototype < 2.0: selected_mask[idx] = True selected_labels = pseudo_labels[selected_mask] return selected_mask, selected_labels ``` ### 3. Adaptive Infilling Strategy For regions filtered out (unreliable pseudo-labels), combine few-shot knowledge with pseudo-label context to label previously unlabeled areas adaptively: ```python def adaptive_infilling(point_cloud: torch.Tensor, selected_mask: torch.Tensor, few_shot_samples: Dict[int, torch.Tensor], vlm_pseudo_labels: torch.Tensor, k_neighbors: int = 5) -> torch.Tensor: """ Adaptively label filtered regions by blending few-shot and pseudo-label signals. Uses spatial proximity to improve coherence of inferred labels. """ num_points = point_cloud.shape[0] infilled_labels = vlm_pseudo_labels.clone() # For each unselected point, find k-nearest reliable selected points unselected_indices = torch.where(~selected_mask)[0] for idx in unselected_indices.tolist(): # Compute distances to all selected points point = point_cloud[idx] selected_points = point_cloud[selected_mask] distances = torch.norm(selected_points - point, dim=1) # Find k nearest neighbors among selected points k_nearest_indices = torch.topk(distances, k=min(k_neighbors, len(distances)), largest=False)[1] # Get labels of k nearest neighbors selected_indices_full = torch.where(selected_mask)[0] neighbor_labels = infilled_labels[selected_indices_full[k_nearest_indices]] # Use majority vote from neighbors infilled_labels[idx] = torch.mode(neighbor_labels)[0] return infilled_labels ``` ### 4. Novel-Base Mix Strategy Embed few-shot samples into training scenes while preserving spatial context. This prevents overfitting to isolated examples: ```python def novel_base_mix_strategy(base_scene: torch.Tensor, base_labels: torch.Tensor, few_shot_samples: Dict[int, torch.Tensor], few_shot_labels: Dict[int, torch.Tensor], num_samples_per_class: int = 10) -> Tuple[torch.Tensor, torch.Tensor]: """ Mix few-shot novel class samples into base class scenes. Preserves spatial context while augmenting training diversity. """ # Select samples from base scene selected_base_indices = np.random.choice(len(base_scene), num_samples_per_class, replace=False) base_subset = base_scene[selected_base_indices] base_subset_labels = base_labels[selected_base_indices] # Collect few-shot samples for novel classes novel_points_list = [] novel_labels_list = [] for class_id, samples in few_shot_samples.items(): # Randomly select subset of few-shot samples num_to_select = min(num_samples_per_class // len(few_shot_samples), len(samples)) selected_indices = np.random.choice(len(samples), num_to_select, replace=False) novel_points_list.append(samples[selected_indices]) novel_labels_list.extend([class_id] * num_to_select) novel_points = torch.cat(novel_points_list, dim=0) # Concatenate base and novel samples mixed_points = torch.cat([base_subset, novel_points], dim=0) mixed_labels = torch.cat([base_subset_labels, torch.tensor(novel_labels_list)], dim=0) return mixed_points, mixed_labels ``` ### 5. Training with Mixed Supervision Train the segmentation model with both base class supervision and adapted novel class labels: ```python import torch.nn.functional as F def train_segmentation_model(model, mixed_points: torch.Tensor, mixed_labels: torch.Tensor, infilled_labels: torch.Tensor, learning_rate: float = 0.001, num_epochs: int = 50): """ Train segmentation model using mixed supervision from base classes and adaptively infilled novel class labels. """ optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate) for epoch in range(num_epochs): # Forward pass on mixed training data logits_mixed = model(mixed_points) loss_mixed = F.cross_entropy(logits_mixed, mixed_labels) # Auxiliary loss on full-scene predictions with infilled labels logits_full = model(mixed_points) # In practice, use full scene loss_infilled = F.cross_entropy(logits_full, infilled_labels[:len(logits_full)]) # Combined loss balances both signals total_loss = loss_mixed + 0.5 * loss_infilled optimizer.zero_grad() total_loss.backward() optimizer.step() if (epoch + 1) % 10 == 0: print(f"Epoch {epoch + 1}: Loss = {total_loss.item():.4f}") return model ``` ## Practical Guidance ### When to Use GFS-VL - **Limited Annotation Budget**: Few-shot annotation of novel classes while reusing base class models - **Diverse Object Categories**: Scenarios with many novel classes but sparse examples per class - **Open-World Deployment**: Adding new classes to existing point cloud systems without full retraining - **Robotics & Autonomous Systems**: Adapting perception models to new environments with minimal labeling - **Scientific Discovery**: Segmenting novel structures in medical imaging or geological point clouds ### When NOT to Use GFS-VL - **Large Labeled Datasets Available**: If thousands of labeled examples exist, standard supervised segmentation is simpler - **Real-Time Inference Constraints**: Pseudo-label filtering adds computational overhead - **Highly Imbalanced Classes**: Few-shot approach struggles when novel classes have extreme size variations - **Noisy 3D Sensors**: If input point clouds are severely corrupted, VLM pseudo-labels may be unreliable - **Extreme Domain Shift**: If novel classes differ drastically from base and training data, adaptation fails ### Hyperparameter Tuning - **Confidence Threshold**: Default 0.7; lower to 0.5 for higher recall on uncertain regions, raise to 0.85 for precision - **Prototype Distance Threshold**: Default 2.0; adjust based on feature space dimensionality - **K-Neighbors in Infilling**: Default 5; increase to 10 for smoother spatial coherence, decrease for local detail preservation - **Novel-Base Mix Ratio**: Default 1:1; skew toward base samples (0.7:0.3) if novel class regions are sparse - **Learning Rate**: Start at 0.001; reduce to 0.0001 if loss oscillates during training - **Number of Samples per Class**: Default 10; increase to 20+ if GPU memory permits for better diversity ### Common Pitfalls 1. **Trusting All VLM Pseudo-Labels**: VLMs generalize broadly but produce noisy predictions. Always filter using prototypes 2. **Ignoring Spatial Coherence**: Infilling without neighbor context creates fragmented, unrealistic segmentations 3. **Overfitting to Few Samples**: If few-shot samples don't cover class variations, model memorizes spurious features 4. **Mixing Incompatible Base-Novel Classes**: Ensure novel classes don't overlap semantically with base classes or infilling fails 5. **Forgetting Base Class Performance**: Regularize training to maintain base class accuracy while adapting to novel classes 6. **Imbalanced Training Mix**: Skewing mix ratios toward base or novel causes class bias in predictions ## References - 3D Vision-Language Models: Cross-modal learning for open-world 3D understanding - Prototype Learning: Distance metric learning for few-shot adaptation - Point Cloud Processing: PointNet and transformer-based architectures for 3D segmentation - Pseudo-Labeling: Semi-supervised learning techniques for leveraging unlabeled data - Few-Shot Learning: Meta-learning and transfer learning fundamentals - CVPR 2025 Challenge Benchmarks: Two new diverse datasets for robust generalization evaluation