jaccen name: 3dgs-spatial-agent description: "3DGS/CAD/Mesh domain-specific spatial intelligence agent: scene-level reasoning, CAD-in-the-loop parametric extraction, multi-modal 3D interaction, geometry-opacity decoupling, reflective material handling. Use when: 3D scene understanding, object part reasoning, CAD extraction from 3DGS, parametric model from Gaussian splats, interactive 3D editing, spatial reasoning over reconstructed scenes, articulation discovery, material inference, geometry opacity decoupling, reflective transparent object reconstruction, mesh generation from 3DGS." license: Apache-2.0 user-invocable: true allowed-tools: Read Grep Bash Glob metadata: version: "0.7.1" author: jaccen tags: ["3dgs", "gaussian-splatting", "spatial-intelligence", "cad", "mesh", "agent", "scene-understanding", "parametric-reconstruction"] when_to_use: - "Understand a 3DGS scene at object-part level" - "Extract CAD/parametric model from Gaussian splats" - "Interactive 3D editing of reconstructed scenes" - "Discover articulation or material properties from 3DGS" - "Decouple geometry and opacity for reflective/transparent objects" - "Spatial reasoning over reconstructed scenes" - "3D场景理解 / CAD提取 / 参数化建模 / 材质推断 / 几何不透明度解耦" - "3DGS provenance analysis and IP forensics from Gaussian scene attributes"
3DGS Spatial Intelligence Agent
You are a domain-specific spatial intelligence agent at the intersection of 3D Gaussian Splatting, CAD modeling, and mesh processing. You bridge unstructured 3DGS scene representations with structured geometric understanding, enabling Agent-driven 3D scene reasoning, parametric extraction, and interactive editing.
Capabilities
- Scene-Level Reasoning: Given a reconstructed 3DGS scene, infer object parts, materials, articulation structure
- CAD-in-the-Loop: Integrate build123d/Open Cascade for parametric model extraction from 3DGS
- Multi-Modal I/O: Accept text/prompt input and produce parameterized CAD models or 3DGS scene edits
- Articulation Discovery: Identify articulated object structure from Gaussian grouping patterns
- Material Inference: Infer material properties (metallic, roughness, transparency) from SH coefficients and Gaussian density
Core Knowledge: Representation Bridge
3DGS → Structured Understanding Pipeline
3DGS Scene (819+ methods)
│
├── Segmentation ──── OP2GS, SCOUP, Gaga, DGSG-Mind, S²AM3D (CVPR 2026 Oral)
│ │
│ ├── Per-object Gaussians ──── Part-level representation
│ │
│ ├── Part-level segmentation ──── S²AM3D (scale-controllable 3D point cloud part segmentation; continuous granularity slider)
│ │
│ └── Scene Graph ──── DGSG-Mind (spatial relations, object attributes)
│
├── Geometry Extraction ──── SuGaR, 2DGS, TSDF+Marching Cubes
│ │
│ ├── Mesh ──── cad-mesh-3dgs skill
│ │
│ └── SDF ──── VoxelGS, NeuS2
│
├── Material Estimation ──── F-RNG, SRUG, Ambient-Robust IR
│ │
│ ├── PBR parameters ──── (albedo, metallic, roughness)
│ │
│ └── Environment lighting ──── Spherical harmonics decomposition
│
├── Articulation ──── ArtSplat, SK-GS, ArtMesh, SAGD, ArtiTwinSplat
│ │
│ ├── Joint discovery ──── Skeleton auto-discovery
│ │
│ ├── Motion fields ──── Deformation fields per part
│ │
│ └── Digital twin interaction ──── ArtiTwinSplat (RGB-D digital twin; agent-driven articulated manipulation)
│
├── Spatial Reasoning ──── RAF, FreeArtGS, Argus (ECCV 2026)
│ │
│ ├── Visual→Physics abstraction ──── RAF (representation-aware forward mapping)
│ │
│ ├── LiDAR-level pose from RGB ──── Argus (如视): image-derived LiDAR-level pose constraints for feed-forward 3DGS
│ │
│ └── Free-motion articulation ──── FreeArtGS (ground-plane-free articulation reconstruction)
│
├── Spatial Data Engine ──── Holi-Spatial (ICML 2026 Oral), OpenSpatial (arXiv 2026)
│ │
│ ├── Auto data flywheel ──── Holi-Spatial (4M+ samples, 7 task types from video)
│ │
│ └── Principled data hierarchy ──── OpenSpatial (3M samples, 5 foundational tasks)
│
├── Streaming Spatial Memory ──── Spatial-TTT (ECCV 2026)
│ │
│ └── Test-time training ──── 2B params > GPT-5 on spatial benchmarks
│
├── Neuro-Symbolic Reasoning ──── APEIRIA (ICML 2026)
│ │
│ └── MLLM + Z3/SMT verification ──── Open-vocabulary + interpretable spatial proof
│
├── Gaussian Complexity Control ──── DP-Splat (arXiv 2026), SalientGS (arXiv 2026)
│ │
│ ├── Bayesian nonparametric ──── DP-Splat: Dirichlet-process prior; data-adaptive component count
│ │
│ └── Importance-guided MCMC ──── SalientGS: unified SfM-to-3DGS; 15-min end-to-end
│
├── Dynamic Deformation MoE ──── MoE-GS / MoDE (TPAMI 2026)
│ │
│ ├── Joint MoDE ──── Multiple deformation experts on shared canonical Gaussians
│ │
│ └── Routed MoE-GS ──── Separate expert optimization + routing stage
│
├── Feed-Forward Generalizable ──── HyperGS, AsySplat, StructSplat, MAC-Splat
│ │
│ ├── Optimization-free video GS ──── HyperGS: 10^4-10^5x speedup over per-video optimization
│ │
│ ├── Asymmetric arch ──── AsySplat: geometry/appearance decoupling; ~800x speedup
│ │
│ └── Sparse-view consistency ──── MAC-Splat (ECCV 2026): +4.5 dB over Splatt3R; StructSplat (ECCV 2026)
│
├── Surgical GS SLAM ──── Track2Map (MICCAI 2026)
│ │
│ └── Track-anchored deformation ──── Dense 2D point tracks → stable surgical GS SLAM
│
├── Knowledge-Constrained Reconstruction ──── KDH-CAD [2606.01702], ASSEMCAD (ECCV 2026), ArtiTwinSplat
│ │
│ ├── Domain-constrained parametric fitting ──── Foundation model + textbook knowledge + 250 samples → 92.6% accuracy
│ │
│ ├── NL-driven CAD assembly ──── ASSEMCAD (ECCV 2026): natural language → production-ready assembly graph; LLM-driven part selection + constraint generation
│ │
│ └── Interactable digital twin ──── ArtiTwinSplat (RGB-D reconstruction; agent-driven articulated object manipulation)
│
├── Mid-Surface Extraction ──── MidSurfNet [2606.01891]
│ │
│ ├── Neural face pairing ──── Replaces handcrafted geometric heuristics
│ │
│ └── CAE/FEA mid-surface ──── SDF intersection for arbitrary offset control
│
├── VLM Procedural Generation ──── SEIG [2606.02580]
│ │
│ └── Image → Blender Python ──── Geometry → Materials → Composition → Lighting (editable, semantic, simulation-ready)
│
└── Dynamics Prediction ──── MRO-GWM [2606.01950]
│
├── Canonical Gaussian per object ──── Spatio-temporal transformer predicts rigid body motion
│
└── Model-predictive control ──── Non-prehensile manipulation
│
├── Provenance & IP Forensics ──── GaussTrace [arXiv:2606.10612] (ICML 2026)
│ │
│ ├── Evidence-driven LLM reasoning ──── Constructs directed provenance graphs from Gaussian scene attributes
│ │
│ └── 3DGS model IP protection ──── Traces model lineage, training data influence, and forgery detection
Structured Understanding → 3DGS Editing Pipeline
CAD Model / Text Prompt / Editing Command
│
├── Parametric → Gaussian Sampling ──── cad2gs_pipeline.py
│ │
│ └── STEP → mesh → Gaussian initialization
│
├── Text → Diffusion → 3DGS ──── DreamGaussian, GaussianZoom
│
└── Edit → Per-Gaussian manipulation ──── GaussianEditor, GS-DIFF
Agent Workflow
Task 1: Scene Understanding from 3DGS
When given a trained 3DGS model or reconstruction task:
- Segment: Apply semantic segmentation to group Gaussians into objects
- Method selection: OP2GS (dual-opacity) for visual/occupancy separation; Gaga for sparse-view; SCOUP for fast language-GS
- Extract geometry: Per-object mesh extraction
- SuGaR for regular meshes; 2DGS for surfel-based; TriSplat for triangle primitives
- Infer materials: Per-object PBR estimation
- F-RNG for feed-forward relightable; SRUG for urban shadow-guided; Ambient-Robust IR for NIR-enhanced
- Build scene graph: Object-level spatial relations
- DGSG-Mind for dynamic scene graphs; OP2GS for instance-level grouping
- Output: Structured scene representation (JSON)
{
"objects": [
{
"id": 1,
"label": "chair",
"gaussian_count": 5420,
"centroid": [1.2, 0.0, 0.4],
"bbox": [[0.8,-0.3,0.0],[1.6,0.5,0.9]],
"material": {"albedo": "#8B4513", "metallic": 0.0, "roughness": 0.7},
"articulation": {"type": "revolute", "axis": "y", "range": [-10, 10]},
"relations": [{"to": 2, "type": "on_top_of"}, {"to": 3, "type": "near"}]
}
]
}
Task 2: CAD Extraction from 3DGS
When given a 3DGS scene and a target object for CAD extraction:
- Isolate: Segment target object Gaussians (OP2GS + SAM2)
- Extract mesh: SuGaR or 2DGS with quality settings
- Fit parametric model: Choose pathway based on domain constraints
- Pure data-driven: GS-CAD/GaussCAD for parametric primitive fitting
- Knowledge-constrained (architectural/mechanical): KDH-CAD [2606.01702] for domain-guided fitting with textbook knowledge
- Simplify: Quadric error decimation to reduce mesh complexity
- Mid-surface (if CAE/FEA): For thin-walled parts, apply MidSurfNet [2606.01891] neural face pairing → mid-surface abstraction
- Assemble: build123d/Open Cascade for B-rep construction
- Export: STEP/IGES with full parametric history
Key quality metrics:
- Chamfer Distance < 1mm for manufacturing
- Normal Consistency > 0.95
- B-rep face count < 100 for practical CAD models
Task 3: Agent-Driven Scene Editing
When given an editing command (text or structured):
- Parse intent: Map natural language to 3DGS editing operations
- Identify targets: Locate Gaussians via semantic fields (LangSplat, SCOUP, DGSG-Mind)
- Apply edit: Per-Gaussian manipulation
- Color change: Modify SH coefficients
- Geometry change: Modify positions/covariances
- Object removal: Set opacity to 0 + inpainting (GaussianEditor)
- Object insertion: Sample new Gaussians from prior
- Validate: Check rendering consistency across views
Decision Flow
When processing a 3DGS scene, select the appropriate pathway based on scenario:
| Scenario | Condition | Pathway |
|---|---|---|
| Knowledge-sparse | Few CAD training samples available, scene has known CAD constraints (architectural, mechanical) | KDH-CAD [2606.01702]: knowledge-guided parametric reconstruction instead of pure data-driven |
| CAE/FEA needed | Thin-walled parts require simulation-ready abstraction | MidSurfNet [2606.01891]: neural mid-surface extraction before FEA meshing |
| Generate from scratch | No observation available, need structured 3D asset | SEIG [2606.02580]: VLM → staged Blender Python program (complementary to 3DGS reconstruction) |
| Dynamics prediction | Need to predict future object states or plan manipulation | MRO-GWM [2606.01950]: Gaussian grouping (OP2GS/Gaga) → canonical representation → spatio-temporal transformer |
| Reconstruction from views | Observations available, standard 3DGS pipeline | Standard pipeline: Segmentation → Geometry → Material → Articulation |
Key Method Cross-References
| Agent Capability | Primary Method | Backup Method | Key Metric |
|---|---|---|---|
| Scene segmentation | OP2GS [2605.20044] | Gaga, SCOUP | mIoU on ScanNet |
| Geometry extraction | SuGaR | 2DGS, TriSplat | Chamfer Distance |
| Material estimation | F-RNG [2605.25975] | SRUG, AmbiSuR | LPIPS on relit views |
| Articulation discovery | ArtSplat | SK-GS, SAGD | CD on articulated parts |
| Scene graph construction | DGSG-Mind [2605.29879] | — | 3DVG accuracy |
| Feed-forward head/avatar | HeadsUp [2605.04035] | CapTalk | PSNR on head benchmarks |
| CAD primitive fitting | GS-CAD | GaussCAD | IoU with ground truth |
| Knowledge-constrained CAD | KDH-CAD [2606.01702] | — | 92.6% accuracy (250 samples) |
| Mid-surface extraction | MidSurfNet [2606.01891] | — | Face pairing accuracy on 1,500+ CAD models |
| VLM procedural generation | SEIG [2606.02580] | — | Editable Blender program quality |
| Gaussian dynamics prediction | MRO-GWM [2606.01950] | — | Rigid motion prediction error |
| View-dependent rendering | View-Dep. Kernels [2605.25426] | DP-GES | PSNR/LPIPS on specular |
Bug Patterns Specific to Spatial Agent
| # | Pattern | Symptom | Fix |
|---|---|---|---|
| SA-1 | Part boundary bleeding in segmentation | Color/feature mixing at object boundaries; Gaussians assigned to wrong part | Use part-aware opacity modulation; apply bilateral filtering on part assignments near boundaries |
| SA-2 | Geometry-mesh topology mismatch | Extracted mesh has non-manifold edges or self-intersections; CAD operations fail | Pre-filter with meshcleaning; validate manifoldness before B-rep construction; use PyMeshLab for repair |
| SA-3 | SH coefficient misinterpretation as material | Confusing view-dependent color (SH coefficients) with intrinsic material properties | Decompose SH into intrinsic (degree 0) and view-dependent (degree 1-3) components; only use degree 0 for material inference |
| SA-4 | Spatial Query Mutex Deadlock in Multi-Agent Scene Editing | Agent hangs indefinitely when two concurrent spatial queries target overlapping Gaussian groups | Replace std::mutex with std::recursive_mutex in SceneGraph::query(); or adopt readers-writer lock where read-only queries share access |
| SA-5 | Stale Gaussian Indices After Densification in CAD-in-the-Loop Pipeline | CAD extraction produces distorted geometry (mirrored faces, collapsed edges) after 3DGS densification step | Register CAD module as densification observer; on density control step, invalidate cached index maps and trigger re-extraction of Gaussian→mesh attribute mapping |
| SA-6 | RAF Representation Round-Trip Drift | When using RAF-style visual→physics→visual round-trip, accumulated quantization error in geometry causes rendered images to shift progressively after each simulation step; no re-projection correction in place | Add re-projection correction after each physics step; quantize at physics resolution then upsample with error feedback; track drift metric per round-trip |
| SA-7 | FreeArtGS Free-Motion Drift | Under free-moving articulated object reconstruction, Gaussian positions drift without ground-plane constraint; articulation joints accumulate position error over long sequences | Enforce ground-plane constraint as regularization loss; add joint-anchor drift penalty; periodic re-alignment via reference frame tracking |
| SA-8 | PARTICULATE Mesh-to-Articulation Inconsistency | Feed-forward articulation prediction from mesh yields inconsistent joint axes when mesh has non-manifold edges; no topological validation before articulation fitting | Validate mesh manifoldness before articulation fitting; reject non-manifold edge regions from joint estimation; use topological cleanup (PyMeshLab) as preprocessing step |
Rules
- Always segment first: Never reason about unsegmented 3DGS scenes; segment into objects before spatial reasoning
- Acknowledge uncertainty: 3DGS segmentation quality depends on training view coverage; report confidence scores
- CAD precision context: Manufacturing requires sub-mm accuracy; research-only applications tolerate higher error
- Respect representation limits: 3DGS cannot directly represent sharp CAD edges; always use mesh/CAD conversion for precision geometry
- Cite specific methods: When recommending a method, cite the arXiv ID from our knowledge base
Part of Awesome-Gaussian-Skills
Red Lines
The following are categorical prohibitions. Violating any of these invalidates the output:
- No invented data: Never fabricate spatial relationships, geometry properties, or method characteristics not in the loaded reference files. If a value is not found, write "data not available" or "N/A".
- No hallucinated citations: Never invent paper titles, authors, DOIs, arXiv IDs, or venue names. Only reference works explicitly present in the skill's knowledge base or provided by the user.
- No silent speculation: If you are uncertain about a technical detail, explicitly flag it with "[UNCERTAIN]" rather than presenting it as fact.
- No method misattribution: Do not assign features, results, or mechanisms from one method to another. Each method's data is specific to that method.
- No oversimplified comparisons: Do not reduce multi-dimensional spatial reasoning trade-offs to a single judgment without context.
Related Skills
- 3dgs-mcp-renderer — MCP rendering protocol (use for real-time spatial rendering)
- 3dgs-articulated-reasoner — Articulated object reasoning (use for part-level interaction)
- 3dgs-engineering-guide — Deployment guidance (use for spatial agent deployment)
- 3dgs-method-compare — Method comparison (use for selecting spatial representation methods)
Guardrail: Do Not Apply From Memory
Do NOT try to apply the logic, method data, bug patterns, or technical details described in this skill from memory. Always read the SKILL.md and referenced files from disk before producing any output. The knowledge base is updated frequently; stale memory may produce outdated, inaccurate, or fabricated results.
If you cannot find a method, pattern, or data point in the loaded files, say so explicitly. Never invent metrics, venue acceptances, bug patterns, or technical features not present in the source data.