Drone CV Expert
Expert in robotics, drone systems, and computer vision for autonomous aerial platforms.
Decision Tree: When to Use This Skill
User mentions drones or UAVs?
├─ YES → Is it about inspection/detection of specific things (fire, roof damage, thermal)?
│ ├─ YES → Use drone-inspection-specialist
│ └─ NO → Is it about flight control, navigation, or general CV?
│ ├─ YES → Use THIS SKILL (drone-cv-expert)
│ └─ NO → Is it about GPU rendering/shaders?
│ ├─ YES → Use metal-shader-expert
│ └─ NO → Use THIS SKILL as default drone skill
└─ NO → Is it general object detection without drone context?
├─ YES → Use clip-aware-embeddings or other CV skill
└─ NO → Probably not a drone question
Core Competencies
Flight Control & Navigation
- PID Tuning: Position, velocity, attitude control loops
- SLAM: ORB-SLAM, LSD-SLAM, visual-inertial odometry (VIO)
- Path Planning: A*, RRT, RRT*, Dijkstra, potential fields
- Sensor Fusion: EKF, UKF, complementary filters
- GPS-Denied Navigation: AprilTags, visual odometry, LiDAR SLAM
Computer Vision
- Object Detection: YOLO (v5/v8/v10), EfficientDet, SSD
- Tracking: ByteTrack, DeepSORT, SORT, optical flow
- Edge Deployment: TensorRT, ONNX, OpenVINO optimization
- 3D Vision: Stereo depth, point clouds, structure-from-motion
Hardware Integration
- Flight Controllers: Pixhawk, Ardupilot, PX4, DJI
- Protocols: MAVLink, DroneKit, MAVSDK
- Edge Compute: Jetson (Nano/Xavier/Orin), Coral TPU
- Sensors: IMU, GPS, barometer, LiDAR, depth cameras
Anti-Patterns to Avoid
1. "Simulation-Only Syndrome"
Wrong: Testing only in Gazebo/AirSim, then deploying directly to real drone.
Right: Simulation → Bench test → Tethered flight → Controlled environment → Field.
2. "EKF Overkill"
Wrong: Using Extended Kalman Filter when complementary filter suffices.
Right: Match filter complexity to requirements:
- Complementary filter: Basic stabilization, attitude only
- EKF: Multi-sensor fusion, GPS+IMU+baro
- UKF: Highly nonlinear systems, aggressive maneuvers
3. "Max Resolution Assumption"
Wrong: Processing 4K frames at 30fps expecting real-time performance.
Right: Resolution trade-offs by altitude/speed:
| Altitude |
Speed |
Resolution |
FPS |
Rationale |
| <30m |
Slow |
1920x1080 |
30 |
Detail needed |
| 30-100m |
Medium |
1280x720 |
30 |
Balance |
| >100m |
Fast |
640x480 |
60 |
Speed priority |
4. "Single-Thread Processing"
Wrong: Sequential detect → track → control in one loop.
Right: Pipeline parallelism:
Thread 1: Camera capture (async)
Thread 2: Object detection (GPU)
Thread 3: Tracking + state estimation
Thread 4: Control commands
5. "GPS Trust"
Wrong: Assuming GPS is always accurate and available.
Right: Multi-source position estimation:
- GPS: 2-5m accuracy outdoor, unavailable indoor
- Visual odometry: 0.1-1% drift, lighting dependent
- AprilTags: cm-level accuracy where deployed
- IMU: Short-term only, drift accumulates
6. "One Model Fits All"
Wrong: Using same YOLO model for all scenarios.
Right: Model selection by constraint:
| Constraint |
Model |
Notes |
| Latency critical |
YOLOv8n |
6ms inference |
| Balanced |
YOLOv8s |
15ms, better accuracy |
| Accuracy first |
YOLOv8x |
50ms, highest mAP |
| Edge device |
YOLOv8n + TensorRT |
3ms on Jetson |
Problem-Solving Framework
1. Constraint Analysis
- Compute: What hardware? (Jetson Nano = ~5 TOPS, Xavier = 32 TOPS)
- Power: Battery capacity? Flight time impact?
- Latency: Control loop rate? Detection response time?
- Weight: Payload capacity? Center of gravity?
- Environment: Indoor/outdoor? GPS available? Lighting conditions?
2. Algorithm Selection Matrix
| Problem |
Classical Approach |
Deep Learning |
When to Use Each |
| Feature tracking |
KLT optical flow |
FlowNet |
Classical: Real-time, limited compute. DL: Robust, more compute |
| Object detection |
HOG+SVM |
YOLO/SSD |
Classical: Simple objects, no GPU. DL: Complex, GPU available |
| SLAM |
ORB-SLAM |
DROID-SLAM |
Classical: Mature, debuggable. DL: Better in challenging scenes |
| Path planning |
A*, RRT |
RL-based |
Classical: Known environments. DL: Complex, dynamic |
3. Safety Checklist
Quick Reference Tables
MAVLink Message Types
| Message |
Purpose |
Frequency |
| HEARTBEAT |
Connection alive |
1 Hz |
| ATTITUDE |
Roll/pitch/yaw |
10-100 Hz |
| LOCAL_POSITION_NED |
Position |
10-50 Hz |
| GPS_RAW_INT |
Raw GPS |
1-10 Hz |
| SET_POSITION_TARGET |
Commands |
As needed |
Kalman Filter Tuning
| Matrix |
High Values |
Low Values |
| Q (process noise) |
Trust measurements more |
Trust model more |
| R (measurement noise) |
Trust model more |
Trust measurements more |
| P (initial covariance) |
Uncertain initial state |
Confident initial state |
Common Coordinate Frames
| Frame |
Origin |
Axes |
Use |
| NED |
Takeoff point |
North-East-Down |
Navigation |
| ENU |
Takeoff point |
East-North-Up |
ROS standard |
| Body |
Drone CG |
Forward-Right-Down |
Control |
| Camera |
Lens center |
Right-Down-Forward |
Vision |
Reference Files
Detailed implementations in references/:
navigation-algorithms.md - SLAM, path planning, localizationsensor-fusion-ekf.md - Kalman filters, multi-sensor fusionobject-detection-tracking.md - YOLO, ByteTrack, optical flow
Simulation Tools
| Tool |
Strengths |
Weaknesses |
Best For |
| Gazebo |
ROS integration, physics |
Graphics quality |
ROS development |
| AirSim |
Photorealistic, CV-focused |
Windows-centric |
Vision algorithms |
| Webots |
Multi-robot, accessible |
Less drone-specific |
Swarm simulations |
| MATLAB/Simulink |
Control design |
Not real-time |
Controller tuning |
Emerging Technologies (2024-2025)
- Event cameras: 1μs temporal resolution, no motion blur
- Neuromorphic computing: Loihi 2 for ultra-low-power inference
- 4D Radar: Velocity + 3D position, works in all weather
- Swarm autonomy: Decentralized coordination, emergent behavior
- Foundation models: SAM, CLIP for zero-shot detection
Integration Points
- drone-inspection-specialist: Domain-specific detection (fire, damage, thermal)
- metal-shader-expert: GPU-accelerated vision processing, custom shaders
- collage-layout-expert: Report generation, visual composition
Key Principle: In drone systems, reliability trumps performance. A 95% accurate system that never crashes is better than 99% accurate that fails unpredictably. Always have fallbacks.
1---2name: drone-cv-expert3description: Expert in drone systems, computer vision, and autonomous navigation. Specializes in flight control, SLAM, object detection, sensor fusion, and path planning. Activate on "drone", "UAV", "SLAM", "visual odometry", "PID control", "MAVLink", "Pixhawk", "path planning", "A*", "RRT", "EKF", "sensor fusion", "optical flow", "ByteTrack". NOT for domain-specific inspection tasks like fire detection, roof damage assessment, or thermal analysis (use drone-inspection-specialist), GPU shader optimization (use metal-shader-expert), or general image classification without drone context (use clip-aware-embeddings).4---5
6# Drone CV Expert
7
8Expert in robotics, drone systems, and computer vision for autonomous aerial platforms.
9
10## Decision Tree: When to Use This Skill
11
12```
13User mentions drones or UAVs?
14├─ YES → Is it about inspection/detection of specific things (fire, roof damage, thermal)?
15│ ├─ YES → Use drone-inspection-specialist
16│ └─ NO → Is it about flight control, navigation, or general CV?
17│ ├─ YES → Use THIS SKILL (drone-cv-expert)
18│ └─ NO → Is it about GPU rendering/shaders?
19│ ├─ YES → Use metal-shader-expert
20│ └─ NO → Use THIS SKILL as default drone skill
21└─ NO → Is it general object detection without drone context?
22 ├─ YES → Use clip-aware-embeddings or other CV skill
23 └─ NO → Probably not a drone question
24```
25
26## Core Competencies
27
28### Flight Control & Navigation
29- **PID Tuning**: Position, velocity, attitude control loops
30- **SLAM**: ORB-SLAM, LSD-SLAM, visual-inertial odometry (VIO)
31- **Path Planning**: A*, RRT, RRT*, Dijkstra, potential fields
32- **Sensor Fusion**: EKF, UKF, complementary filters
33- **GPS-Denied Navigation**: AprilTags, visual odometry, LiDAR SLAM
34
35### Computer Vision
36- **Object Detection**: YOLO (v5/v8/v10), EfficientDet, SSD
37- **Tracking**: ByteTrack, DeepSORT, SORT, optical flow
38- **Edge Deployment**: TensorRT, ONNX, OpenVINO optimization
39- **3D Vision**: Stereo depth, point clouds, structure-from-motion
40
41### Hardware Integration
42- **Flight Controllers**: Pixhawk, Ardupilot, PX4, DJI
43- **Protocols**: MAVLink, DroneKit, MAVSDK
44- **Edge Compute**: Jetson (Nano/Xavier/Orin), Coral TPU
45- **Sensors**: IMU, GPS, barometer, LiDAR, depth cameras
46
47## Anti-Patterns to Avoid
48
49### 1. "Simulation-Only Syndrome"
50**Wrong**: Testing only in Gazebo/AirSim, then deploying directly to real drone.
51**Right**: Simulation → Bench test → Tethered flight → Controlled environment → Field.
52
53### 2. "EKF Overkill"
54**Wrong**: Using Extended Kalman Filter when complementary filter suffices.
55**Right**: Match filter complexity to requirements:
56- Complementary filter: Basic stabilization, attitude only
57- EKF: Multi-sensor fusion, GPS+IMU+baro
58- UKF: Highly nonlinear systems, aggressive maneuvers
59
60### 3. "Max Resolution Assumption"
61**Wrong**: Processing 4K frames at 30fps expecting real-time performance.
62**Right**: Resolution trade-offs by altitude/speed:
63| Altitude | Speed | Resolution | FPS | Rationale |
64|----------|-------|------------|-----|-----------|
65| <30m | Slow | 1920x1080 | 30 | Detail needed |
66| 30-100m | Medium | 1280x720 | 30 | Balance |
67| >100m | Fast | 640x480 | 60 | Speed priority |
68
69### 4. "Single-Thread Processing"
70**Wrong**: Sequential detect → track → control in one loop.
71**Right**: Pipeline parallelism:
72```
73Thread 1: Camera capture (async)
74Thread 2: Object detection (GPU)
75Thread 3: Tracking + state estimation
76Thread 4: Control commands
77```
78
79### 5. "GPS Trust"
80**Wrong**: Assuming GPS is always accurate and available.
81**Right**: Multi-source position estimation:
82- GPS: 2-5m accuracy outdoor, unavailable indoor
83- Visual odometry: 0.1-1% drift, lighting dependent
84- AprilTags: cm-level accuracy where deployed
85- IMU: Short-term only, drift accumulates
86
87### 6. "One Model Fits All"
88**Wrong**: Using same YOLO model for all scenarios.
89**Right**: Model selection by constraint:
90| Constraint | Model | Notes |
91|------------|-------|-------|
92| Latency critical | YOLOv8n | 6ms inference |
93| Balanced | YOLOv8s | 15ms, better accuracy |
94| Accuracy first | YOLOv8x | 50ms, highest mAP |
95| Edge device | YOLOv8n + TensorRT | 3ms on Jetson |
96
97## Problem-Solving Framework
98
99### 1. Constraint Analysis
100- **Compute**: What hardware? (Jetson Nano = ~5 TOPS, Xavier = 32 TOPS)
101- **Power**: Battery capacity? Flight time impact?
102- **Latency**: Control loop rate? Detection response time?
103- **Weight**: Payload capacity? Center of gravity?
104- **Environment**: Indoor/outdoor? GPS available? Lighting conditions?
105
106### 2. Algorithm Selection Matrix
107
108| Problem | Classical Approach | Deep Learning | When to Use Each |
109|---------|-------------------|---------------|------------------|
110| Feature tracking | KLT optical flow | FlowNet | Classical: Real-time, limited compute. DL: Robust, more compute |
111| Object detection | HOG+SVM | YOLO/SSD | Classical: Simple objects, no GPU. DL: Complex, GPU available |
112| SLAM | ORB-SLAM | DROID-SLAM | Classical: Mature, debuggable. DL: Better in challenging scenes |
113| Path planning | A*, RRT | RL-based | Classical: Known environments. DL: Complex, dynamic |
114
115### 3. Safety Checklist
116- [ ] Kill switch tested and accessible
117- [ ] Geofence configured
118- [ ] Return-to-home altitude set
119- [ ] Low battery action defined
120- [ ] Signal loss action defined
121- [ ] Propeller guards (if applicable)
122- [ ] Pre-flight sensor calibration
123- [ ] Weather conditions checked
124
125## Quick Reference Tables
126
127### MAVLink Message Types
128| Message | Purpose | Frequency |
129|---------|---------|-----------|
130| HEARTBEAT | Connection alive | 1 Hz |
131| ATTITUDE | Roll/pitch/yaw | 10-100 Hz |
132| LOCAL_POSITION_NED | Position | 10-50 Hz |
133| GPS_RAW_INT | Raw GPS | 1-10 Hz |
134| SET_POSITION_TARGET | Commands | As needed |
135
136### Kalman Filter Tuning
137| Matrix | High Values | Low Values |
138|--------|-------------|------------|
139| Q (process noise) | Trust measurements more | Trust model more |
140| R (measurement noise) | Trust model more | Trust measurements more |
141| P (initial covariance) | Uncertain initial state | Confident initial state |
142
143### Common Coordinate Frames
144| Frame | Origin | Axes | Use |
145|-------|--------|------|-----|
146| NED | Takeoff point | North-East-Down | Navigation |
147| ENU | Takeoff point | East-North-Up | ROS standard |
148| Body | Drone CG | Forward-Right-Down | Control |
149| Camera | Lens center | Right-Down-Forward | Vision |
150
151## Reference Files
152
153Detailed implementations in `references/`:
154- `navigation-algorithms.md` - SLAM, path planning, localization
155- `sensor-fusion-ekf.md` - Kalman filters, multi-sensor fusion
156- `object-detection-tracking.md` - YOLO, ByteTrack, optical flow
157
158## Simulation Tools
159
160| Tool | Strengths | Weaknesses | Best For |
161|------|-----------|------------|----------|
162| Gazebo | ROS integration, physics | Graphics quality | ROS development |
163| AirSim | Photorealistic, CV-focused | Windows-centric | Vision algorithms |
164| Webots | Multi-robot, accessible | Less drone-specific | Swarm simulations |
165| MATLAB/Simulink | Control design | Not real-time | Controller tuning |
166
167## Emerging Technologies (2024-2025)
168
169- **Event cameras**: 1μs temporal resolution, no motion blur
170- **Neuromorphic computing**: Loihi 2 for ultra-low-power inference
171- **4D Radar**: Velocity + 3D position, works in all weather
172- **Swarm autonomy**: Decentralized coordination, emergent behavior
173- **Foundation models**: SAM, CLIP for zero-shot detection
174
175## Integration Points
176
177- **drone-inspection-specialist**: Domain-specific detection (fire, damage, thermal)
178- **metal-shader-expert**: GPU-accelerated vision processing, custom shaders
179- **collage-layout-expert**: Report generation, visual composition
180
181---
182
183**Key Principle**: In drone systems, reliability trumps performance. A 95% accurate system that never crashes is better than 99% accurate that fails unpredictably. Always have fallbacks.
majiayu000