Charting Intelligence & Data-to-Viz Pipeline Engineering
Enable any agent to (1) instantly identify the exact chart needed from raw data, (2) generate the precise path of queries/transforms to materialize that chart, and (3) evaluate and choose the optimal charting library/stack based on performance, scale, and interactivity requirements.
This is not "just call a library" — it is full-stack visualization strategy.
1. Core Decision Framework — Choosing the Chart That Fits the Data AND the Story
Before any code runs, answer these questions in order:
What is the goal of the viewer?
| Goal |
Chart Type |
| Compare values |
Bar/Column (grouped or stacked) |
| Show trend over time |
Line or Area |
| Show distribution / spread |
Histogram, Box Plot, Violin |
| Show relationship / correlation |
Scatter, Bubble, Heatmap |
| Show composition / parts-of-whole |
Stacked Bar or Area (never pie if >5 slices) |
| Show hierarchy / flow |
Treemap, Sunburst, Sankey |
| Show geographic pattern |
Choropleth or Symbol Map |
How many variables and what types?
| Variables |
Chart |
| 1 numeric, unordered |
Histogram / Density |
| 1 numeric + time |
Line |
| 1 categorical + 1 numeric |
Bar |
| 2 numeric |
Scatter |
| 1 categorical + time series |
Grouped or Stacked Line/Area |
| Many-to-many relationships |
Heatmap or Parallel Coordinates |
Audience & Context Check
| Audience |
Approach |
| Executive dashboard |
Big numbers + simple bars/lines, zero clutter |
| Analyst/explorer |
Interactive tooltips, zoom, hover details, multiple linked views |
| Mobile |
Horizontal bars, large text, minimal colors |
| Accessibility |
High contrast, patterns instead of color-only, alt-text descriptions |
Rule of Thumb Table
| Data Situation |
Best Chart (first choice) |
Avoid |
| >5 categories |
Bar (horizontal) |
Pie |
| Time series >20 points |
Line |
Column |
| Correlation between 2 measures |
Scatter |
Line (unless ordered) |
| Parts of whole >5 slices |
Stacked Bar or Treemap |
Pie/Donut |
| Outliers or distribution shape |
Box + Violin |
Bar |
| Flow between stages |
Sankey |
Anything else |
2. The Data Pipeline Engine
Most databases do NOT have the exact aggregation ready. Auto-generate the full pipeline:
Step A — Inventory
- Scan schema or sample 100 rows — detect column types, null rates, cardinality
- Flag missing aggregations (e.g., "no daily_sales_by_region view exists")
Step B — Required Transformations
Auto-generate SQL or pandas code for:
- Joins needed?
- GROUP BY + SUM/AVG/COUNT?
- Window functions for running totals or YoY?
- Binning (e.g., age into decades)?
- Pivot/unpivot?
- Outlier flagging or imputation?
Step C — Materialization Strategy
| Scale |
Strategy |
| One-off (<10k rows) |
Run query on-the-fly |
| Medium |
Create materialized view or cached table |
| Large/Real-time |
Pre-aggregate in Spark/DuckDB, incremental refresh |
| Extreme |
Stream + windowed aggregates (Flink/Kafka) |
Step D — Validation
- Run a tiny sample query first — confirm the shape matches the chosen chart type
- If not, loop back and adjust aggregation
Example
User says "show monthly revenue by product category":
"I need: LEFT JOIN orders -> products -> categories; GROUP BY month, category; SUM(revenue). No view exists -> I will create temp table or run inline. Chart type: Stacked Area. Library recommendation below."
3. Library Selection Matrix
Always output the performance trade-off and recommended stack.
| Scale / Requirement |
Recommended Library |
Why |
Fallback |
| <10k points, simple web dashboard |
Chart.js or Recharts |
<10 ms render, ~60 KB bundle |
N/A |
| 10k-500k points, interactive |
Apache ECharts or Plotly.js |
Canvas + WebGL, 60 fps on 100k points |
D3 (slower) |
| 500k-10M+ points, real-time |
LightningChart or Highcharts Stock + WebGL |
GPU accelerated, <50 ms at 5M points |
Anything SVG-based fails |
| Python backend + web |
Plotly Dash or Bokeh |
Server-side render + client streaming |
Matplotlib (static only) |
| Python notebook exploration |
Seaborn + Plotly |
Instant, beautiful defaults |
-- |
| Extremely large / streaming |
DuckDB + Observable Plot or Perspective |
In-memory columnar, sub-second on billions |
-- |
| No JavaScript (PDF reports) |
Matplotlib + WeasyPrint or ReportLab |
Pure Python, vector output |
-- |
Optimization Rules (apply automatically)
- Downsample for overview, show full detail on zoom (ECharts built-in)
- Use Canvas instead of SVG above ~5k elements
- Pre-aggregate at DB level whenever possible (biggest single win)
- Lazy load charts below the fold
- Bundle size: tree-shake everything except the one chart type you need
- GPU vs CPU: if >100k points and user needs pan/zoom, force WebGL path
4. Full Workflow
- Parse intent — identify required chart type from user request
- Schema scan — detect column types, cardinality, row estimates
- Decision framework — output chart recommendation + rationale
- Generate transforms — exact SQL/pandas/transform code needed
- Choose library — select by performance tier based on row estimate
- Emit deliverables:
- Chart spec (JSON for the library or React component)
- SQL/transform script
- Performance warning or confirmation
- Accessibility note + alt-text template
5. Advanced Capabilities
- "Show me what I should be charting but aren't" — auto-correlation scan + suggested visuals
- "Optimize this dashboard for 10x speed" — rewrite query + switch library
- "Make this mobile-first" — auto-switch to horizontal bars + simplify
- Color-blind & accessibility mode — toggle patterns, high contrast
- Export — SVG/PNG/PDF with embedded data table
1---2name: charting3description: Data visualization and charting. Use for "create a chart", "visualize this data", "build a dashboard", "plot this", "graph these metrics", "make a bar chart", "build a heatmap", or help choosing a chart type, picking a charting library by data scale, or building the pipeline from raw database state to rendered visualization.4---56# Charting Intelligence & Data-to-Viz Pipeline Engineering78Enable any agent to (1) instantly identify the exact chart needed from raw data, (2) generate the precise path of queries/transforms to materialize that chart, and (3) evaluate and choose the optimal charting library/stack based on performance, scale, and interactivity requirements.910This is not "just call a library" — it is full-stack visualization strategy.1112## 1. Core Decision Framework — Choosing the Chart That Fits the Data AND the Story1314Before any code runs, answer these questions in order:1516### What is the goal of the viewer?1718| Goal | Chart Type |19|------|-----------|20| Compare values | Bar/Column (grouped or stacked) |21| Show trend over time | Line or Area |22| Show distribution / spread | Histogram, Box Plot, Violin |23| Show relationship / correlation | Scatter, Bubble, Heatmap |24| Show composition / parts-of-whole | Stacked Bar or Area (never pie if >5 slices) |25| Show hierarchy / flow | Treemap, Sunburst, Sankey |26| Show geographic pattern | Choropleth or Symbol Map |2728### How many variables and what types?2930| Variables | Chart |31|-----------|-------|32| 1 numeric, unordered | Histogram / Density |33| 1 numeric + time | Line |34| 1 categorical + 1 numeric | Bar |35| 2 numeric | Scatter |36| 1 categorical + time series | Grouped or Stacked Line/Area |37| Many-to-many relationships | Heatmap or Parallel Coordinates |3839### Audience & Context Check4041| Audience | Approach |42|----------|----------|43| Executive dashboard | Big numbers + simple bars/lines, zero clutter |44| Analyst/explorer | Interactive tooltips, zoom, hover details, multiple linked views |45| Mobile | Horizontal bars, large text, minimal colors |46| Accessibility | High contrast, patterns instead of color-only, alt-text descriptions |4748### Rule of Thumb Table4950| Data Situation | Best Chart (first choice) | Avoid |51|---------------|--------------------------|-------|52| >5 categories | Bar (horizontal) | Pie |53| Time series >20 points | Line | Column |54| Correlation between 2 measures | Scatter | Line (unless ordered) |55| Parts of whole >5 slices | Stacked Bar or Treemap | Pie/Donut |56| Outliers or distribution shape | Box + Violin | Bar |57| Flow between stages | Sankey | Anything else |5859## 2. The Data Pipeline Engine6061Most databases do NOT have the exact aggregation ready. Auto-generate the full pipeline:6263### Step A — Inventory6465- Scan schema or sample 100 rows — detect column types, null rates, cardinality66- Flag missing aggregations (e.g., "no daily_sales_by_region view exists")6768### Step B — Required Transformations6970Auto-generate SQL or pandas code for:71- Joins needed?72- GROUP BY + SUM/AVG/COUNT?73- Window functions for running totals or YoY?74- Binning (e.g., age into decades)?75- Pivot/unpivot?76- Outlier flagging or imputation?7778### Step C — Materialization Strategy7980| Scale | Strategy |81|-------|----------|82| One-off (<10k rows) | Run query on-the-fly |83| Medium | Create materialized view or cached table |84| Large/Real-time | Pre-aggregate in Spark/DuckDB, incremental refresh |85| Extreme | Stream + windowed aggregates (Flink/Kafka) |8687### Step D — Validation8889- Run a tiny sample query first — confirm the shape matches the chosen chart type90- If not, loop back and adjust aggregation9192### Example9394User says "show monthly revenue by product category":9596> "I need: LEFT JOIN orders -> products -> categories; GROUP BY month, category; SUM(revenue). No view exists -> I will create temp table or run inline. Chart type: Stacked Area. Library recommendation below."9798## 3. Library Selection Matrix99100Always output the performance trade-off and recommended stack.101102| Scale / Requirement | Recommended Library | Why | Fallback |103|--------------------|-------------------|-----|----------|104| <10k points, simple web dashboard | Chart.js or Recharts | <10 ms render, ~60 KB bundle | N/A |105| 10k-500k points, interactive | Apache ECharts or Plotly.js | Canvas + WebGL, 60 fps on 100k points | D3 (slower) |106| 500k-10M+ points, real-time | LightningChart or Highcharts Stock + WebGL | GPU accelerated, <50 ms at 5M points | Anything SVG-based fails |107| Python backend + web | Plotly Dash or Bokeh | Server-side render + client streaming | Matplotlib (static only) |108| Python notebook exploration | Seaborn + Plotly | Instant, beautiful defaults | -- |109| Extremely large / streaming | DuckDB + Observable Plot or Perspective | In-memory columnar, sub-second on billions | -- |110| No JavaScript (PDF reports) | Matplotlib + WeasyPrint or ReportLab | Pure Python, vector output | -- |111112### Optimization Rules (apply automatically)113114- **Downsample** for overview, show full detail on zoom (ECharts built-in)115- **Use Canvas instead of SVG** above ~5k elements116- **Pre-aggregate at DB level** whenever possible (biggest single win)117- **Lazy load** charts below the fold118- **Bundle size**: tree-shake everything except the one chart type you need119- **GPU vs CPU**: if >100k points and user needs pan/zoom, force WebGL path120121## 4. Full Workflow1221231. **Parse intent** — identify required chart type from user request1242. **Schema scan** — detect column types, cardinality, row estimates1253. **Decision framework** — output chart recommendation + rationale1264. **Generate transforms** — exact SQL/pandas/transform code needed1275. **Choose library** — select by performance tier based on row estimate1286. **Emit deliverables**:129 - Chart spec (JSON for the library or React component)130 - SQL/transform script131 - Performance warning or confirmation132 - Accessibility note + alt-text template133134## 5. Advanced Capabilities135136- **"Show me what I should be charting but aren't"** — auto-correlation scan + suggested visuals137- **"Optimize this dashboard for 10x speed"** — rewrite query + switch library138- **"Make this mobile-first"** — auto-switch to horizontal bars + simplify139- **Color-blind & accessibility mode** — toggle patterns, high contrast140- **Export** — SVG/PNG/PDF with embedded data table
b-open-io