Create New Module
Use this when adding a new integration like ClickHouse, Prometheus, Loki, Dora, or Ethnode.
The architecture is module + server operations, not plugins + bespoke proxy endpoints.
Files To Create
Create the module folder:
modules/{name}/
├── config.go
├── module.go
├── examples.go
├── examples.yaml
└── python/{name}.py
Add these when needed:
resources.go for custom MCP resources
- helper files for schemas, discovery, or API clients
pkg/proxy/handlers/{name}_operations.go
pkg/proxy/handlers/{name}_operations_test.go
Copy From
modules/prometheus/ or modules/loki/ for simple JSON passthrough APIs
modules/clickhouse/ for streamed table results and datasource discovery
modules/dora/ or modules/ethnode/ for external HTTP APIs with curated helpers
Required Wiring
pkg/app/app.go
Add reg.Add({name}module.New()) in buildModuleRegistry().
pkg/server/operations_<domain>.go
Add new server-owned operation handling for the module.
pkg/server/operations_dispatch.go
Wire the module's handler into dispatch.
pkg/proxy/server_config.go
Add proxy server config translation only if the module needs proxy-held credentials or typed proxy config.
sandbox/ethpandaops/ethpandaops/__init__.py
Add the lazy import if the module exposes a Python module.
sandbox/Dockerfile
Copy modules/{name}/python/{name}.py into the installed ethpandaops package.
Architecture Rules
- Do not add a new MCP tool. MCP stays limited to platform primitives.
- Module semantics live in Go.
- Python wrappers stay thin and go through
sandbox/ethpandaops/ethpandaops/_runtime.py.
- Default remote execution path is
POST /api/v1/operations/{module.operation}.
- Server operation handlers own validation, defaults, routing, and error mapping.
- Upstream-backed bulk data must stay passthrough when possible.
Do not materialize large tables into row-object JSON in the server.
- Small synthetic/object operations can still return the JSON envelope.
- Credentials never go into the sandbox. Only metadata and server runtime tokens do.
Module Contract
Implement module.Module from pkg/module/module.go and only the optional capability interfaces you need:
- base lifecycle:
Name, Init, ApplyDefaults, Validate, Start, Stop
- optional capabilities:
SandboxEnvProvider, DatasourceInfoProvider, ExamplesProvider, PythonAPIDocsProvider, ResourceProvider
Python Rules
- Keep Python ergonomic, not semantic.
clickhouse.query(...) style wrappers should call _runtime helpers and parse only at the edge.
- If the operation returns bulk tables, prefer streaming formats like TSV instead of normalized JSON rows.
- If the operation already returns useful upstream JSON, return that JSON rather than reshaping it in Python.
CLI Rules
- Module-specific CLI commands should be thin adapters over server operations.
- Do not duplicate validation/default logic already implemented in the server operation handler.
- Pretty-printing belongs in CLI; semantic behavior does not.
Checklist
1---2name: create-module3description: "Add a new datasource module to ethpandaops/panda. Triggers on: add module, new module, create module, add plugin, new plugin, create plugin, add datasource."4argument-hint: <module-name>5disable-model-invocation: true6metadata:7 source: https://skillmd.com/skills/ethpandaops/create-module8 internal: true9---1011# Create New Module1213Use this when adding a new integration like ClickHouse, Prometheus, Loki, Dora, or Ethnode.14The architecture is **module + server operations**, not plugins + bespoke proxy endpoints.1516## Files To Create1718Create the module folder:1920```text21modules/{name}/22├── config.go23├── module.go24├── examples.go25├── examples.yaml26└── python/{name}.py27```2829Add these when needed:30- `resources.go` for custom MCP resources31- helper files for schemas, discovery, or API clients32- `pkg/proxy/handlers/{name}_operations.go`33- `pkg/proxy/handlers/{name}_operations_test.go`3435## Copy From3637- `modules/prometheus/` or `modules/loki/` for simple JSON passthrough APIs38- `modules/clickhouse/` for streamed table results and datasource discovery39- `modules/dora/` or `modules/ethnode/` for external HTTP APIs with curated helpers4041## Required Wiring42431. `pkg/app/app.go`44 Add `reg.Add({name}module.New())` in `buildModuleRegistry()`.452. `pkg/server/operations_<domain>.go`46 Add new server-owned operation handling for the module.473. `pkg/server/operations_dispatch.go`48 Wire the module's handler into dispatch.494. `pkg/proxy/server_config.go`50 Add proxy server config translation only if the module needs proxy-held credentials or typed proxy config.515. `sandbox/ethpandaops/ethpandaops/__init__.py`52 Add the lazy import if the module exposes a Python module.536. `sandbox/Dockerfile`54 Copy `modules/{name}/python/{name}.py` into the installed `ethpandaops` package.5556## Architecture Rules5758- Do **not** add a new MCP tool. MCP stays limited to platform primitives.59- Module semantics live in Go.60- Python wrappers stay thin and go through `sandbox/ethpandaops/ethpandaops/_runtime.py`.61- Default remote execution path is `POST /api/v1/operations/{module.operation}`.62- Server operation handlers own validation, defaults, routing, and error mapping.63- Upstream-backed bulk data must stay passthrough when possible.64 Do not materialize large tables into row-object JSON in the server.65- Small synthetic/object operations can still return the JSON envelope.66- Credentials never go into the sandbox. Only metadata and server runtime tokens do.6768## Module Contract6970Implement `module.Module` from `pkg/module/module.go` and only the optional capability interfaces you need:71- base lifecycle: `Name`, `Init`, `ApplyDefaults`, `Validate`, `Start`, `Stop`72- optional capabilities: `SandboxEnvProvider`, `DatasourceInfoProvider`, `ExamplesProvider`, `PythonAPIDocsProvider`, `ResourceProvider`7374## Python Rules7576- Keep Python ergonomic, not semantic.77- `clickhouse.query(...)` style wrappers should call `_runtime` helpers and parse only at the edge.78- If the operation returns bulk tables, prefer streaming formats like TSV instead of normalized JSON rows.79- If the operation already returns useful upstream JSON, return that JSON rather than reshaping it in Python.8081## CLI Rules8283- Module-specific CLI commands should be thin adapters over server operations.84- Do not duplicate validation/default logic already implemented in the server operation handler.85- Pretty-printing belongs in CLI; semantic behavior does not.8687## Checklist8889- [ ] New module implements `module.Module`90- [ ] Examples and Python API docs are added91- [ ] Server operation handler is wired into `pkg/server/operations_dispatch.go`92- [ ] No new MCP tool was added93- [ ] Python module is thin and copied via `sandbox/Dockerfile`94- [ ] CLI, if added, calls operations instead of bespoke transport logic95- [ ] `go test ./...` passes96- [ ] `python3 -m py_compile ...` passes for touched Python files97- [ ] `make docker-sandbox` builds if sandbox files changed