Development Guide

Development Guide

This guide is for contributors who want to develop M4 locally.

Setup

Clone and install

git clone https://github.com/hannesill/m4.git
cd m4
uv venv
uv sync

Initialize test data

uv run m4 init mimic-iv-demo

CLI Commands

Dataset Management

# Initialize a dataset (downloads demo data if needed)
uv run m4 init mimic-iv-demo

# Materialize derived concept tables (MIMIC-IV only)
# Requires a database initialized with current M4 schema mapping.
# If you get a "Required schemas not found" error, reinitialize first:
#   uv run m4 init mimic-iv --force
uv run m4 init-derived mimic-iv

# List available derived tables without materializing
uv run m4 init-derived mimic-iv --list

# Switch active dataset
uv run m4 use mimic-iv

# Show active dataset status (detailed view)
uv run m4 status

# List all datasets (compact table)
uv run m4 status --all

# Show per-table derived materialization status
uv run m4 status --derived

MCP Client Configuration

# Auto-configure Claude Desktop
uv run m4 config claude

# Generate config for other clients
uv run m4 config --quick

Development Commands

# Run all tests
uv run pytest -v

# Run specific test file
uv run pytest tests/test_mcp_server.py -v

# Run tests matching pattern
uv run pytest -k "test_name" -v

# Lint and format
uv run pre-commit run --all-files

# Lint only
uv run ruff check src/

# Format only
uv run ruff format src/

MCP Configuration for Development

Point your MCP client to your local development environment:

{
  "mcpServers": {
    "m4": {
      "command": "/absolute/path/to/m4/.venv/bin/python",
      "args": ["-m", "m4.mcp_server"],
      "cwd": "/absolute/path/to/m4"
    }
  }
}

The active backend is configured via m4 backend duckdb (or bigquery), not through the MCP env block.

Architecture Overview

M4 has three main layers:

MCP Layer (mcp_server.py)
    │
    ├── Exposes tools via Model Context Protocol
    └── Thin adapter over core functionality

Core Layer (src/m4/core/)
    │
    ├── datasets.py    - Dataset definitions and modalities
    ├── tools/         - Tool implementations (tabular, notes, management)
    ├── backends/      - Database backends (DuckDB, BigQuery)
    └── derived/       - Derived concept tables (vendored mimic-code SQL)

Infrastructure Layer
    │
    ├── data_io.py     - Download, convert, initialize databases
    ├── cli.py         - Command-line interface
    └── config.py      - Configuration management

Modality-Based Tool System

Tools declare required modalities to specify which data types they need:

class ExecuteQueryTool:
    required_modalities = frozenset({Modality.TABULAR})

The ToolSelector automatically filters tools based on the active dataset's modalities. If a dataset lacks a required modality, the tool returns a helpful error message instead of failing silently.

Backend Abstraction

The Backend protocol defines the interface for query execution:

class Backend(Protocol):
    def execute_query(self, sql: str, dataset: DatasetDefinition) -> QueryResult: ...
    def get_table_list(self, dataset: DatasetDefinition) -> list[str]: ...

Implementations:

• DuckDBBackend - Local Parquet files via DuckDB views
• BigQueryBackend - Google Cloud BigQuery

Adding a New Tool

M4 uses a protocol-based design (structural typing). Tools don't inherit from a base class - they simply implement the required interface.

1. Create the tool class in src/m4/core/tools/:

from dataclasses import dataclass
from m4.core.datasets import DatasetDefinition, Modality
from m4.core.tools.base import ToolInput, ToolOutput

# Define input parameters
@dataclass
class MyNewToolInput(ToolInput):
    param1: str
    limit: int = 10

# Define tool class (no inheritance needed!)
class MyNewTool:
    """Tool description for documentation."""

    name = "my_new_tool"
    description = "Description shown to LLMs"
    input_model = MyNewToolInput
    output_model = ToolOutput

    # Modality constraints (use frozenset!)
    required_modalities: frozenset[Modality] = frozenset({Modality.TABULAR})
    supported_datasets: frozenset[str] | None = None  # None = all compatible

    def invoke(
        self, dataset: DatasetDefinition, params: MyNewToolInput
    ) -> ToolOutput:
        """Execute the tool."""
        # Implementation here
        return ToolOutput(result="Success")

    def is_compatible(self, dataset: DatasetDefinition) -> bool:
        """Check if tool works with this dataset."""
        if self.supported_datasets and dataset.name not in self.supported_datasets:
            return False
        if not self.required_modalities.issubset(dataset.modalities):
            return False
        return True

2. Register it in src/m4/core/tools/__init__.py:

from .my_module import MyNewTool

def init_tools():
    ToolRegistry.register(MyNewTool())

3. Add the MCP handler in mcp_server.py:

@mcp.tool()
@require_oauth2
def my_new_tool(param1: str, limit: int = 10) -> str:
    dataset = DatasetRegistry.get_active()
    result = _tool_selector.check_compatibility("my_new_tool", dataset)
    if not result.compatible:
        return result.error_message
    tool = ToolRegistry.get("my_new_tool")
    return tool.invoke(dataset, MyNewToolInput(param1=param1, limit=limit)).result

Code Style

• Formatter: Ruff (line-length 88)
• Type hints: Required on all functions
• Docstrings: Google style on public APIs
• Tests: pytest with asyncio_mode = "auto"

Testing

Tests mirror the src/m4/ structure:

tests/
├── test_mcp_server.py
├── core/
│   ├── test_datasets.py
│   ├── tools/
│   │   └── test_tabular.py
│   └── backends/
│       └── test_duckdb.py

Run the full test suite before submitting PRs:

uv run pre-commit run --all-files

Updating Vendored Derived SQL

The derived table SQL in src/m4/core/derived/builtins/mimic_iv/ is vendored from the mimic-code repository. When mimic-code releases updated SQL (e.g., bug fixes or new concept tables), follow these steps to update:

1. Check upstream changes: Review the mimic-code repository for changes to the mimic-iv/concepts_duckdb/ directory.

2. Copy updated SQL files: Replace the corresponding files under src/m4/core/derived/builtins/mimic_iv/. Preserve the existing directory structure (score/, sepsis/, medication/, etc.).

3. Update the orchestrator: If new tables were added or execution order changed, update duckdb.sql to reflect the new .read directives from mimic-code's orchestrator.

4. Test materialization: Run m4 init-derived mimic-iv against a local MIMIC-IV database to verify all tables build successfully.

5. Update documentation: If new table categories or tables were added, update docs/TOOLS.md (Derived Table Categories section) and README.md.

The vendored approach means M4 works offline and ensures reproducibility -- users get the exact SQL version bundled with their M4 release, regardless of upstream changes.

Pull Request Process

1. Fork the repository
2. Create a feature branch
3. Make your changes with tests
4. Run uv run pre-commit run --all-files
5. Submit a PR with a clear description

Docker

For containerized development:

Local (DuckDB):

docker build -t m4:lite --target lite .
docker run -d --name m4-server m4:lite tail -f /dev/null

BigQuery:

docker build -t m4:bigquery --target bigquery .
docker run -d --name m4-server \
  -e M4_BACKEND=bigquery \
  -e M4_PROJECT_ID=your-project-id \
  -v $HOME/.config/gcloud:/root/.config/gcloud:ro \
  m4:bigquery tail -f /dev/null

MCP config for Docker:

{
  "mcpServers": {
    "m4": {
      "command": "docker",
      "args": ["exec", "-i", "m4-server", "python", "-m", "m4.mcp_server"]
    }
  }
}