Template-Based Code Generation
Three inputs produce repetitive (typically interface) code:
- Data model: Standardized representation -- XML, Domain-Specific Language (DSL), or schema-enforced JSON/YAML.
- Template: Templating meta-language markup (Jinja2, ERB, etc.).
- Helper functions: Functions that reshape the data model for template consumption.
Data Model ──► Parser ──► Helpers ──► Template Engine ──► Generated Output
(SVD, (cmsis-svd, (reshape, (Jinja2, (one file per
IDL, sqlparse) filter) ERB) generation unit)
DBML...)
Terms
| Term |
Definition |
| Data Model |
Input describing the structure to replicate (e.g. SVD file for hardware registers). |
| Helpers/Plugins |
Functions that transform/filter/restructure the data model for templates. |
| Template(s) |
Defines output structure in the target language with template markup for variable parts. |
| Target Language |
Language of the generated output (e.g. C++, Python, Verilog). |
| Generated Output |
Files produced by the generator. Must never be hand-edited (see: Rules). |
| Generation Unit |
Data model component a template operates on (e.g. peripheral, table, message). One output file per unit (see: Step 3). |
| Data Bridge Template |
Template converting the data model into target-language compile-time representations (enums, constants, macros). Consumed by in-language meta-programming (see: Meta-Programming Rules). |
Generator Types
| Type |
Description |
Template Location |
Flexibility |
| Fixed function |
Template embedded in generator tool code |
Inside the tool |
Output format fixed; change requires modifying the tool |
| Customizable template-based |
Template separated from tool |
User-editable files |
User can modify/replace templates freely |
Directive: Prefer customizable template-based generators for agent use. The agent can read, understand, and modify templates without understanding the tool's internals.
Meta-Programming
| Type |
Mechanism |
When to use |
| In-language |
C++ templates/constexpr/consteval, C preprocessor, Python decorators, Verilog generate |
Data can be represented in target language compile-time constructs |
| External template-based |
Jinja2/ERB generating source files |
Data cannot be naturally represented in target language (e.g. register maps, protocol definitions) |
Directive: Prefer in-language meta-programming -- the target toolchain handles validation, error reporting, and IDE support. Combine with external generation using the layered approach (see: Meta-Programming Rules).
Examples
Generator Tools
| Tool |
Type |
Templating |
Domain |
Protobuf (protoc) |
Fixed |
Embedded |
Serialization / RPC |
Cyclone DDS (idlc) |
Fixed |
Embedded |
DDS/OMG IDL interfaces |
| SWIG |
Fixed |
Embedded |
Foreign Function Interface (FFI) wrappers |
| Ruby on Rails |
Template |
ERB |
Web scaffolding |
| Cookiecutter |
Template |
Jinja2 |
Project scaffolding |
| development-utils |
Template |
Jinja2 |
Hardware description (SVD, SystemRDL, DBML, Device Tree, IDL) |
Data Model Formats
| Format |
Domain |
Type |
| SQL DDL |
Databases |
Data exchange |
| XML (SVD, ATML) |
Hardware, general |
Data exchange |
| JSON / YAML / TOML |
General (schema required) |
Data exchange |
| IDL (Interface Definition Language) |
Distributed systems (CORBA, DDS, Protobuf) |
DSL |
| SystemRDL |
Register descriptions |
DSL |
| DBML (Database Markup Language) |
Database schemas |
DSL |
| AADL (Architecture Analysis and Design Language) |
Safety-critical systems |
DSL |
| SysML |
Systems modeling |
DSL |
| Device Tree |
Linux/embedded hardware |
DSL |
Templating Languages
| Language |
Ecosystem |
Logic |
Strengths |
| Jinja2 |
Python |
Full (loops, macros, filters) |
Most expressive; StrictUndefined; dominant for standalone generators |
| ERB |
Ruby |
Full Ruby execution |
Native to Rails |
| Liquid |
Ruby |
Safe subset |
Sandboxed; safe for user-supplied templates |
| Mako |
Python |
Full Python execution |
Fast; used by Alembic |
| Mustache/Handlebars |
Language-agnostic |
Logic-less |
Forces logic into helpers |
Process
Step 1: Identify the Data Model
Use the user's existing data model. If none exists, select a domain standard:
| Domain |
Standard formats |
| Hardware registers |
SVD, SystemRDL, IP-XACT (IEEE 1685) |
| Communication protocols |
Protobuf, IDL, ASN.1 |
| Database schemas |
SQL DDL, DBML |
| APIs |
OpenAPI/Swagger, GraphQL |
| Hardware architecture |
Device Tree, AADL |
Step 2: Understand the Interface Boundary
Identify the specification-defined interface:
| Boundary |
Examples |
| Persistent data |
Database tables, file formats |
| Inter-Process Communication (IPC) |
Shared memory, message queues, pub/sub |
| Hardware/software |
Memory-Mapped I/O (MMIO) registers, Direct Memory Access (DMA) descriptors |
| Language interop |
Java Native Interface (JNI), Python C extensions, FFI |
| Network protocols |
RPC definitions, serialization formats |
Step 3: Determine the Generation Granularity
| Granularity |
Scope |
Use case |
| Component-level |
One output file per generation unit (peripheral, message, table) |
Direct usage patterns; simpler templates |
| System-level |
One output file covering all components |
Cross-cutting concerns (dispatch tables, device maps, schema migrations) |
Split templates by granularity. A single system-wide template that includes all components is difficult to maintain.
Step 4: Template Output Conventions
| Convention |
Pattern |
Example |
| Template naming |
<generation_unit>_<output_name>.<target_ext>.jinja2 |
peripheral_regs.hpp.jinja2, table_model.py.jinja2 |
| Output directory |
Separate from hand-written source: generated/, gen/, build/gen/ |
Never mix generated and hand-written files |
| Version control |
Add output directories to .gitignore |
Exception: version output if users lack the generator toolchain |
Step 5: Write the Generator Pipeline
| Stage |
Responsibility |
Notes |
| Parser |
Read and validate data model; produce in-memory representation |
Reuse open-source parsers: cmsis-svd, systemrdl-compiler, sqlparse. Custom parser = last resort. |
| Helpers |
Reshape/filter parsed data for template consumption |
Keep logic out of templates |
| Templates |
Emit target-language constructs; read like target source code |
Minimal control flow; iterate over data |
| Driver script |
Connect parser + helpers + templates; write output files |
Script or Makefile target |
Rules for Using Generators
| # |
Rule |
Detail |
| 1 |
Never edit generated output |
Fix the data model, template, or helper instead. If output is wrong, fix the input. |
| 2 |
Interface changes go in the data model |
New field/register/message = modify the data model source, not the template. |
| 3 |
Data reshaping goes in helpers |
Filtering, renaming, restructuring = helper functions. Keep templates simple. |
| 4 |
Templates read like target source code |
Template markup ({% for %}, {{ }}) is a thin layer over recognizable target code. |
| 5 |
Control whitespace explicitly |
Jinja2: enable trim_blocks + lstrip_blocks. Use {%-/-%} for fine-grained control. Verify output formatting. |
| 6 |
One concern per template |
Each template produces one logical output artifact. |
Rules for Meta-Programming with Generators
When combining external generation with in-language meta-programming:
| # |
Rule |
Detail |
| 1 |
Prefer target-language mechanisms |
C++ constexpr/consteval, Python metaclasses, Verilog generate, C macros. |
| 2 |
Data bridge template |
One template converts data model to target-language compile-time data (enums, constants, constexpr arrays). |
| 3 |
Separate runtime templates |
Consume compile-time data via target-language meta-programming (e.g. C++ template class parameterized by generated enum). |
| 4 |
Layering principle |
External generation = data declarations. In-language meta-programming = behavior. Each layer independently testable. |
Testing Generators
| Strategy |
What it catches |
How |
| Golden-file tests |
Unintended output changes |
Store known-good output as reference; diff against current output |
| Compilation tests |
Syntax errors, type mismatches |
Compile (or lint) generated output after generation |
| Round-trip tests |
Semantic correctness |
Verify generated code represents the data model correctly (e.g. register access returns expected values) |
| Sample data models |
Template edge cases |
Maintain small test data exercising all control flow paths (optional fields, empty lists, single-element lists) |
Build System Integration
| Concern |
Approach |
| Dependency tracking |
Makefile/script declares deps on data model + templates + helpers. Any input change triggers regeneration. |
| Makefile pattern |
generated/%.hpp: data/%.xml templates/peripheral_regs.hpp.jinja2 helpers.py |
| Incremental generation |
For large data models, regenerate only changed generation units. |
| CI verification |
Run generator in CI; verify checked-in output (if versioned) matches generator output. |
Post-Generation Hooks
| Hook |
Purpose |
Example |
| Code formatters |
Match project style without burdening templates |
clang-format (C/C++), black (Python), rustfmt (Rust) |
| Linters |
Catch issues template engine cannot detect |
Unused imports, naming violations |
| Hook integration |
Run in driver script after rendering |
clang-format -i generated/*.hpp |
Error Handling
| Scenario |
Action |
| Malformed data model |
Validate against schema before rendering. Report errors with file location + field name. Do not render invalid data. |
| Missing template fields |
Use strict undefined handling (Jinja2: undefined=StrictUndefined). Fail loudly. |
| Non-deterministic output |
Same inputs must produce identical output. If not, there is a bug in the generator. |
Example Generator Repository
development-utils demonstrates:
| Aspect |
Details |
| Parsers |
Python-based: SVD, SystemRDL, DBML, Device Tree, IDL |
| Output |
C++17/C++20 MMIO register interfaces, SQLAlchemy/Pydantic models from DBML, RISC-V CSR access code (C/C++/Rust) |
| Naming |
<generation_unit>_<name>.<ext>.jinja2 |
| Granularity |
Component-level (per peripheral) + system-level (device maps) |
1---2name: generator-coding3description: Template-based code generation pattern using data models, templates, and helper functions to generate repetitive interface code.4---56# Template-Based Code Generation78Three inputs produce repetitive (typically interface) code:910- **Data model**: Standardized representation -- XML, Domain-Specific Language (DSL), or schema-enforced JSON/YAML.11- **Template**: Templating meta-language markup ([Jinja2](https://jinja.palletsprojects.com/), [ERB](https://docs.ruby-lang.org/en/master/ERB.html), etc.).12- **Helper functions**: Functions that reshape the data model for template consumption.1314```15Data Model ──► Parser ──► Helpers ──► Template Engine ──► Generated Output16 (SVD, (cmsis-svd, (reshape, (Jinja2, (one file per17 IDL, sqlparse) filter) ERB) generation unit)18 DBML...)19```202122## Terms2324| Term | Definition |25|------|-----------|26| **Data Model** | Input describing the structure to replicate (e.g. SVD file for hardware registers). |27| **Helpers/Plugins** | Functions that transform/filter/restructure the data model for templates. |28| **Template(s)** | Defines output structure in the target language with template markup for variable parts. |29| **Target Language** | Language of the generated output (e.g. C++, Python, Verilog). |30| **Generated Output** | Files produced by the generator. Must never be hand-edited (see: Rules). |31| **Generation Unit** | Data model component a template operates on (e.g. peripheral, table, message). One output file per unit (see: Step 3). |32| **Data Bridge Template** | Template converting the data model into target-language compile-time representations (enums, constants, macros). Consumed by in-language meta-programming (see: Meta-Programming Rules). |333435## Generator Types3637| Type | Description | Template Location | Flexibility |38|------|-------------|-------------------|-------------|39| **Fixed function** | Template embedded in generator tool code | Inside the tool | Output format fixed; change requires modifying the tool |40| **Customizable template-based** | Template separated from tool | User-editable files | User can modify/replace templates freely |4142**Directive**: Prefer customizable template-based generators for agent use. The agent can read, understand, and modify templates without understanding the tool's internals.434445## Meta-Programming4647| Type | Mechanism | When to use |48|------|-----------|-------------|49| **In-language** | C++ templates/`constexpr`/`consteval`, C preprocessor, Python decorators, Verilog `generate` | Data can be represented in target language compile-time constructs |50| **External template-based** | Jinja2/ERB generating source files | Data cannot be naturally represented in target language (e.g. register maps, protocol definitions) |5152**Directive**: Prefer in-language meta-programming -- the target toolchain handles validation, error reporting, and IDE support. Combine with external generation using the layered approach (see: Meta-Programming Rules).535455## Examples5657### Generator Tools5859| Tool | Type | Templating | Domain |60|------|------|-----------|--------|61| [Protobuf](https://protobuf.dev/) (`protoc`) | Fixed | Embedded | Serialization / RPC |62| [Cyclone DDS](https://cyclonedds.io/) (`idlc`) | Fixed | Embedded | DDS/OMG IDL interfaces |63| [SWIG](https://www.swig.org/) | Fixed | Embedded | Foreign Function Interface (FFI) wrappers |64| [Ruby on Rails](https://guides.rubyonrails.org/generators.html) | Template | ERB | Web scaffolding |65| [Cookiecutter](https://www.cookiecutter.io/) | Template | Jinja2 | Project scaffolding |66| [development-utils](https://github.com/nakane1chome/development-utils) | Template | Jinja2 | Hardware description (SVD, SystemRDL, DBML, Device Tree, IDL) |6768### Data Model Formats6970| Format | Domain | Type |71|--------|--------|------|72| SQL DDL | Databases | Data exchange |73| XML ([SVD](https://www.keil.com/pack/doc/CMSIS/SVD/html/index.html), ATML) | Hardware, general | Data exchange |74| JSON / YAML / TOML | General (schema required) | Data exchange |75| [IDL](https://www.omg.org/spec/IDL/) (Interface Definition Language) | Distributed systems (CORBA, DDS, Protobuf) | DSL |76| [SystemRDL](https://www.accellera.org/downloads/standards/systemrdl) | Register descriptions | DSL |77| [DBML](https://dbml.dbdiagram.io/) (Database Markup Language) | Database schemas | DSL |78| [AADL](https://www.aadl.info/) (Architecture Analysis and Design Language) | Safety-critical systems | DSL |79| [SysML](https://sysml.org/) | Systems modeling | DSL |80| [Device Tree](https://www.devicetree.org/) | Linux/embedded hardware | DSL |8182### Templating Languages8384| Language | Ecosystem | Logic | Strengths |85|----------|-----------|-------|-----------|86| [Jinja2](https://jinja.palletsprojects.com/) | Python | Full (loops, macros, filters) | Most expressive; `StrictUndefined`; dominant for standalone generators |87| [ERB](https://docs.ruby-lang.org/en/master/ERB.html) | Ruby | Full Ruby execution | Native to Rails |88| [Liquid](https://shopify.github.io/liquid/) | Ruby | Safe subset | Sandboxed; safe for user-supplied templates |89| [Mako](https://www.makotemplates.org/) | Python | Full Python execution | Fast; used by Alembic |90| [Mustache](https://mustache.github.io/)/[Handlebars](https://handlebarsjs.com/) | Language-agnostic | Logic-less | Forces logic into helpers |919293## Process9495### Step 1: Identify the Data Model9697Use the user's existing data model. If none exists, select a domain standard:9899| Domain | Standard formats |100|--------|-----------------|101| Hardware registers | SVD, SystemRDL, [IP-XACT](https://www.accellera.org/downloads/standards/ip-xact) (IEEE 1685) |102| Communication protocols | Protobuf, IDL, [ASN.1](https://www.itu.int/en/ITU-T/asn1/Pages/introduction.aspx) |103| Database schemas | SQL DDL, DBML |104| APIs | [OpenAPI](https://www.openapis.org/)/Swagger, [GraphQL](https://graphql.org/) |105| Hardware architecture | Device Tree, AADL |106107### Step 2: Understand the Interface Boundary108109Identify the specification-defined interface:110111| Boundary | Examples |112|----------|---------|113| Persistent data | Database tables, file formats |114| Inter-Process Communication (IPC) | Shared memory, message queues, pub/sub |115| Hardware/software | Memory-Mapped I/O (MMIO) registers, Direct Memory Access (DMA) descriptors |116| Language interop | Java Native Interface (JNI), Python C extensions, FFI |117| Network protocols | RPC definitions, serialization formats |118119### Step 3: Determine the Generation Granularity120121| Granularity | Scope | Use case |122|-------------|-------|----------|123| **Component-level** | One output file per generation unit (peripheral, message, table) | Direct usage patterns; simpler templates |124| **System-level** | One output file covering all components | Cross-cutting concerns (dispatch tables, device maps, schema migrations) |125126Split templates by granularity. A single system-wide template that includes all components is difficult to maintain.127128### Step 4: Template Output Conventions129130| Convention | Pattern | Example |131|------------|---------|---------|132| **Template naming** | `<generation_unit>_<output_name>.<target_ext>.jinja2` | `peripheral_regs.hpp.jinja2`, `table_model.py.jinja2` |133| **Output directory** | Separate from hand-written source: `generated/`, `gen/`, `build/gen/` | Never mix generated and hand-written files |134| **Version control** | Add output directories to `.gitignore` | Exception: version output if users lack the generator toolchain |135136### Step 5: Write the Generator Pipeline137138| Stage | Responsibility | Notes |139|-------|---------------|-------|140| **Parser** | Read and validate data model; produce in-memory representation | Reuse open-source parsers: [`cmsis-svd`](https://github.com/cmsis-svd/cmsis-svd), [`systemrdl-compiler`](https://github.com/SystemRDL/systemrdl-compiler), [`sqlparse`](https://github.com/andialbrecht/sqlparse). Custom parser = last resort. |141| **Helpers** | Reshape/filter parsed data for template consumption | Keep logic out of templates |142| **Templates** | Emit target-language constructs; read like target source code | Minimal control flow; iterate over data |143| **Driver script** | Connect parser + helpers + templates; write output files | Script or Makefile target |144145146## Rules for Using Generators147148| # | Rule | Detail |149|---|------|--------|150| 1 | **Never edit generated output** | Fix the data model, template, or helper instead. If output is wrong, fix the input. |151| 2 | **Interface changes go in the data model** | New field/register/message = modify the data model source, not the template. |152| 3 | **Data reshaping goes in helpers** | Filtering, renaming, restructuring = helper functions. Keep templates simple. |153| 4 | **Templates read like target source code** | Template markup (`{% for %}`, `{{ }}`) is a thin layer over recognizable target code. |154| 5 | **Control whitespace explicitly** | Jinja2: enable `trim_blocks` + `lstrip_blocks`. Use `{%-`/`-%}` for fine-grained control. Verify output formatting. |155| 6 | **One concern per template** | Each template produces one logical output artifact. |156157158## Rules for Meta-Programming with Generators159160When combining external generation with in-language meta-programming:161162| # | Rule | Detail |163|---|------|--------|164| 1 | **Prefer target-language mechanisms** | C++ `constexpr`/`consteval`, Python metaclasses, Verilog `generate`, C macros. |165| 2 | **Data bridge template** | One template converts data model to target-language compile-time data (enums, constants, `constexpr` arrays). |166| 3 | **Separate runtime templates** | Consume compile-time data via target-language meta-programming (e.g. C++ template class parameterized by generated enum). |167| 4 | **Layering principle** | External generation = data declarations. In-language meta-programming = behavior. Each layer independently testable. |168169170## Testing Generators171172| Strategy | What it catches | How |173|----------|----------------|-----|174| **Golden-file tests** | Unintended output changes | Store known-good output as reference; diff against current output |175| **Compilation tests** | Syntax errors, type mismatches | Compile (or lint) generated output after generation |176| **Round-trip tests** | Semantic correctness | Verify generated code represents the data model correctly (e.g. register access returns expected values) |177| **Sample data models** | Template edge cases | Maintain small test data exercising all control flow paths (optional fields, empty lists, single-element lists) |178179180## Build System Integration181182| Concern | Approach |183|---------|----------|184| **Dependency tracking** | Makefile/script declares deps on data model + templates + helpers. Any input change triggers regeneration. |185| **Makefile pattern** | `generated/%.hpp: data/%.xml templates/peripheral_regs.hpp.jinja2 helpers.py` |186| **Incremental generation** | For large data models, regenerate only changed generation units. |187| **CI verification** | Run generator in CI; verify checked-in output (if versioned) matches generator output. |188189190## Post-Generation Hooks191192| Hook | Purpose | Example |193|------|---------|---------|194| **Code formatters** | Match project style without burdening templates | [`clang-format`](https://clang.llvm.org/docs/ClangFormat.html) (C/C++), [`black`](https://black.readthedocs.io/) (Python), [`rustfmt`](https://github.com/rust-lang/rustfmt) (Rust) |195| **Linters** | Catch issues template engine cannot detect | Unused imports, naming violations |196| **Hook integration** | Run in driver script after rendering | `clang-format -i generated/*.hpp` |197198199## Error Handling200201| Scenario | Action |202|----------|--------|203| **Malformed data model** | Validate against schema before rendering. Report errors with file location + field name. Do not render invalid data. |204| **Missing template fields** | Use strict undefined handling (Jinja2: [`undefined=StrictUndefined`](https://jinja.palletsprojects.com/en/3.1.x/api/#undefined-types)). Fail loudly. |205| **Non-deterministic output** | Same inputs must produce identical output. If not, there is a bug in the generator. |206207208## Example Generator Repository209210[development-utils](https://github.com/nakane1chome/development-utils) demonstrates:211212| Aspect | Details |213|--------|---------|214| **Parsers** | Python-based: SVD, SystemRDL, DBML, Device Tree, IDL |215| **Output** | C++17/C++20 MMIO register interfaces, SQLAlchemy/Pydantic models from DBML, RISC-V CSR access code (C/C++/Rust) |216| **Naming** | `<generation_unit>_<name>.<ext>.jinja2` |217| **Granularity** | Component-level (per peripheral) + system-level (device maps) |
nakane1chome