Runner Dev

Lemline Runner Development Guide

Purpose

Guide development across the modular lemline-runner ecosystem - a collection of feature-focused modules built on shared infrastructure. The main lemline-runner module provides messaging, CLI, and configuration, while feature modules (lemline-runner-*) implement specific workflow capabilities.

Core Documentation:

• Main Runner README - Configuration, database, messaging setup
• CLI Commands
• Configuration
• Messaging Architecture
• Database Tables
• Repositories Guide

Module Architecture

Core Infrastructure

Feature Modules

Each feature module has its own README.md with detailed architecture, file reference, and usage patterns.

Quick Reference

If you need to work on...

A Specific Feature:

Read the module's README first:

• Wait task → lemline-runner-waits/README.md
• Retry logic → lemline-runner-retries/README.md
• Parent-child workflows → lemline-runner-parents/README.md
• Fork/parallel execution → lemline-runner-forks/README.md
• Scheduled workflows → lemline-runner-schedules/README.md
• CloudEvent listeners → lemline-runner-listeners/README.md
• Failure tracking → lemline-runner-failures/README.md
• Definition storage → lemline-runner-definitions/README.md

Messaging:

First, read runner-messaging.md

• Add a new command/event type → Modify WorkflowState.kt
• Update commands behavior → Modify WorkflowCommandHandler.kt
• Update events behavior → Modify WorkflowEventHandler.kt

Shared Infrastructure:

Read lemline-runner-common/README.md first

• Add new outbox processor → Extend AbstractOutbox<T> in lemline-runner-common
• Add new cleaner → Extend AbstractCleaner<T> in lemline-runner-common
• Add repository operations → Add to lemline-runner-common/repositories/ops/
• Add model interface → Create in lemline-runner-common/models/

Configuration:

First read runner-configuration.md

• Add/Update config property → Modify LemlineConfiguration.kt
• Add database/broker type → Read runner-configuration.md

CLI:

First read runner-cli.md

Critical Rules

✅ ALWAYS Do This

1. Use suspend functions for all database operations (Kotlin coroutines, NOT Mutiny)
2. Use native SQL via repositories (NOT Hibernate ORM/Panache)
3. Use Flyway migrations for all schema changes
4. Support all databases (PostgreSQL, MySQL, H2) - use database-agnostic SQL
5. Use FOR UPDATE SKIP LOCKED for outbox queries to prevent double-processing
6. Use IDV7 (UUID v7) for all entity IDs - time-sortable, globally unique
7. Test with all supported databases when touching persistence layer

❌ NEVER Do This

1. Use Mutiny (Uni/Multi) - use Kotlin coroutines with suspend functions instead
2. Use Hibernate ORM/Panache - use native SQL with repositories
3. Block the event loop - all I/O must be non-blocking
4. Skip database migrations - never modify tables directly
5. Use database-specific SQL without providing variants for all databases
6. Commit sensitive data (.env, credentials, API keys)

Architecture Overview

Modular Feature Design

┌────────────────────────────────────────────────────────────────────┐
│                      lemline-runner (core)                         │
│  ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐ │
│  │   Messaging      │  │   CLI Commands   │  │  Configuration   │ │
│  │ (commands/events)│  │   (Picocli)      │  │   (Quarkus)      │ │
│  └──────────────────┘  └──────────────────┘  └──────────────────┘ │
└────────────────────────────────────────────────────────────────────┘
                                │
                ┌───────────────┴────────────────┐
                │  lemline-runner-common         │
                │  (shared infrastructure)       │
                │  • Outbox pattern              │
                │  • Cleaner pattern             │
                │  • Repository abstractions     │
                │  • Model interfaces            │
                └───────────────┬────────────────┘
                                │
        ┌───────────────────────┼────────────────────────┐
        │                       │                        │
┌───────▼──────┐   ┌───────────▼──────┐   ┌────────────▼─────┐
│ Feature      │   │ Feature           │   │ Feature          │
│ Modules      │   │ Modules           │   │ Modules          │
│              │   │                   │   │                  │
│ • waits      │   │ • schedules       │   │ • definitions    │
│ • retries    │   │ • listeners       │   │ • failures       │
│ • parents    │   │                   │   │                  │
│ • forks      │   │                   │   │                  │
└──────────────┘   └───────────────────┘   └──────────────────┘

Dual-Channel Messaging

┌─────────────────────────────────────────────────────────────────────────────┐
│                  COMMANDS CHANNEL (high-throughput)                         │
│  commands ──► WorkflowCommandHandler ──► commands                           │
│       ▲                 │                                                   │
│       │                 │ (needs persistence)                               │
└───────│─────────────────│───────────────────────────────────────────────────┘
        │                 │
┌───────│─────────────────│───────────────────────────────────────────────────┐
│       │                 ▼              EVENTS CHANNEL                       │
│       │               events ──► WorkflowEventHandler ──► Feature Modules   │
│       │                                   │                                 │
│       │                         ┌─────────┴─────────┐                       │
│       │                         ▼                   ▼                       │
│       │              ┌─────────────────┐  ┌─────────────────┐              │
│       │              │ Feature Service │  │ Feature Service │              │
│       │              │ (e.g., Waits)   │  │ (e.g., Parents) │              │
│       │              └────────┬────────┘  └────────┬────────┘              │
│       │                       │                    │                       │
│       │                       ▼                    ▼                       │
│       │              ┌─────────────────┐  ┌─────────────────┐              │
│       │              │   DB Table      │  │   DB Table      │              │
│       │              │ (lemline_waits) │  │(lemline_parents)│              │
│       │              └────────┬────────┘  └─────────────────┘              │
│       │                       │                                            │
│       │                       ▼                                            │
│       │              ┌─────────────────┐                                   │
│       │              │  Outbox Relay   │                                   │
│       └──────────────│ (scheduled poll)│                                   │
│                      └─────────────────┘                                   │
└─────────────────────────────────────────────────────────────────────────────┘

Key principle: State travels with messages. Database only used when necessary. Each feature is self-contained.

Key Files

Common Patterns

Creating a New Feature Module

1. Module Structure:

lemline-runner-myfeature/
├── src/
│   ├── main/kotlin/com/lemline/runner/myfeature/
│   │   ├── MyFeatureService.kt         ← Business logic
│   │   ├── MyFeatureModel.kt           ← Database entity
│   │   ├── MyFeatureRepository.kt      ← Database operations
│   │   ├── MyFeatureOutbox.kt          ← Outbox processor (optional)
│   │   ├── MyFeatureCleaner.kt         ← Cleanup scheduler (optional)
│   │   └── MyFeatureConfig.kt          ← Configuration
│   ├── test/kotlin/                    ← Tests for all databases
│   └── testFixtures/kotlin/            ← Test utilities
├── build.gradle.kts
└── README.md                           ← Architecture and usage

2. Dependencies in build.gradle.kts:

dependencies {
    implementation(project(":lemline-common"))
    implementation(project(":lemline-core"))
    implementation(project(":lemline-runner-common"))  // Always required!
    // Add other feature modules if needed
}

3. Model with Interfaces:

// Compose behavior from lemline-runner-common interfaces
data class MyFeatureModel(
    override val id: IDV7,
    override val instanceMessage: InstanceMessage<MyEvent>,
    // Outbox fields
    override val outboxScheduledFor: Instant,
    override var outboxDelayedUntil: Instant? = outboxScheduledFor,
    override var outboxAttemptCount: Int = 0,
    override var outboxCompletedAt: Instant? = null,
    override var outboxFailedAt: Instant? = null,
    // ... other outbox fields
    // Cleanup field
    override var cleanupAfter: Instant? = null,
) : WithId, WithInstanceMessage, WithOutbox, WithCleanup

4. Repository:

@ApplicationScoped
class MyFeatureRepository : Repository<MyFeatureModel>(),
    WithIdRepository<MyFeatureModel>,
    OutboxRepository<MyFeatureModel>,
    CleanerRepository<MyFeatureModel> {

    override suspend fun findByUUID(uuid: IDV7): MyFeatureModel? {
        // Use pool from base Repository
        return pool.withConnection { conn ->
            conn.preparedQuery("SELECT * FROM lemline_myfeature WHERE id = $1")
                .execute(Tuple.of(uuid.value))
                .awaitSuspending()
                .firstOrNull()
                ?.let { MyFeatureModel.fromRow(it) }
        }
    }

    // OutboxRepository provides findPendingWithLock() automatically
    // CleanerRepository provides findOldCompleted() automatically
}

5. Service:

@ApplicationScoped
class MyFeatureService @Inject constructor(
    private val repository: MyFeatureRepository,
    private val commandEmitter: CommandEmitter
) {
    suspend fun handleMyEventStarted(message: InstanceMessage<MyEventStarted>) {
        val model = MyFeatureModel.from(message)
        repository.insert(model)
    }
}

6. Outbox (if needed):

@ApplicationScoped
class MyFeatureOutbox @Inject constructor(
    private val repository: MyFeatureRepository,
    private val emitter: WorkflowCommandEmitter,
    private val config: MyFeatureConfig
) : AbstractOutbox<MyFeatureModel>(
    name = "MyFeature",
    config = config.outbox
) {
    override suspend fun findEntitiesToProcess(limit: Int) = 
        repository.findPendingWithLock(limit)

    override suspend fun process(entity: MyFeatureModel) {
        emitter.send(createResumeCommand(entity))
    }

    override suspend fun markCompleted(entity: MyFeatureModel) {
        repository.markCompleted(entity.id)
    }
}

7. Register in WorkflowEventHandler:

// In lemline-runner/src/.../messaging/events/WorkflowEventHandler.kt
when (val state = message.state) {
    is MyEventStarted -> myFeatureService.handleMyEventStarted(message)
    // ...
}

Event Handling Pattern

// In feature module service
suspend fun handleMyEvent(message: InstanceMessage<MyEvent>) {
    val event = message.state

    // 1. Create model from event
    val model = MyFeatureModel.from(message, event)

    // 2. Persist to database
    repository.insert(model)

    // 3. If immediate resume needed (no outbox), emit command
    if (!needsDelay) {
        commandEmitter.send(createResumeCommand(message))
    }
    // Otherwise, outbox processor will handle it later
}

Testing Patterns

Multi-Database Repository Test

Test all 3 databases: PostgreSQL, MySQL, H2

// Base test with test logic
abstract class MyRepositoryTestBase : FunSpec({
    lateinit var repository: MyRepository

    test("should find by UUID") {
        val model = createTestModel()
        repository.insert(model)

        val found = repository.findByUUID(model.id)

        found shouldNotBe null
        found?.id shouldBe model.id
    }
})

// PostgreSQL test
@QuarkusTest
@TestProfile(PostgresProfile::class)
class MyRepositoryPostgresTest : MyRepositoryTestBase() {
    @Inject
    override lateinit var repository: MyRepository
}

// MySQL test
@QuarkusTest
@TestProfile(MySQLProfile::class)
class MyRepositoryMySQLTest : MyRepositoryTestBase() {
    @Inject
    override lateinit var repository: MyRepository
}

// H2 test
@QuarkusTest
@TestProfile(H2Profile::class)
class MyRepositoryH2Test : MyRepositoryTestBase() {
    @Inject
    override lateinit var repository: MyRepository
}

Feature Service Test

@QuarkusTest
class MyFeatureServiceTest : FunSpec({

    @Inject
    lateinit var service: MyFeatureService

    @Inject
    lateinit var repository: MyFeatureRepository

    test("should handle event") {
        val message = createTestMessage()

        service.handleMyEvent(message)

        // Verify database state
        val stored = repository.findByUUID(message.workflowId)
        stored shouldNotBe null
    }
})

Outbox Test

@QuarkusTest
class MyFeatureOutboxTest : FunSpec({

    @Inject
    lateinit var outbox: MyFeatureOutbox

    @Inject
    lateinit var repository: MyFeatureRepository

    test("should process pending entities") {
        // Insert pending entity
        val model = createPendingModel()
        repository.insert(model)

        // Process
        outbox.doWork()

        // Verify marked completed
        val processed = repository.findByUUID(model.id)
        processed?.outboxCompletedAt shouldNotBe null
    }
})

Database Migrations

IMPORTANT: Place migrations in the feature module, not in lemline-runner!

Location in Feature Module

lemline-runner-myfeature/
└── src/main/resources/db/migration/
    ├── postgresql/
    │   └── V{N}__Create_myfeature_table.sql
    ├── mysql/
    │   └── V{N}__Create_myfeature_table.sql
    └── h2/
        └── V{N}__Create_myfeature_table.sql

Naming: V{N}__Description.sql where N is the next available version number across all modules.

Check existing versions first:

# Find highest version number
find . -name "V*.sql" | sort

Migration Template

-- PostgreSQL version
CREATE TABLE lemline_myfeature
(
    id                   UUID PRIMARY KEY,
    
    -- Instance message (serialized workflow state)
    instance_message     TEXT         NOT NULL,
    
    -- Feature-specific columns
    my_custom_field      VARCHAR(255),
    
    -- Outbox columns (if using outbox pattern)
    outbox_scheduled_for TIMESTAMP    NOT NULL,
    outbox_delayed_until TIMESTAMP    NOT NULL,
    outbox_attempt_count INT          NOT NULL DEFAULT 0,
    outbox_completed_at  TIMESTAMP,
    outbox_failed_at     TIMESTAMP,
    outbox_error_class   VARCHAR(255),
    outbox_error_message VARCHAR(500),
    outbox_error_stacktrace TEXT,
    
    -- Cleanup column (if using cleaner pattern)
    cleanup_after        TIMESTAMP,
    
    -- Timestamps
    created_at           TIMESTAMP    NOT NULL DEFAULT NOW()
);

-- Index for outbox queries (FOR UPDATE SKIP LOCKED)
CREATE INDEX idx_lemline_myfeature_pending
    ON lemline_myfeature (outbox_delayed_until)
    WHERE outbox_completed_at IS NULL AND outbox_failed_at IS NULL;

-- Index for cleanup queries
CREATE INDEX idx_lemline_myfeature_cleanup
    ON lemline_myfeature (cleanup_after)
    WHERE cleanup_after IS NOT NULL;

Database-Specific Considerations

PostgreSQL:

• Use UUID type
• Use TEXT for long strings
• Partial indexes with WHERE clause

MySQL:

• Use CHAR(36) instead of UUID
• Use LONGTEXT for long strings
• No partial indexes - use regular index

H2:

• Use UUID type
• Use CLOB for very long strings
• Supports partial indexes

Testing Migrations

# Test with PostgreSQL
./gradlew :lemline-runner-myfeature:test -Dquarkus.test.profile=postgres

# Test with MySQL
./gradlew :lemline-runner-myfeature:test -Dquarkus.test.profile=mysql

# Test with H2
./gradlew :lemline-runner-myfeature:test -Dquarkus.test.profile=h2

Running Tests

# All tests in main runner
./gradlew :lemline-runner:test

# All tests in feature module
./gradlew :lemline-runner-myfeature:test

# Specific test class
./gradlew :lemline-runner-myfeature:test --tests "com.lemline.runner.myfeature.MyTest"

# Test specific module with specific database
./gradlew :lemline-runner-waits:test -Dquarkus.test.profile=postgres
./gradlew :lemline-runner-waits:test -Dquarkus.test.profile=mysql
./gradlew :lemline-runner-waits:test -Dquarkus.test.profile=h2

# Test all runner modules
./gradlew test -p lemline-runner -p lemline-runner-common -p lemline-runner-waits -p lemline-runner-retries ...

Adding a New DSL Feature

Example: Adding support for a new task type that requires persistence

1. Add task to lemline-core (see core-dev skill)

   • Create model in lemline-core/src/.../models/tasks/
   • Create processor in lemline-core/src/.../processors/
   • Throw AsyncTaskException if needs persistence

2. Create feature module

   mkdir -p lemline-runner-myfeature/src/{main,test}/kotlin/com/lemline/runner/myfeature
   mkdir -p lemline-runner-myfeature/src/main/resources/db/migration/{postgresql,mysql,h2}
   

3. Add to settings.gradle.kts

   include("lemline-runner-myfeature")
   

4. Create build.gradle.kts

   dependencies {
       implementation(project(":lemline-common"))
       implementation(project(":lemline-core"))
       implementation(project(":lemline-runner-common"))
   }
   

5. Implement feature (Model, Repository, Service, Outbox, Cleaner)

6. Add migrations for all 3 databases

7. Register in WorkflowEventHandler

   // In lemline-runner
   when (val state = message.state) {
       is MyFeatureStarted -> myFeatureService.handleStarted(message)
       // ...
   }
   

8. Add tests for all databases

9. Add to main runner dependencies

   // In lemline-runner/build.gradle.kts
   implementation(project(":lemline-runner-myfeature"))
   

10. Document in README.md in the feature module

Related Documentation

• CLAUDE.md - Project-wide guidelines and architecture overview
• AGENTS.md - Build, test, and lint commands
• core-dev skill - lemline-core module (DSL, orchestrators, processors)
• Serverless Workflow DSL - https://serverlessworkflow.io/

Module Dependency Graph

lemline-common
    ↑
    │
lemline-core
    ↑
    │
lemline-runner-common ←────┐
    ↑                      │
    │                      │
    ├──────────────────────┼─────────────────┐
    │                      │                 │
lemline-runner-*     lemline-runner-*   lemline-runner-*
(feature modules)    (feature modules)   (feature modules)
    ↑                      ↑                 ↑
    │                      │                 │
    └──────────────────────┴─────────────────┘
                           │
                    lemline-runner
                   (main runtime)

Key principle: Feature modules depend only on common modules, not on each other (except rare cases like schedules/parents).