Vledicfranco Constellation Engine Developer

Constellation Engine Developer Context

You are working on Constellation Engine, a type-safe pipeline orchestration framework for Scala 3. Before starting any task, internalize this context.

1. What Is Constellation Engine?

A framework that lets users:

1. Define processing modules in Scala using ModuleBuilder API
2. Compose pipelines in constellation-lang DSL (.cst files)
3. Execute pipelines with automatic dependency resolution and parallelization
4. Expose pipelines via HTTP API or embed in Scala applications

Tech stack: Scala 3.3.4 | Cats Effect 3 | http4s | Circe | cats-parse

Processing Pipeline

Source (.cst) --> Parser --> AST --> Type Checker --> Typed IR --> DAG Compiler --> Execution DAG --> Runtime --> Results

Each stage catches different errors: syntax errors at parse, type errors at type-check, dependency errors at compile, runtime failures at execute.

2. Module Dependency Graph

CRITICAL: modules can only depend on modules above them. NEVER create circular dependencies.

core (foundation - no dependencies)
  |-> runtime
  |     |-> cache-memcached
  |     |-> doc-generator
  |
  +-> lang-ast
        +-> lang-parser
              +-> lang-compiler
                    |-> lang-stdlib
                    |-> lang-lsp
                    |-> lang-cli
                    |-> http-api
                    +-> example-app

Layering:

1. Core (core): Pure type definitions, no side effects
2. Execution (runtime): Module execution, caching, pools
3. Language (lang-*): Parsing, compilation, type checking
4. Service (http-api, lang-lsp, lang-cli): External interfaces
5. Application (example-app): User-facing modules

3. Key File Locations

Core Entry Points

Issue Triage: Where to Look

4. Development Commands

ALWAYS use make commands. NEVER use raw sbt commands directly.

Windows (if make unavailable): .\scripts\dev.ps1 or .\scripts\dev.ps1 -ServerOnly

sbt on Windows caveat: Cannot run multiple sbt instances in parallel (lock conflict). Run tests sequentially.

5. Type System Overview

Constellation has two parallel type systems:

Runtime Types (CType/CValue) - in TypeSystem.scala

CType (sealed trait)              CValue (sealed trait)
|-- CString                       |-- CString(value: String)
|-- CInt                          |-- CInt(value: Long)
|-- CFloat                        |-- CFloat(value: Double)
|-- CBoolean                      |-- CBoolean(value: Boolean)
|-- CList(valuesType)             |-- CList(value: Vector[CValue])
|-- CMap(keysType, valuesType)    |-- CMap(value: Vector[(CValue, CValue)])
|-- CProduct(structure: Map)      |-- CProduct(value: Map[String, CValue])
|-- CUnion(structure: Map)        |-- CUnion(tag: String, value: CValue, unionType: CType)
|-- COptional(innerType)          |-- COptional(value: Option[CValue])

Compile-Time Types (SemanticType) - in SemanticType.scala

Used during type checking. Maps to CType for runtime. Supports unions, optionals, row polymorphism, type inference.

Known API Quirks

• CValue.CList has field value (not values), type Vector[CValue]
• CValue.CMap takes Vector[(CValue, CValue)] (not Map)
• RawValue.RMap has field entries (not a length method)
• ResilienceModules uses majorVersion/minorVersion (not version.major)

6. Code Conventions

Module Creation Pattern

import cats.effect.IO
import cats.implicits._  // Required for .traverse
import io.constellation._

// ALWAYS use case classes, NEVER tuples
case class MyInput(text: String)
case class MyOutput(result: String)

val myModule: Module.Uninitialized = ModuleBuilder
  .metadata("MyModule", "Description", 1, 0)
  .implementationPure[MyInput, MyOutput] { input =>
    MyOutput(input.text.toUpperCase)
  }
  .build

Rules:

• Module name in metadata() MUST match usage in .cst files (case-sensitive)
• Case class field names MUST match constellation-lang variable names
• ALWAYS use case classes for I/O, NEVER tuples (Scala 3 doesn't support single-element tuples)
• Import cats.implicits._ when using .traverse for module registration

Implementation Types

// Pure (no side effects)
.implementationPure[Input, Output] { input => Output(...) }

// IO-based (side effects allowed)
.implementation[Input, Output] { input => IO { Output(...) } }

Naming Conventions

• Modules: PascalCase (Uppercase, WordCount)
• Variables: camelCase (val uppercase, val wordCount)
• Case classes: PascalCase (TextInput, WordCountOutput)
• Fields: camelCase, must match constellation-lang variable names
• Packages: match source location (io.constellation, io.constellation.lang.compiler)

constellation-lang Syntax

# Input declarations
in text: String
in count: Int
in user: {name: String, age: Int}

# Module calls (PascalCase, must match ModuleBuilder name)
trimmed = Trim(text)
result = Uppercase(trimmed)

# With resilience options
cached = ExpensiveAPI(input) with {
  retry: 3,
  timeout: 10s,
  cache: 1h,
  fallback: DefaultValue(input)
}

# Output declarations
out result

7. Testing Conventions

• Framework: ScalaTest (AnyFlatSpec + Matchers)
• Package matches source: io.constellation, io.constellation.lang.compiler, etc.
• IO tests: import cats.effect.unsafe.implicits.global then .unsafeRunSync()
• Error tests: intercept[ErrorType] { ... } or .attempt.unsafeRunSync() for IO
• Section comments: // ===== Section Name =====
• Run tests before committing: make test

8. Adding New Modules to example-app

1. Define module in modules/example-app/src/main/scala/io/constellation/examples/app/modules/
2. Add FunctionSignature to ExampleLib.scala
3. Register in allSignatures and allModules
4. Restart server

9. Adding Standard Library Functions

1. Create module in appropriate category: modules/lang-stdlib/src/main/scala/io/constellation/stdlib/categories/
2. Register in modules/lang-stdlib/src/main/scala/io/constellation/stdlib/StdLib.scala
3. Add signature to allSignatures
4. Add module to allModules

10. HTTP API & Server

• Default port: 8080 (configurable via CONSTELLATION_PORT)
• Health: GET /health, /health/live, /health/ready
• Execute pipeline: POST /execute
• List modules: GET /modules
• Validate: POST /validate
• LSP WebSocket: ws://localhost:{port}/lsp

Server Builder Pattern

ConstellationServer.builder(constellation, compiler)
  .withAuth(AuthConfig(apiKeys = Map("key1" -> ApiRole.Admin)))
  .withCors(CorsConfig(allowedOrigins = Set("https://app.example.com")))
  .withRateLimit(RateLimitConfig(requestsPerMinute = 100, burst = 20))
  .withHealthChecks(HealthCheckConfig(enableDetailEndpoint = true))
  .run

11. Deep-Dive Documentation (Read On-Demand)

When you need deeper context on a specific topic, read these files:

Foundations

Patterns

Reference

Integration

Architecture & Philosophy

Component Details (organon)

Feature Details (organon)

Auto-Generated Type Catalogs

For exact Scala signatures, check organon/generated/io.constellation.*.md files. These are auto-generated from TASTy and are read-only.

12. Task Workflow

When starting any development task:

1. Understand the task - Read the issue or request carefully
2. Locate relevant code - Use the issue triage table above to find the right module
3. Read the code - Always read existing code before modifying. Read relevant deep-dive docs if needed.
4. Make changes - Follow code conventions above
5. Test - Run make test (or module-specific make test-core, make test-compiler, etc.)
6. Self-review - Check correctness, consistency, edge cases before committing

Performance Targets (for performance-critical changes)

13. Common Pitfalls

1. Don't forget cats.implicits._ - Required for .traverse
2. Don't mismatch field names - Case class fields must match constellation-lang variable names
3. Don't use raw sbt - Always use make commands
4. Don't create circular dependencies - Respect the module dependency graph
5. Don't use tuples for module I/O - Always use case classes
6. Don't skip testing - Run make test before committing
7. Module names are case-sensitive - "Uppercase" in Scala must match Uppercase in .cst

Converted and distributed by TomeVault — claim your Tome and manage your conversions.