Tilebox Workflow Authoring

Writing Tilebox Workflows

Create or modify Python workflow source code and local/runtime behavior. Keep platform APIs in Tilebox references, portable science and IO in geospatial references, and end-to-end combinations in recipes.

Start With Current APIs And The Right Skill

When an API is unclear or remembered behavior may be stale, inspect the installed version or search current docs:

tilebox docs search "Task ExecutionContext submit_subtasks"
tilebox docs search "logging tracing context.logger context.tracer"
tilebox docs search "caches job_cache"

Delegate adjacent ownership:

• tilebox-workflow-releases: project initialization, pyproject.toml scaffold, build, publish, deploy, and runner configuration.
• tilebox-workflow-jobs: submit, wait, list, inspect, retry, cancel, and cluster operations.
• tilebox-datasets: source/product recommendations, exact dataset slugs, provider credentials, auxiliary catalogues, and CLI inspection.
• tilebox-cli: authentication, CLI discovery, JSON output, and docs search.
• tilebox-workflow-automations: cron and storage-triggered automations.

For a new project, use tilebox workflow init through tilebox-workflow-releases; do not hand-scaffold project configuration. Authoring may edit the generated code after initialization.

Plan Before Coding

For outcome-oriented Earth observation work, read reference/geospatial/project-planning.md. Confirm resolution, temporal coverage, observation quality, comparability, required inputs, validation, and delivery. Use tilebox-datasets to select and inspect the source; do not duplicate its catalogue or credential guidance.

Sketch the task graph:

1. Identify root, worker, and aggregation stages.
2. Choose a fanout axis: scenes, products, AOIs, time windows, spatial chunks, or model tiles.
3. Mark real barriers with depends_on; avoid unnecessary chains.
4. Choose task fields versus context.job_cache versus durable object/Zarr artifacts.
5. Choose retry behavior for idempotent network/storage operations.

Structure Non-Trivial Workflows As Packages

Treat the generated runner.py as a starting scaffold, not the default home for the entire implementation. A single file is acceptable for a genuinely small prototype with one or two short tasks and little processing logic. When the workflow has multiple stages or task types, substantial dataset/raster/encoding logic, reusable helpers, or independently testable processing logic, split it into an importable package with modules organized by coherent responsibility.

Keep runner.py as a thin composition root: import every registered task, construct the Runner, and configure runner-level cache or logging. Do not put task execution methods, dataset queries, raster processing, scientific algorithms, or output encoding there. Name the module containing the root task and main fanout after the workflow capability, such as rgb_timelapse.py, rather than a generic orchestration.py. Group other related task classes by workflow stage or domain, use concrete stage names such as aggregation.py, and keep portable processing/IO functions separate from Tilebox orchestration when that makes them easier to test. Avoid both a catch-all utils.py and one module per tiny class; extract a module only when it has a clear role.

A non-trivial timelapse workflow might use:

runner.py                         # task registration and runner configuration only
rgb_timelapse/
  __init__.py
  tasks/
    __init__.py
    rgb_timelapse.py              # root workflow task and fanout
    frames.py                     # frame worker task
    aggregation.py                # frame aggregation and encoding task
  imagery.py                      # bounded reads, masking, and rendering
  encoding.py                     # deterministic GIF encoding functions
  models.py                       # shared typed values, only when needed
tests/
  test_imagery.py
  test_encoding.py

Adapt the depth to the workflow: a smaller project can use tasks.py, imagery.py, and runner.py without a tasks/ subpackage. Do not unit-test Tilebox task classes merely to assert submitted tasks, dependencies, progress, or logging; verify orchestration through release build/task discovery and a representative job's actual graph. Add focused tests only where useful for substantive underlying functions such as scene selection/grouping, transforms, masking, rendering, or encoding. Extract those functions from task execution methods so they can be tested without mocking ExecutionContext, but do not create trivial helpers solely to produce tests. Ensure every package module needed at runtime is included by the release build configuration, and verify imports from the built artifact rather than only from the working tree.

Require Task-Level Parallelism

Task-level fanout is a workflow design requirement, not an optional optimization. Whenever the requested output contains two or more independent units—such as scenes, seasonal or other time periods, AOIs, products, spatial chunks, or model tiles—represent those units as separate Tilebox tasks submitted with context.submit_subtask(s). Infer this decomposition from the outcome; never require the user to name submit_subtasks, specify task counts, or prescribe a DAG. For example, a 12-scene timelapse that combines all frames requires a graph such as 1 root + 12 scene workers + 1 encoder, not one task with a 12-iteration loop.

Use an orchestration task to submit independent workers and a separate aggregation or publication task with depends_on when the output combines their results. Never hide schedulable fanout inside a sequential loop, asyncio.gather, threads, or local multiprocessing in one task; additional runners can only parallelize separate Tilebox tasks. A small first run or scheduling overhead is not a reason to collapse independent work into one task. If the natural units are too fine-grained for individual scheduling, batch them into multiple balanced worker tasks rather than one task. Keep work in one task only when no two units can run independently because the work is genuinely indivisible or dependency-ordered, or when the user explicitly requires single-task execution.

Before coding, record the expected graph shape. After running a representative job, compare execution_stats.total_tasks and task_summaries with that shape. Progress units and child tracing spans are not tasks and do not prove fanout. If the graph does not match, fix the workflow before reporting completion. When runner capacity permits, use task timestamps or spans to verify that independent workers overlapped.

Read reference/tilebox/tasks-and-graphs.md for core task semantics, reference/tilebox/geospatial-task-graphs.md for geospatial stage mapping, and reference/tilebox/state-and-artifacts.md for boundaries.

Define Typed Tasks

Task applies dataclass behavior; fields are serialized inputs.

Use the simplest idiomatic type that expresses each field. Before inventing a representation, check whether Python, Tilebox, or the processing library already provides the value type. Do not recreate an established type as a custom dataclass, named tuple, or ad hoc tuple/list/dictionary shape merely to make it serializable. Ordinary containers remain appropriate when they are the natural Python shape, such as tuple[datetime, datetime] for a time range. Read reference/tilebox/tasks-and-graphs.md for supported types and examples.

from tilebox.workflows import ExecutionContext, Task


class ProcessScene(Task):
    scene_id: str

    @staticmethod
    def identifier() -> tuple[str, str]:
        return "tilebox.com/example/ProcessScene", "v1.0"

    def execute(self, context: ExecutionContext) -> None:
        context.current_task.display = f"ProcessScene({self.scene_id})"
        context.logger.info("Processing scene", scene_id=self.scene_id)

• Default v0.0 identifiers are acceptable for prototypes. Stable identifiers return (name, vX.Y).
• Minor versions are forward-compatible; bump major for breaking input/behavior changes.
• Keep the complete serialized input at most 2048 bytes. Pass compact plain-Python or supported library values, IDs, keys, and small configuration—not arrays, dataframes, xarray datasets, large geometry, manifests, credentials, clients, open files, or local paths. Use context.job_cache or durable storage according to reference/tilebox/state-and-artifacts.md.
• Register every task class used by jobs with the runner.

Await Async IO Directly In Tasks

Define async def execute(self, context: ExecutionContext) -> None whenever a task uses async APIs, including tilebox.storage.aio, async HTTP clients, or bounded concurrent requests. Await those APIs directly; never wrap them in asyncio.run(...). The workflow executor awaits async execute methods while preserving normal failure, retry, tracing, and completion behavior.

import asyncio

from niquests import AsyncSession


class FetchMetadata(Task):
    urls: list[str]

    async def execute(self, context: ExecutionContext) -> None:
        async with AsyncSession() as client:
            responses = await asyncio.gather(
                *(client.get(url) for url in self.urls)
            )
        for response in responses:
            response.raise_for_status()
        context.logger.info("Metadata fetched", count=len(responses))

Choose client lifetime according to its event-loop behavior: create niquests.AsyncSession inside execute, while tilebox.storage.aio.Client may be reused across task executions. See reference/tilebox/tasks-and-graphs.md for details. Bound large request sets with a semaphore or bounded batches. Async concurrency is appropriate for multiple IO requests that belong to one task, such as the bands or byte ranges needed for one scene. It does not replace task-level fanout for independent scenes, AOIs, time periods, products, chunks, or model tiles. Keep runner.run_all() and runner.run_forever() in synchronous runner entrypoints rather than calling them from async code.

Build Simple Dynamic Graphs

class ProcessScenes(Task):
    scene_ids: list[str]

    def execute(self, context: ExecutionContext) -> None:
        context.current_task.display = f"ProcessScenes(n={len(self.scene_ids)})"
        workers = context.submit_subtasks(
            [ProcessScene(scene_id) for scene_id in self.scene_ids],
            max_retries=3,
        )
        context.submit_subtask(PublishSummary(), depends_on=workers)

• Use submit_subtask for one child and submit_subtasks for homogeneous batches.
• Pass returned handles to depends_on; prefer stage barriers over thousands of unique pairwise dependencies.
• Use optional=True only for non-critical work.
• Make side effects idempotent: deterministic keys, overwrite-safe writes, valid-output checks, or atomic commits. Retrying uses the original task input.

Progress And Observability

Set concise current_task.display labels before expensive work. For meaningful fanout, the submitting task calls context.progress(name).add(n) and each successful worker calls .done(1) after completing its represented unit. Totals and completions must match.

Use context.logger with structured fields and logger.bind for repeated context. In exception handlers, call logger.exception and re-raise. Wrap expensive IO/compute/publish phases in context.tracer.span(...) and add useful filter attributes. Configure console logging in the runner entrypoint, not task classes.

Dataset, Asset, And Storage Boundaries

• Query mechanics after source selection: reference/tilebox/datasets-and-datapoints.md.
• Asset decoding and metadata: reference/tilebox/assets.md.
• Asset byte access and bounded COG windows: reference/tilebox/storage-access.md.

Do not reconstruct provider paths, repeat storage API snippets, or embed source/provider setup. Apply scale/offset, nodata, masks, alignment, and reprojection explicitly. Keep downloads in the consuming task and pass durable IDs/keys—not local paths—to another task.

Dependencies And Operations

Declare runtime dependencies in pyproject.toml, including optional libraries whose types appear in task fields. Tilebox loads their serializers lazily but does not install those libraries. Run uv sync, avoid editable/local-path dependencies, and see reference/tilebox/dependencies-and-packaging.md. Use authoring for source and focused local checks only. Use tilebox-workflow-releases for init/build/deploy and tilebox-workflow-jobs for submission/wait/clusters.

Reference Routing

Tilebox Platform

Mixed Tilebox + Geospatial Recipes

Portable Geospatial

Verification Checklist

1. All submitted tasks are registered and identifiers/versions agree.
2. Task fields use idiomatic Python or existing library value types rather than structurally duplicating them in custom dataclasses or ad hoc containers.
3. Every serialized task input is at most 2048 bytes.
4. Cross-task data uses the right state/artifact boundary and deterministic keys.
5. Retryable execution is re-entrant and input-compatible.
6. High-fanout tasks have labels, useful progress, structured logs, and spans where warranted.
7. Dependencies sync and task modules import in the intended environment.
8. For scaffolded release projects, tilebox workflow build-release --debug --json succeeds after editing generated task code.
9. Non-trivial workflows use coherent package modules and keep the runner entrypoint limited to registration and runner-level configuration.
10. Substantive underlying processing functions have focused tests where useful; Tilebox task orchestration is verified through build/task discovery and a representative job rather than task-class unit tests.
11. Earth observation outputs have representative scientific validation, coverage/alignment checks, provenance, and stated limitations.
12. A representative job's actual task summaries match the planned root, fanout, and aggregation stages; progress counters or child spans are not treated as task fanout.