aetos/docs

Files

Vishesh 'ironeagle' Bangotra 406affcfcf feat: renumber lib reference into 4-pillar anatomy

Renumber the 15 reference docs into pillar groups (recipe -> anatomy):
- 01 Build Recipe (entry), 02 Composition Root
- domains: 10 Layering Rules
- contract: 20 Spec-First Routing, 21 CRUD Template, 22 Auth Integration
- interfaces/infra: 30 Infra Adapters
- UI: 40 UI: Auto-Rendered Admin & Custom Pages (new; @aetoskia packages,
  client template, x- catalog via react-openapi MCP module)
- pipelines: 50 Pipeline Scaffold, 51 Worker & Progress, 52 Human-in-the-Loop
- delivery/meta: 90 Drone Pipelines, 91 Versioning & Changelog,
  92 Genericity Matrix, 93 Library Registry, 94 Repository Inventory

MCP module names are unchanged (NN_ prefix stripped); all cross-references
(wiki, templates, hooks, registry) updated; lib index rewritten pillar-first;
build recipe bootstraps the client template. Bundle + HTML sites regenerated.

2026-10-19 17:42:55 +05:30

2.4 KiB

Raw Blame History

Pipeline Scaffold

Summary

Data-driven processing is expressed as a composition graph. The graph is built by a DI library; execution is driven by a DAG engine. Stages are pure and individually testable.

When to use a pipeline

Use a pipeline when a resource is produced by a multi-step transformation of an input (a document, a file, a message) rather than by a simple CRUD write.

Structure of a family

A family is one input shape (e.g. a file format or a message kind). Each family has the same skeleton:

pipelines/<family>/
    abc.py            # port contracts (stage interfaces + root interface)
    impl.py           # concrete stages + root implementation
    pipeline.yaml     # graph spec: nodes, edges, roots, schema
    resources/
        pipeline.yaml # runtime spec consumed by the loader
        patterns/*.yaml  # rule fragments (pattern registry)
    banks/<variant>.py   # optional variants that re-pin a few slots

Shared scaffolding

pipelines/shared/
    ports.py     # RegistryPort, FinalizePort, etc.
    loader.py    # build(root_cls) -> wires the graph + engine
    registry.py  # resolve_root(...) -> root class for a (kind, variant)

The loader

def build(root_cls, *, instances=None, config=None):
    # merge resources/patterns/*.yaml into a pattern registry
    # bind registry/config into each graph node by id
    # build the schema (state) class and the engine
    root = root_cls()
    root._engine = Engine(graph, state_cls, payload_cls)
    return root

The registry

A single lookup replaces hand-rolled if/else selection:

_FAMILIES = {
    "csv":  {"acme": {"heuristic": AcmeCsvPipeline}},
    "json": {"globex": {"heuristic": GlobexJsonPipeline}},
}

def resolve_root(kind, variant, mode="heuristic"):
    return _FAMILIES[kind][variant][mode]

Adding a new variant means adding one class and one registry entry.

Driving a pipeline

Stages are driven explicitly so intermediate state can be persisted:

pipeline = build(resolve_root(kind, variant))
content = pipeline.ingest.materialize(raw_bytes)
records = pipeline.reader.read(content)
items   = pipeline.normalize.build(records)

Testing

Because stages are pure, test each stage in isolation and test the assembled root against fixture inputs whose expected outputs are stored as golden files. A parity test replays fixtures headlessly.

2.4 KiB Raw Blame History