Skip to content

🗺️ Library Overview

dagpipe is a small execution framework that propagates immutable State objects through Node units connected in a directed acyclic graph (DAG). It is designed for deterministic, observable, and resumable data-processing pipelines.

🏗️ Architecture

At runtime a pipeline is made of four cooperating layers:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
┌────────────────────────────────────────────────────────────┐
│                       YAML (optional)                       │
│     schema · initial · nodes · graph  →  load_pipeline()    │
└──────────────────────────────┬─────────────────────────────┘
                               ▼
┌────────────────────────────────────────────────────────────┐
│               Graph (execution topology)                    │
│   roots · add_edge(child) · children() · parents()          │
│   acyclic — validated on every mutation                     │
└──────────────────────────────┬─────────────────────────────┘
                               ▼
┌────────────────────────────────────────────────────────────┐
│               Engine (orchestration)                        │
│   run / run_async / run_steps / run_steps_async             │
│   mode: linear sequence  OR  graph BFS                      │
└──────────────────────────────┬─────────────────────────────┘
                               ▼
┌────────────────────────────────────────────────────────────┐
│       State · Payload · Schema (data plane)                 │
│   immutable State · fork() · dot-path payload · validation  │
└────────────────────────────────────────────────────────────┘
  • Node — a pure unit of work. Each node consumes one State and yields zero, one, or many derived State objects.
  • Graph — the execution topology. It only stores connectivity; it never executes anything.
  • Engine — the orchestrator. It accepts either a linear Sequence[Node] or a Graph and produces terminal states.
  • State / Payload / Schema — the immutable data plane. Nodes read a State, fork new ones, and let the Schema guarantee shape.

🔄 Lifecycle of a run

Step Who What happens
1 You Define Node subclasses and a State subclass bound to a Schema.
2 You Build a Graph (order + add_edge) or a linear Sequence[Node].
3 You Construct Engine(nodes_or_graph).
4 You Create a root State (MyState(payload=Payload(...))) — validated here.
5 Engine Executes roots first, then fans State out along edges (graph mode) or feeds each sequence element (linear mode).
6 Engine Collects terminal states — states from nodes with no children (or the last step in linear mode).

Nodes that yield no states prune the branch — see use case 02.

⚖️ Which execution model should you use?

Model Construct Use when Runs
Linear Engine([a, b, c]) A fixed pipeline of steps, no branches run
Graph Engine(Graph) Branching, merging, multiple roots run
Async Engine + AsyncNode subclasses I/O-bound steps (HTTP, DB, files) run_async
Steps Engine.run_steps(...) Progress bars, resume-after-interrupt run_steps

A linear sequence is equivalent to a chain graph: each step receives every state its predecessor produced. Graph mode gives you explicit fan-out and fan-in.