dagpipe executes pipelines composed of nodes connected in a directed acyclic graph (DAG). Each node receives an immutable State and optionally produces derived states for downstream nodes. Execution is deterministic: the same input state and pipeline always produce the same output states and lineage.
Doc model: this wiki is written for humans \u2014 how\u2011to guides, examples, and testing recipes. The authoritative API contracts live in the code (GSDFC docstrings) and the machine\u2011readable bundle under docs/mcp/.
State \u2014 states are never mutated; fork() creates childrenPayload \u2014 dot-path read/update over nested dataSchema validation \u2014 payloads and updates validated at constructionAsyncNode + run_async for I/O-bound stepsrun_stepsload_pipeline)From your internal PyPI:
pip install --extra-index-url https://$PYPI_USERNAME:$PYPI_PASSWORD@pip.aetoskia.com/simple dagpipe\nFrom local source:
pip install -e .\nfrom dagpipe import Payload, Schema, State, Graph, Engine, Node\n\nclass HelloNode(Node):\n id = \"hello\"\n\n def resolve(self, state):\n yield self.fork(state, payload_update={\"msg\": \"hello\"})\n\nclass MyState(State):\n schema = Schema({\"msg\": str | None})\n\ngraph = Graph()\ngraph.add_root(HelloNode())\n\nengine = Engine(graph)\nresults = engine.run(MyState(payload=Payload({})))\n\nassert results[0].get(\"msg\") == \"hello\"\nload_pipeline \u00b7 State & Data \u00b7 04 \u2013 Schema and payload Nested schemas and dot-path access \u00b7 05 \u2013 State fork and lineage Immutability, lineage, confidence \u00b7 Async & Steps \u00b7 06 \u2013 Async execution AsyncNode and run_async \u00b7 07 \u2013 Step-wise execution Resumable runs and progress hooks Best Practices Node design, state design, and gotchas Error Handling Library exceptions and where they surface Testing Mock-based quickstart"},{"location":"#related-resources","title":"\ud83d\udd17 Related Resources","text":"\u00a9 Aetoskia Internal \u2014 dagpipe 0.0.1
dagpipe is a small execution framework that propagates immutable State objects through Node units connected in a directed acyclic graph (DAG). It is designed for deterministic, observable, and resumable data-processing pipelines.
At runtime a pipeline is made of four cooperating layers:
\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 YAML (optional) \u2502\n\u2502 schema \u00b7 initial \u00b7 nodes \u00b7 graph \u2192 load_pipeline() \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u25bc\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 Graph (execution topology) \u2502\n\u2502 roots \u00b7 add_edge(child) \u00b7 children() \u00b7 parents() \u2502\n\u2502 acyclic \u2014 validated on every mutation \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u25bc\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 Engine (orchestration) \u2502\n\u2502 run / run_async / run_steps / run_steps_async \u2502\n\u2502 mode: linear sequence OR graph BFS \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u25bc\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 State \u00b7 Payload \u00b7 Schema (data plane) \u2502\n\u2502 immutable State \u00b7 fork() \u00b7 dot-path payload \u00b7 validation \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\nNode \u2014 a pure unit of work. Each node consumes one State and yields zero, one, or many derived State objects.Graph \u2014 the execution topology. It only stores connectivity; it never executes anything.Engine \u2014 the orchestrator. It accepts either a linear Sequence[Node] or a Graph and produces terminal states.State / Payload / Schema \u2014 the immutable data plane. Nodes read a State, fork new ones, and let the Schema guarantee shape.Node subclasses and a State subclass bound to a Schema. 2 You Build a Graph (order + add_edge) or a linear Sequence[Node]. 3 You Construct Engine(nodes_or_graph). 4 You Create a root State (MyState(payload=Payload(...))) \u2014 validated here. 5 Engine Executes roots first, then fans State out along edges (graph mode) or feeds each sequence element (linear mode). 6 Engine Collects terminal states \u2014 states from nodes with no children (or the last step in linear mode). Nodes that yield no states prune the branch \u2014 see use case 02.
"},{"location":"01_overview/#which-execution-model-should-you-use","title":"\u2696\ufe0f Which execution model should you use?","text":"Model Construct Use when Runs LinearEngine([a, b, c]) A fixed pipeline of steps, no branches run Graph Engine(Graph) Branching, merging, multiple roots run Async Engine + AsyncNode subclasses I/O-bound steps (HTTP, DB, files) run_async Steps Engine.run_steps(...) Progress bars, resume-after-interrupt run_steps A linear sequence is equivalent to a chain graph: each step receives every state its predecessor produced. Graph mode gives you explicit fan-out and fan-in.
"},{"location":"01_overview/#read-next","title":"\ud83d\udcda Read Next","text":"This page is the validated reference for the public API surface. For step-by-step recipes see the use cases index. For exact signatures and docstrings, see the library reference (docs/lib) or the MCP bundle (docs/mcp).
Node","text":"Base class for all execution nodes. It is an abstract base class: subclasses implement resolve() and declare an id (dotted snake_case).
from dagpipe import Node, State\n\nclass Increment(Node):\n id = \"math.increment\"\n\n def resolve(self, state: State):\n yield self.fork(state, payload_update={\"value\": state.get(\"value\") + 1})\nKey facts:
A() is A() is True. Subclasses that declare a custom __init__ get one instance per construction (use this to inject per-run dependencies).fork() \u2014 convenience wrapper around State.fork() that records the node ID into state history.resolve() must yield State \u2014 anything else raises TypeError.node_id_to_name(node_id) \u2014 converts entity.resolve.numeric_merchant into Entity \u203a Resolve \u203a Numeric Merchant.name \u2014 auto-derived from the ID unless set explicitly.AsyncNode","text":"Base class for asynchronous nodes. Subclasses implement resolve_async() (an async generator yielding State objects). See use case 06.
from dagpipe import AsyncNode\n\nclass FetchRemote(AsyncNode):\n id = \"web.fetch\"\n\n async def resolve_async(self, state: State):\n data = await fetch(state.get(\"url\"))\n yield self.fork(state, payload_update={\"body\": data})\nA sync engine that hits an AsyncNode treats it as a no-op (yields no states). Use Engine.run_async when the graph contains AsyncNodes.
Graph","text":"DAG topology container. Stores connectivity only \u2014 it never executes nodes.
from dagpipe import Graph\n\ngraph = Graph()\ngraph.add_root(a) # node with no parents\ngraph.add_edge(a, b) # b is a child of a\ngraph.add_edge(a, c)\ngraph.children(a) # (b, c)\ngraph.parents(b) # (a,)\ngraph.roots() # nodes with no incoming edges\ngraph.nodes() # all registered nodes\nValueError (self-edges too).add_edge / add_root.Engine","text":"Orchestrator that runs a linear Sequence[Node] or a Graph.
from dagpipe import Engine\n\nengine = Engine(graph) # or Engine([n1, n2, n3])\nresults = engine.run(root_state) # list[State] of terminal states\nresults_async = await engine.run_async(root_state)\nrun_steps / run_steps_async and a nodes property.State, Node, or Graph instances.State","text":"Immutable execution snapshot at one point in traversal. Subclass and bind a schema:
from dagpipe import State, Schema\n\nclass MyState(State):\n schema = Schema({\"value\": int, \"label\": str | None})\npayload against schema at construction (SchemaError on violation).fork() is the only supported mechanism for producing a new state. Use State.fork directly, or the Node.fork convenience wrapper.confidence, parent, depth, and history (ordered node-ID lineage) for observability.lineage() \u2014 root-to-this ordered tuple.get(key) / has(key) read dot-paths from the underlying Payload.Payload","text":"Immutable hierarchical container with dot-path access.
payload = Payload({\"user\": {\"address\": {\"city\": \"Mumbai\"}}})\npayload.get(\"user.address.city\") # 'Mumbai'\npayload.has(\"user.address.zip\") # False\npayload.update({\"user.address.zip\": 400001}) # new Payload, original untouched\npayload.keys() # ('user',)\npayload.as_dict() # read-only view of underlying mapping\nSchema","text":"Immutable hierarchical validation tree. Leaf nodes are types or PEP-604 unions; nested Schema instances describe nested structure.
AddressSchema = Schema({\"city\": str, \"zip\": int | None})\nUserSchema = Schema({\"name\": str, \"address\": AddressSchema})\n\nuser_payload = Payload({\"name\": \"Ada\", \"address\": {\"city\": \"London\"}})\nUserSchema.validate_payload(user_payload) # no-op on success\nUserSchema.validate_update({\"address.city\": \"Paris\"})\nvalidate_payload \u2014 full structure check.validate_update \u2014 path existence check for fork() updates.SchemaError","text":"Raised when payload data violates the declared schema: invalid structure, undefined path, or invalid type. See Error Handling.
"},{"location":"02_components/#declarative-pipelines","title":"\ud83d\udcdc Declarative Pipelines","text":""},{"location":"02_components/#pipeline","title":"Pipeline","text":"Dataclass wrapping engine, state_cls, and initial_payload. Executes with run(payload_override=None) and returns terminal states.
load_pipeline(path)","text":"Factory that builds a Pipeline from one YAML file:
version: 1\nschema:\n value: int\ninitial:\n value: 1\nnodes:\n step1:\n class: mymod.Increment\ngraph:\n roots:\n - step1\nSee use case 03 for the full walkthrough.
"},{"location":"02_components/#progress-types","title":"\ud83d\udd01 Progress Types","text":"StepResult \u2014 (index, node_id, states, completed) for one executed step.ProgressMessage \u2014 keyword-only, optional fields (lines, blocks, count, unit, raw_ocr_line, error, step, status) plus as_dict(). Passed to step hooks for progress reporting.Both are produced by Engine.run_steps / run_steps_async \u2014 see use case 07.
Conventions adopted across dagpipe pipelines. Follow these to keep graphs deterministic, observable, and debuggable.
"},{"location":"04_best_practices/#node-design","title":"\ud83e\uddf1 Node design","text":"State \u2014 fork a new one instead. Inputs are shared across branches, so mutation would corrupt siblings.return () in resolve kills the branch: children never run and this path contributes no terminal state.entity.resolve_numeric_merchant, text.normalize), mirroring their file.module location.__init__ when you need per-run dependencies (then inspect how instances are created \u2014 custom __init__ opts out of the singleton).Node.fork inside resolve so the node ID is recorded in history automatically.State validates its payload in __post_init__ \u2014 construct roots inside a factory with a clear error path.Schema-building layer and reuse it across state subclasses and YAML definitions.object sparingly. Free-form object fields defeat validation. Prefer explicit types or unions (str | None) wherever the contract is known.fork() updates are path-validated; adding a new path is a breaking change for existing payloads.if not state.get(...): return in resolve keeps downstream work minimal.Graph for dynamic topologies; use YAML only for static pipelines you can diff in review.AsyncNode.resolve_async; keep the engine generic.await run_async(...) for all-sync graphs; run is cheaper.resume_from is 0-based \u2014 persist last_index + 1 and resume from it.completed=False as \"no output\", not \"failure\".ProgressMessage through on_step hooks for tracing/progress bars without coupling the engine.State via a small factory (see the make_state fixture pattern in Testing) \u2014 one call per scenario.history, not intermediate prints.tmp_path fixture (as the integration suite does), keeping tests offline.state.payload or state.metadata Breaks immutability; shared across branches Building nodes with heavy __init__ Skips singleton reuse; couples pipeline to instance state Logging inside resolve Hard to test; pollutes output. Use on_step hooks instead Reusing one Graph across concurrent runs Assumes nodes are stateless \u2014 they must be"},{"location":"04_best_practices/#read-next","title":"\ud83d\udcda Read Next","text":"dagpipe raises a small, predictable set of exceptions. Know when they surface and how to react.
"},{"location":"06_error_handling/#exception-reference","title":"\ud83d\udccb Exception reference","text":"Exception Raised when WhereSchemaError Payload violates the declared schema State.__post_init__, Schema.validate_payload, Schema.validate_update, Schema._walk, Schema._check_type, Schema._validate_path TypeError resolve()/resolve_async() yields a non-State Node.run, AsyncNode.run_async TypeError Engine(...) receives a non-Node element or a non-Sequence/Graph Engine.__init__ TypeError Engine.run/run_async receives a non-State root Engine.run, Engine.run_async TypeError Graph.add_edge/add_root receives a non-Node Graph.add_edge, Graph.add_root TypeError A YAML node class path is not a Node subclass _load_nodes ValueError Adding an edge would create a cycle (incl. self-cycle) Graph.add_edge ValueError Node.id is not valid dotted snake_case Node.clean_id_and_name RuntimeError Engine mode is corrupt (should never happen) Engine.run"},{"location":"06_error_handling/#schema-failures","title":"\ud83e\uddca Schema failures","text":"from dagpipe import Payload, Schema, SchemaError, State\n\nclass UserState(State):\n schema = Schema({\"name\": str, \"zip\": int | None})\n\ntry:\n UserState(payload=Payload({\"name\": 42, \"zip\": \"abc\"}))\nexcept SchemaError as e:\n print(e) # Path 'name' must be str\nGuidance:
SchemaError during construction means the root can never exist \u2014 fail the request/concept early.SchemaError during fork(payload_update=...) means the update path was not declared \u2014 fix the schema or the update, don't swallow it.from dagpipe import Graph\n\ng = Graph()\ng.add_edge(a, b)\ntry:\n g.add_edge(b, a) # would create a cycle\nexcept ValueError as e:\n print(e) # Adding edge b \u2192 a would create cycle\nCycles (including a \u2192 a) always raise immediately at wiring time, never at execution. Treat them as a design error: redraw the topology.
class BadNode(Node):\n id = \"bad.output\"\n\n def resolve(self, state):\n yield \"not a state\" # \u2190 not a State\n\nengine = Engine([BadNode()])\ntry:\n engine.run(root)\nexcept TypeError as e:\n print(e) # bad.output.resolve must yield State, got <class 'str'>\nThis check is applied per yielded object, so one bad element in a generator fails the whole run.
run_steps doesn't raise when a node produces no output \u2014 it reports completed=False:
for step in engine.run_steps(root):\n if not step.completed:\n log.info(\"%s produced no state (branch pruned)\", step.node_id)\nTreat completed=False as a signal, not an exception. See use case 07.
SchemaError into a user-facing 4xx (API) or a descriptive FAILED status (batch jobs).ValueError from add_edge propagate during setup \u2014 it's a programming error you want to see in CI.State/Payload errors mid-pipeline and continue with a fallback payload \u2014 the pipeline was designed to enforce invariants.dagpipe's own test suite runs without any external services \u2014 nodes are the only things you need to fake. The pattern: small schema/state factories, then assert on terminal states and history.
"},{"location":"07_testing_example/#setup-state-factories","title":"\ud83c\udfaf Setup: state factories","text":"Import from dagpipe and declare tiny schemas/states, exactly like the project's conftest.py:
# conftest.py\nimport pytest\nfrom dagpipe import Payload, Schema, State\n\nTestSchema = Schema({\"value\": object})\nItemSchema = Schema({\n \"raw\": object,\n \"entity\": Schema({\"candidate\": str | None, \"name\": str | None}),\n \"tags\": list | None,\n})\n\nclass TestState(State):\n schema = TestSchema\n\nclass ItemState(State):\n schema = ItemSchema\n\n@pytest.fixture\ndef make_state():\n def _make(value=1):\n return TestState(payload=Payload({\"value\": value}))\n return _make\n\n@pytest.fixture\ndef make_item_state():\n def _make(**payload):\n base = {\"raw\": object(), \"tags\": []}\n base.update(payload)\n return ItemState(payload=Payload(base))\n return _make\nfrom dagpipe import Node\n\nclass Increment(Node):\n id = \"test.increment\"\n\n def resolve(self, state):\n yield self.fork(state, payload_update={\"value\": state.get(\"value\") + 1})\ndef test_increment(make_state):\n node = Increment()\n (result,) = node.run(make_state(value=1))\n\n assert result.get(\"value\") == 2\n assert result.depth == 1\n assert result.history == (\"test.increment\",)\nfrom dagpipe import Engine, Graph\n\ndef test_branching_and_merge(make_item_state):\n class Extract(Node):\n id = \"entity.extract\"\n def resolve(self, state):\n yield self.fork(state, payload_update={\"entity.candidate\": \"amzn\"})\n yield self.fork(state, payload_update={\"entity.candidate\": \"flipkart\"})\n\n class Resolve(Node):\n id = \"entity.resolve\"\n def resolve(self, state):\n if state.get(\"entity.candidate\"):\n yield self.fork(state, payload_update={\n \"entity.name\": state.get(\"entity.candidate\").upper()})\n\n graph = Graph()\n graph.add_edge(Extract(), Resolve())\n\n results = Engine(graph).run(make_item_state())\n names = sorted(s.get(\"entity.name\") for s in results)\n\n assert names == [\"AMZN\", \"FLIPKART\"]\n assert all(len(s.history) == 2 for s in results) # extract \u2192 resolve\nimport pytest\nfrom dagpipe import Payload, Schema, SchemaError, State\n\ndef test_invalid_payload_raises():\n class S(State):\n schema = Schema({\"name\": str})\n\n with pytest.raises(SchemaError, match=\"must be str\"):\n S(payload=Payload({\"name\": 42}))\n\ndef test_bad_yield_raises_type_error(make_state):\n class Bad(Node):\n id = \"bad.output\"\n def resolve(self, state):\n yield \"nope\"\n\n with pytest.raises(TypeError, match=\"must yield State\"):\n Bad().run(make_state())\nUse a tiny importable node module + tmp_path, as the integration suite does:
def test_yaml_pipeline(tmp_path, monkeypatch):\n module_dir = tmp_path / \"testnodes\"\n module_dir.mkdir()\n (module_dir / \"__init__.py\").write_text(\"\"\"\nfrom dagpipe.node import Node\n\nclass Increment(Node):\n id = \"test.increment\"\n def resolve(self, state):\n yield self.fork(state, payload_update={\"value\": state.get(\"value\") + 1})\n\"\"\")\n\n monkeypatch.syspath_prepend(tmp_path)\n\n yaml_file = tmp_path / \"p.yaml\"\n yaml_file.write_text(\"\"\"\nschema:\n value: int\ninitial:\n value: 1\nnodes:\n inc:\n class: testnodes.Increment\ngraph:\n roots:\n - inc\n\"\"\")\n\n from dagpipe import load_pipeline\n results = load_pipeline(yaml_file).run()\n\n assert results[0].get(\"value\") == 2\nhistory \u2014 they capture the full behavior without brittle intermediate assertions.make_item_state(entity={\"candidate\": \"x\"}) beats hand-building Payload dicts everywhere.run (assert on it explicitly).pytest -q\nA fixed sequence of steps: each step receives every state the previous step produced, and the last step's states are the terminal results.
"},{"location":"03_use_cases/01_basic_linear_pipeline/#goal","title":"\ud83c\udfaf Goal","text":"Normalize and sum a list of numbers with a three-node chain: Clean \u2192 Sum \u2192 Announce.
from dagpipe import Node, State\n\nclass Clean(Node):\n id = \"math.clean\"\n\n def resolve(self, state: State):\n numbers = [n for n in state.get(\"numbers\", []) if isinstance(n, (int, float))]\n yield self.fork(state, payload_update={\"numbers\": numbers})\n\nclass Sum(Node):\n id = \"math.sum\"\n\n def resolve(self, state: State):\n yield self.fork(\n state,\n payload_update={\"total\": sum(state.get(\"numbers\", []))},\n )\n\nclass Announce(Node):\n id = \"math.announce\"\n\n def resolve(self, state: State):\n print(f\"total={state.get('total')}\")\n yield self.fork(state, payload_update={\"announced\": True})\nfrom dagpipe import Engine, Payload, Schema, State\n\nclass CalcState(State):\n schema = Schema({\n \"numbers\": list,\n \"total\": int | float | None,\n \"announced\": bool | None,\n })\n\nengine = Engine([Clean(), Sum(), Announce()])\nresults = engine.run(CalcState(payload=Payload({\"numbers\": [1, \"x\", 2, 3.5]})))\n\nstate = results[0]\nassert state.get(\"numbers\") == [1, 2, 3.5]\nassert state.get(\"total\") == 6.5\nassert state.get(\"announced\") is True\nassert len(state.history) == 3 # clean \u2192 sum \u2192 announce\nassert state.depth == 3\nstates = [root].Clean yielded one new state \u2192 states = [clean_state].Sum consumed it \u2192 states = [sum_state].Announce consumed it \u2192 states = [announce_state] (terminal).If any node yields zero states, the engine breaks and returns an empty list \u2014 that is how a pipeline can terminate early.
"},{"location":"03_use_cases/01_basic_linear_pipeline/#tips","title":"\ud83d\udca1 Tips","text":"Node.fork over State.fork inside resolve() so the node ID lands in history automatically.A DAG lets states fan out (one node, many children) and merge (many nodes, one child). Both are plain State flow \u2014 the engine handles the plumbing.
Resolve an entity candidate into a name through two alternative resolvers, then tag the result \u2014 while a Direct path bypasses resolution entirely.
\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 entity.extract \u2502 \u2500\u2500\u252c\u2500\u2500\u25b6 resolve_amzn \u2500\u2500\u2510\n\u2502 \u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2510\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 \u251c\u2500\u2500\u25b6 entity.tag\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n\u2502 entity.direct \u2502 \u2500\u2500\u253c\u2500\u2500\u25b6 entity.direct \u2500\u2524 (no children \u2192 terminal)\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 \u2502 \n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n\u2502 entity.kill \u2502 \u2500\u2500\u2518 (yields nothing \u2192 branch pruned)\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\nfrom dagpipe import Graph, Node\n\nclass ExtractCandidate(Node):\n id = \"entity.extract\"\n\n def resolve(self, state):\n # fan-out: two candidate states for one input\n yield self.fork(state, payload_update={\"entity.candidate\": \"amzn\"})\n yield self.fork(state, payload_update={\"entity.candidate\": \"flipkart\"})\n\nclass ResolveAmazon(Node):\n id = \"entity.resolve_amzn\"\n\n def resolve(self, state):\n if state.get(\"entity.candidate\") == \"amzn\":\n yield self.fork(state, payload_update={\"entity.name\": \"Amazon\"})\n\nclass ResolveFlipkart(Node):\n id = \"entity.resolve_fk\"\n\n def resolve(self, state):\n if state.get(\"entity.candidate\") == \"flipkart\":\n yield self.fork(state, payload_update={\"entity.name\": \"Flipkart\"})\n\nclass TagNode(Node):\n id = \"entity.tag\"\n\n def resolve(self, state):\n # only runs when a name was resolved\n if state.get(\"entity.name\"):\n tags = state.get(\"tags\") or []\n yield self.fork(state, payload_update={\"tags\": tags + [\"resolved\"]})\n\nclass DirectNode(Node):\n id = \"entity.direct\"\n\n def resolve(self, state):\n yield self.fork(state, payload_update={\"entity.name\": \"Direct\"})\n\nclass KillNode(Node):\n id = \"entity.kill\"\n\n def resolve(self, state):\n return () # branch pruned: no states, no children visited\ngraph = Graph()\ngraph.add_edge(ExtractCandidate(), ResolveAmazon())\ngraph.add_edge(ExtractCandidate(), ResolveFlipkart())\ngraph.add_edge(ResolveAmazon(), TagNode())\ngraph.add_edge(ResolveFlipkart(), TagNode())\ngraph.add_root(DirectNode())\ngraph.add_root(KillNode())\nfrom dagpipe import Engine, Payload, Schema, State\n\nclass ItemState(State):\n schema = Schema({\n \"raw\": object,\n \"entity\": Schema({\"candidate\": str | None, \"name\": str | None}),\n \"tags\": list | None,\n })\n\nroot = ItemState(payload=Payload({\"raw\": object(), \"tags\": []}))\nresults = Engine(graph).run(root)\n\nnames = sorted(s.get(\"entity.name\") for s in results)\nassert names == [\"Amazon\", \"Direct\", \"Flipkart\"]\n\ntagged = {s.get(\"entity.name\"): s.get(\"tags\") for s in results}\nassert tagged[\"Amazon\"] == [\"resolved\"]\nassert tagged[\"Flipkart\"] == [\"resolved\"]\nassert tagged[\"Direct\"] is None # direct path never reached tag\nState.ExtractCandidate forked two states \u2192 each resolver ran once per branch.ResolveAmazon/ResolveFlipkart each produced at most one state (the wrong candidate branch pruned itself by yielding nothing).TagNode merged both successful branches and appended \"resolved\".DirectNode had no children \u2192 its state is terminal immediately.KillNode yielded nothing \u2192 pruned, never contributes.State \u2014 they don't interact.TagNode; the tag step runs once per incoming state.ValueError on a self/indirect cycle at add_edge time.load_pipeline() builds a complete pipeline \u2014 schema, state subclass, nodes, graph, engine, and initial payload \u2014 from one declarative YAML file.
Recreate the entity-resolution DAG from use case 02 purely in YAML.
"},{"location":"03_use_cases/03_yaml_pipeline/#the-yaml-file","title":"\ud83d\udcc4 The YAML file","text":"version: 1\n\nschema:\n raw: object\n tags: list | None\n entity:\n candidate: str | None\n name: str | None\n\ninitial:\n raw: {}\n\nnodes:\n extract:\n class: testnodes.ExtractCandidate\n resolve_amzn:\n class: testnodes.ResolveAmazon\n resolve_fk:\n class: testnodes.ResolveFlipkart\n tag:\n class: testnodes.TagNode\n direct:\n class: testnodes.DirectNode\n kill:\n class: testnodes.KillNode\n\ngraph:\n roots:\n - extract\n - direct\n - kill\n edges:\n extract:\n - resolve_amzn\n - resolve_fk\n resolve_amzn:\n - tag\n resolve_fk:\n - tag\nNode class paths are fully qualified (module.ClassName). The modules are imported with importlib at load time, so they must be importable from your runtime environment.
from dagpipe import load_pipeline\n\npipeline = load_pipeline(\"pipeline.yaml\")\nresults = pipeline.run()\n\nnames = sorted(s.get(\"entity.name\") for s in results)\nassert names == [\"Amazon\", \"Direct\", \"Flipkart\"]\nLeaf values in schema: map to Python types:
str str int / float int / float bool bool dict / list dict / list bytes bytes object object None NoneType str | None str \\| None (PEP-604 union) Nested schema blocks become nested Schema instances.
Pipeline.run(payload_override=None) merges override values on top of initial: using Payload.update \u2014 the root payload is the override, not the default:
results = pipeline.run(payload_override={\"raw\": \"custom\"})\nclass intent \u2014 they become the dict keys used by roots/edges.graph.roots/graph.edges must exist in nodes: or you get a KeyError.Schema declares the allowed shape of a Payload, and is enforced whenever a State is constructed or forked.
Validate a document with a nested address object, using unions for optional fields.
"},{"location":"03_use_cases/04_schema_and_payload/#the-schema","title":"\ud83e\uddf1 The schema","text":"from dagpipe import Payload, Schema\n\nAddressSchema = Schema({\n \"city\": str,\n \"zip\": int | None,\n})\n\nUserSchema = Schema({\n \"name\": str,\n \"address\": AddressSchema,\n})\n\nDeepItemSchema = Schema({\n \"raw\": object,\n \"user\": UserSchema,\n})\nOuter keys inside a Schema tree point at one of:
str, int, ...) \u2014 single allowed typestr | None) \u2014 any of the membersSchema \u2014 the value must be a mapping following that sub-schemaobject \u2014 unrestrictedok = Payload({\n \"raw\": {\"anything\": \"goes\"},\n \"user\": {\n \"name\": \"John\",\n \"address\": {\"city\": \"Mumbai\", \"zip\": 400001},\n },\n})\nDeepItemSchema.validate_payload(ok) # no-op\nvalidate_payload raises SchemaError for any of:
\"phone\" not declared Invalid path 'phone' not defined in schema Wrong scalar type \"name\": 42 Path 'user.name' must be str Union violation \"zip\": \"abc\" Path 'user.address.zip' must be one of (int, NoneType) Non-container for nested schema \"user\": \"joe\" Path 'user' must be a container from dagpipe import SchemaError\n\ntry:\n DeepItemSchema.validate_payload(\n Payload({\"raw\": {}, \"user\": {\"name\": 42, \"address\": {\"city\": \"M\"}}})\n )\nexcept SchemaError as e:\n print(e) # Path 'user.name' must be str\nState.fork(payload_update=...) calls validate_update, which checks that every dot-path in the update is declared \u2014 before anything is copied:
UserSchema.validate_update({\"address.city\": \"Paris\"}) # OK\nUserSchema.validate_update({\"address.country\": \"FR\"}) # SchemaError: not in schema\nPayload gives typed, immutable access to nested values:
p = Payload({\"user\": {\"address\": {\"city\": \"Mumbai\", \"zip\": None}}})\n\np.get(\"user.address.city\") # 'Mumbai'\np.get(\"user.address.zip\") # None\np.has(\"user.address.zip\") # True\np.has(\"user.phone\") # False\np.update({\"user.phone\": \"123\"}) # Schema may reject this on the State level!\nPayload.update itself is schema-unaware \u2014 the shape check belongs to the State/Schema layer. Use State.fork when you want validation.
Schema is immutable and reusable across all state instances.SchemaError raised during __post_init__ means a State can never exist with an invalid payload \u2014 catch it early at construction.str | None for 3.10+, Union[str, None] otherwise).State is immutable: nodes never modify it. They produce new states via fork(), which carries lineage, confidence, and history forward.
Walk a branch tree and inspect how states are related, how confidence compounds, and how history records the exact node path taken.
from dagpipe import Payload, Schema, State, Node\n\nclass ScoreState(State):\n schema = Schema({\"score\": int | float, \"label\": str | None})\n\nclass AddBonus(Node):\n id = \"score.bonus\"\n\n def resolve(self, state):\n yield self.fork(state, confidence_delta=0.1,\n payload_update={\"score\": state.get(\"score\") + 10})\nNode.fork passes node_id=self.id, so history records the point of derivation automatically.
root = ScoreState(payload=Payload({\"score\": 50}))\n\ns1 = root.fork(\n payload_update={\"score\": 60},\n confidence_delta=0.1,\n node_id=\"score.bonus\",\n metadata_update={\"source\": \"manual\"},\n)\n\ns2 = s1.fork(payload_update={\"label\": \"high\"})\nroot s1 s2 depth 0 1 2 score 50 60 60 confidence 1.0 1.1 1.1 history () ('score.bonus',) ('score.bonus',) parent None root s1"},{"location":"03_use_cases/05_state_fork_and_lineage/#lineage","title":"\ud83d\udd0d Lineage","text":"lineage() walks parents back to the root, root-first:
states = s2.lineage()\nassert states == (root, s1, s2)\nassert states[0] is root\nassert states[-1] is s2\nassert root.get(\"label\") is None # s2's update didn't touch root\nassert s1.get(\"label\") is None # fork never mutates its input\nassert root.confidence == 1.0 # confidence travels forward only\nassert root is s1.parent # parent is by reference\nconfidence_delta is added (parent.confidence + delta), so a chain of bonuses compounds: 1.0 \u2192 1.1 \u2192 1.2.metadata_update is shallow-merged into metadata \u2014 untouched states share the same metadata dict (no copy unless a fork changes it).payload_update keys are dot-paths; they are validated by validate_update against the state's schema before copying.Node.fork in resolve() \u2014 it records node_id into history for free.payload_update, confidence_delta, node_id, metadata_update) \u2014 there is no positional API.ScoreState(payload=...)) validates against schema in __post_init__ \u2014 you can't create an invalid root.Payload.update).AsyncNode subclasses run I/O-bound steps (HTTP, DB, file reads) via Engine.run_async. Sync and async nodes can be mixed in the same graph \u2014 the engine dispatches each node to the right path automatically.
Fetch a document from a remote service, then run a synchronous post-processor.
import asyncio\nfrom dagpipe import AsyncNode, Node, Schema, State, Payload, Graph, Engine\n\nclass FetchDoc(AsyncNode):\n id = \"web.fetch\"\n\n async def resolve_async(self, state: State):\n body = await fetch_body(state.get(\"url\")) # your async I/O\n yield self.fork(state, payload_update={\"body\": body, \"fetched\": True})\n\nclass Summarize(Node):\n id = \"text.summarize\"\n\n def resolve(self, state: State):\n words = len(state.get(\"body\", \"\").split())\n yield self.fork(state, payload_update={\"word_count\": words})\n\nclass DocState(State):\n schema = Schema({\n \"url\": str,\n \"body\": str | None,\n \"fetched\": bool | None,\n \"word_count\": int | None,\n })\nasync def main():\n graph = Graph()\n graph.add_edge(FetchDoc(), Summarize())\n\n engine = Engine(graph)\n results = await engine.run_async(DocState(payload=Payload({\"url\": \"https://example.com\"})))\n\n assert results[0].get(\"fetched\") is True\n assert results[0].get(\"word_count\") > 0\n\nasyncio.run(main())\nEngine.run_async checks each node at execution time:
node is AsyncNode ? \u2192 await node.run_async(state) # resolve_async()\n \u2192 node.run(state) # resolve()\nAsyncNode.resolve() is a no-op (returns ()), so a sync engine (run) treats async nodes as if they pruned the branch.AsyncNode.resolve_async() may be a plain async def returning an iterable, or an async generator \u2014 both are supported.run_async validates that yielded objects are State (raises TypeError otherwise), exactly like the sync path.run_steps_async pairs with async nodes for progress + resume:
async for step in engine.run_steps_async(root):\n print(step.index, step.node_id, len(step.states), step.completed)\nasyncio.gather inside a single wrapper node.run_async on an engine whose nodes are all sync \u2014 you pay event loop overhead for nothing; run is fine there.resolve_async, not the engine \u2014 keep the engine generic.Engine.run_steps runs a pipeline one step at a time, yielding a StepResult per executed node. It supports resume (skip already-done steps) and progress hooks (report started / completed / skipped).
Run a three-node pipeline, observe progress, and resume a partial run.
"},{"location":"03_use_cases/07_step_execution/#setup","title":"\ud83e\uddf1 Setup","text":"from dagpipe import Engine, Node, Payload, Schema, State\n\nclass StepA(Node):\n id = \"qa.step_a\"\n def resolve(self, state):\n yield self.fork(state, payload_update={\"a\": True})\n\nclass StepB(Node):\n id = \"qa.step_b\"\n def resolve(self, state):\n if not state.get(\"a\"):\n return # prune branch if precondition unmet\n yield self.fork(state, payload_update={\"b\": True})\n\nclass StepC(Node):\n id = \"qa.step_c\"\n def resolve(self, state):\n yield self.fork(state, payload_update={\"c\": True})\n\nclass QState(State):\n schema = Schema({\"a\": bool | None, \"b\": bool | None, \"c\": bool | None})\n\nengine = Engine([StepA(), StepB(), StepC()])\nroot = QState(payload=Payload({}))\nfor step in engine.run_steps(root):\n print(f\"#{step.index} {step.node_id:10} completed={step.completed} states={len(step.states)}\")\nOutput:
#0 qa.step_a completed=True states=1\n#1 qa.step_b completed=True states=1\n#2 qa.step_c completed=True states=1\nEach StepResult exposes:
index 0-based step ordinal (resume_from is relative to this) node_id The node that ran in this step states States produced by this step completed True if any state was produced"},{"location":"03_use_cases/07_step_execution/#resume-after-interruption","title":"\ud83d\udd0d Resume after interruption","text":"If you already processed #0 and #1, skip ahead:
for step in engine.run_steps(root, resume_from=2):\n print(step.node_id) # only qa.step_c\nPass on_step (or set it at engine construction) to be notified per step:
engine = Engine([StepA(), StepB(), StepC()],\n on_step=lambda step, status, msg: print(step, status))\n\nfor step in engine.run_steps(root):\n ...\nOutput:
qa.step_a started\nqa.step_a completed\nqa.step_b started\nqa.step_b completed\n...\nA ProgressMessage can be passed as the third argument for richer progress (lines, blocks, count, unit, error, ...). Use as_dict() to surface it in your logging.
The async variant run_steps_async mirrors this API with on_step: AsyncStepHook \u2014 see use case 06.
In graph mode, run_steps derives a deterministic topological order (DFS post-order, roots first), one StepResult per node. A node that yields no states marks that step completed=False and the branch stops there.
resume_from for idempotent retries: record the last consumed index (e.g. in a DB), then resume the next run from index + 1.completed=False means \"no states produced\" (pruned/static node) \u2014 treat it as \"no work available\", not \"error\".