Vishesh 'ironeagle' Bangotra
7aadae1736
Enables dagpipe wiki on the docs hub alongside lib and mcp, removes stale nested lib pages, and picks up regenerated GSDFC docstrings in the MCP bundle.
1 line
64 KiB
JSON
1 line
64 KiB
JSON
{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"\u26d3\ufe0f dagpipe \u2014 Directed Acyclic Graph Execution for Deterministic State","text":"<p><code>dagpipe</code> executes pipelines composed of nodes connected in a directed acyclic graph (DAG). Each node receives an immutable <code>State</code> and optionally produces derived states for downstream nodes. Execution is deterministic: the same input state and pipeline always produce the same output states and lineage.</p> <p>Doc model: this wiki is written for humans \u2014 how\u2011to guides, examples, and testing recipes. The authoritative API contracts live in the code (GSDFC docstrings) and the machine\u2011readable bundle under <code>docs/mcp/</code>.</p>"},{"location":"#key-features","title":"\ud83d\ude80 Key Features","text":"<ul> <li>\ud83e\uddf1 Node-based execution \u2014 each node is a pure, deterministic unit of work</li> <li>\ud83d\udd00 DAG topology \u2014 branching, merging, and multiple roots out of the box</li> <li>\ud83e\uddca Immutable <code>State</code> \u2014 states are never mutated; <code>fork()</code> creates children</li> <li>\ud83d\uddc2\ufe0f Hierarchical <code>Payload</code> \u2014 dot-path read/update over nested data</li> <li>\ud83d\udcd0 <code>Schema</code> validation \u2014 payloads and updates validated at construction</li> <li>\u26a1 Async execution \u2014 <code>AsyncNode</code> + <code>run_async</code> for I/O-bound steps</li> <li>\ud83e\ude9c Step-wise runs \u2014 resumable, progress-hooked execution via <code>run_steps</code></li> <li>\ud83d\udcdc YAML pipelines \u2014 declare schema, nodes, and graph in one file (<code>load_pipeline</code>)</li> </ul>"},{"location":"#installation","title":"\ud83d\udce6 Installation","text":"<p>From your internal PyPI:</p> <pre><code>pip install --extra-index-url https://$PYPI_USERNAME:$PYPI_PASSWORD@pip.aetoskia.com/simple dagpipe\n</code></pre> <p>From local source:</p> <pre><code>pip install -e .\n</code></pre>"},{"location":"#quick-start","title":"\u26a1 Quick Start","text":"<pre><code>from dagpipe import Payload, Schema, State, Graph, Engine, Node\n\nclass HelloNode(Node):\n id = \"hello\"\n\n def resolve(self, state):\n yield self.fork(state, payload_update={\"msg\": \"hello\"})\n\nclass MyState(State):\n schema = Schema({\"msg\": str | None})\n\ngraph = Graph()\ngraph.add_root(HelloNode())\n\nengine = Engine(graph)\nresults = engine.run(MyState(payload=Payload({})))\n\nassert results[0].get(\"msg\") == \"hello\"\n</code></pre>"},{"location":"#documentation-structure","title":"\ud83d\udcc1 Documentation Structure","text":"Section Description Overview Architecture, lifecycle, and execution model decision table Core Components Validated reference for the public API surface Use cases Step-by-step recipes, grouped by theme \u00b7 Getting Started \u00b7 01 \u2013 Basic linear pipeline Sequential multi-node pipeline \u00b7 02 \u2013 Graph execution Branching, merging, multiple roots \u00b7 03 \u2013 YAML pipelines Declarative pipelines via <code>load_pipeline</code> \u00b7 State & Data \u00b7 04 \u2013 Schema and payload Nested schemas and dot-path access \u00b7 05 \u2013 State fork and lineage Immutability, lineage, confidence \u00b7 Async & Steps \u00b7 06 \u2013 Async execution <code>AsyncNode</code> and <code>run_async</code> \u00b7 07 \u2013 Step-wise execution Resumable runs and progress hooks Best Practices Node design, state design, and gotchas Error Handling Library exceptions and where they surface Testing Mock-based quickstart"},{"location":"#related-resources","title":"\ud83d\udd17 Related Resources","text":"<ul> <li>Source Code: Gitea Repository</li> <li>Internal PyPI: pip.aetoskia.com/simple/dagpipe</li> <li>CI: Builds and publishes tagged releases, gated on black / ruff / mypy / pytest.</li> </ul> <p>\u00a9 Aetoskia Internal \u2014 <code>dagpipe</code> 0.0.1</p>"},{"location":"01_overview/","title":"\ud83d\uddfa\ufe0f Library Overview","text":"<p><code>dagpipe</code> is a small execution framework that propagates immutable <code>State</code> objects through <code>Node</code> units connected in a directed acyclic graph (DAG). It is designed for deterministic, observable, and resumable data-processing pipelines.</p>"},{"location":"01_overview/#architecture","title":"\ud83c\udfd7\ufe0f Architecture","text":"<p>At runtime a pipeline is made of four cooperating layers:</p> <pre><code>\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 YAML (optional) \u2502\n\u2502 schema \u00b7 initial \u00b7 nodes \u00b7 graph \u2192 load_pipeline() \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u25bc\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 Graph (execution topology) \u2502\n\u2502 roots \u00b7 add_edge(child) \u00b7 children() \u00b7 parents() \u2502\n\u2502 acyclic \u2014 validated on every mutation \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u25bc\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 Engine (orchestration) \u2502\n\u2502 run / run_async / run_steps / run_steps_async \u2502\n\u2502 mode: linear sequence OR graph BFS \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u25bc\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 State \u00b7 Payload \u00b7 Schema (data plane) \u2502\n\u2502 immutable State \u00b7 fork() \u00b7 dot-path payload \u00b7 validation \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n</code></pre> <ul> <li><code>Node</code> \u2014 a pure unit of work. Each node consumes one <code>State</code> and yields zero, one, or many derived <code>State</code> objects.</li> <li><code>Graph</code> \u2014 the execution topology. It only stores connectivity; it never executes anything.</li> <li><code>Engine</code> \u2014 the orchestrator. It accepts either a linear <code>Sequence[Node]</code> or a <code>Graph</code> and produces terminal states.</li> <li><code>State</code> / <code>Payload</code> / <code>Schema</code> \u2014 the immutable data plane. Nodes read a <code>State</code>, fork new ones, and let the <code>Schema</code> guarantee shape.</li> </ul>"},{"location":"01_overview/#lifecycle-of-a-run","title":"\ud83d\udd04 Lifecycle of a run","text":"Step Who What happens 1 You Define <code>Node</code> subclasses and a <code>State</code> subclass bound to a <code>Schema</code>. 2 You Build a <code>Graph</code> (order + <code>add_edge</code>) or a linear <code>Sequence[Node]</code>. 3 You Construct <code>Engine(nodes_or_graph)</code>. 4 You Create a root <code>State</code> (<code>MyState(payload=Payload(...))</code>) \u2014 validated here. 5 Engine Executes roots first, then fans <code>State</code> out along edges (graph mode) or feeds each sequence element (linear mode). 6 Engine Collects terminal states \u2014 states from nodes with no children (or the last step in linear mode). <p>Nodes that yield no states prune the branch \u2014 see use case 02.</p>"},{"location":"01_overview/#which-execution-model-should-you-use","title":"\u2696\ufe0f Which execution model should you use?","text":"Model Construct Use when Runs Linear <code>Engine([a, b, c])</code> A fixed pipeline of steps, no branches <code>run</code> Graph <code>Engine(Graph)</code> Branching, merging, multiple roots <code>run</code> Async <code>Engine</code> + <code>AsyncNode</code> subclasses I/O-bound steps (HTTP, DB, files) <code>run_async</code> Steps <code>Engine.run_steps(...)</code> Progress bars, resume-after-interrupt <code>run_steps</code> <p>A linear sequence is equivalent to a chain graph: each step receives every state its predecessor produced. Graph mode gives you explicit fan-out and fan-in.</p>"},{"location":"01_overview/#read-next","title":"\ud83d\udcda Read Next","text":"<ul> <li>Core Components \u2014 the validated public API surface.</li> <li>Use case 01 \u2014 the minimal pipeline.</li> <li>Use case 02 \u2014 branching and merging.</li> <li>Best Practices \u2014 team-wide conventions.</li> </ul>"},{"location":"02_components/","title":"\ud83e\udde9 Core Components","text":"<p>This page is the validated reference for the public API surface. For step-by-step recipes see the use cases index. For exact signatures and docstrings, see the library reference (<code>docs/lib</code>) or the MCP bundle (<code>docs/mcp</code>).</p>"},{"location":"02_components/#execution-core","title":"\u26a1 Execution Core","text":""},{"location":"02_components/#node","title":"<code>Node</code>","text":"<p>Base class for all execution nodes. It is an abstract base class: subclasses implement <code>resolve()</code> and declare an <code>id</code> (dotted snake_case).</p> <pre><code>from dagpipe import Node, State\n\nclass Increment(Node):\n id = \"math.increment\"\n\n def resolve(self, state: State):\n yield self.fork(state, payload_update={\"value\": state.get(\"value\") + 1})\n</code></pre> <p>Key facts:</p> <ul> <li>Singleton per subclass \u2014 stateless subclasses share one instance. <code>A() is A()</code> is <code>True</code>. Subclasses that declare a custom <code>__init__</code> get one instance per construction (use this to inject per-run dependencies).</li> <li><code>fork()</code> \u2014 convenience wrapper around <code>State.fork()</code> that records the node ID into state history.</li> <li><code>resolve()</code> must yield <code>State</code> \u2014 anything else raises <code>TypeError</code>.</li> <li><code>node_id_to_name(node_id)</code> \u2014 converts <code>entity.resolve.numeric_merchant</code> into <code>Entity \u203a Resolve \u203a Numeric Merchant</code>.</li> <li><code>name</code> \u2014 auto-derived from the ID unless set explicitly.</li> </ul>"},{"location":"02_components/#asyncnode","title":"<code>AsyncNode</code>","text":"<p>Base class for asynchronous nodes. Subclasses implement <code>resolve_async()</code> (an async generator yielding <code>State</code> objects). See use case 06.</p> <pre><code>from dagpipe import AsyncNode\n\nclass FetchRemote(AsyncNode):\n id = \"web.fetch\"\n\n async def resolve_async(self, state: State):\n data = await fetch(state.get(\"url\"))\n yield self.fork(state, payload_update={\"body\": data})\n</code></pre> <p>A sync engine that hits an <code>AsyncNode</code> treats it as a no-op (yields no states). Use <code>Engine.run_async</code> when the graph contains <code>AsyncNode</code>s.</p>"},{"location":"02_components/#graph","title":"<code>Graph</code>","text":"<p>DAG topology container. Stores connectivity only \u2014 it never executes nodes.</p> <pre><code>from dagpipe import Graph\n\ngraph = Graph()\ngraph.add_root(a) # node with no parents\ngraph.add_edge(a, b) # b is a child of a\ngraph.add_edge(a, c)\ngraph.children(a) # (b, c)\ngraph.parents(b) # (a,)\ngraph.roots() # nodes with no incoming edges\ngraph.nodes() # all registered nodes\n</code></pre> <ul> <li>Cycle detection is automatic: adding an edge that would create a cycle raises <code>ValueError</code> (self-edges too).</li> <li>Nodes are registered implicitly by <code>add_edge</code> / <code>add_root</code>.</li> <li>The graph is mutable during construction but treated as immutable at runtime.</li> </ul>"},{"location":"02_components/#engine","title":"<code>Engine</code>","text":"<p>Orchestrator that runs a linear <code>Sequence[Node]</code> or a <code>Graph</code>.</p> <pre><code>from dagpipe import Engine\n\nengine = Engine(graph) # or Engine([n1, n2, n3])\nresults = engine.run(root_state) # list[State] of terminal states\nresults_async = await engine.run_async(root_state)\n</code></pre> <ul> <li>Linear mode \u2014 each node feeds every downstream state it receives, in order.</li> <li>Graph mode \u2014 BFS traversal from all roots; states fan out along edges and are collected as terminal states at nodes with no children.</li> <li>Also exposes <code>run_steps</code> / <code>run_steps_async</code> and a <code>nodes</code> property.</li> <li>Never mutates <code>State</code>, <code>Node</code>, or <code>Graph</code> instances.</li> </ul>"},{"location":"02_components/#state-data","title":"\ud83e\uddca State & Data","text":""},{"location":"02_components/#state","title":"<code>State</code>","text":"<p>Immutable execution snapshot at one point in traversal. Subclass and bind a <code>schema</code>:</p> <pre><code>from dagpipe import State, Schema\n\nclass MyState(State):\n schema = Schema({\"value\": int, \"label\": str | None})\n</code></pre> <ul> <li>Validates its <code>payload</code> against <code>schema</code> at construction (<code>SchemaError</code> on violation).</li> <li><code>fork()</code> is the only supported mechanism for producing a new state. Use <code>State.fork</code> directly, or the <code>Node.fork</code> convenience wrapper.</li> <li>Tracks <code>confidence</code>, <code>parent</code>, <code>depth</code>, and <code>history</code> (ordered node-ID lineage) for observability.</li> <li><code>lineage()</code> \u2014 root-to-this ordered tuple.</li> <li><code>get(key)</code> / <code>has(key)</code> read dot-paths from the underlying <code>Payload</code>.</li> </ul>"},{"location":"02_components/#payload","title":"<code>Payload</code>","text":"<p>Immutable hierarchical container with dot-path access.</p> <pre><code>payload = Payload({\"user\": {\"address\": {\"city\": \"Mumbai\"}}})\npayload.get(\"user.address.city\") # 'Mumbai'\npayload.has(\"user.address.zip\") # False\npayload.update({\"user.address.zip\": 400001}) # new Payload, original untouched\npayload.keys() # ('user',)\npayload.as_dict() # read-only view of underlying mapping\n</code></pre> <ul> <li>Updates are atomic and cheap \u2014 only modified branches are copied.</li> </ul>"},{"location":"02_components/#schema","title":"<code>Schema</code>","text":"<p>Immutable hierarchical validation tree. Leaf nodes are types or PEP-604 unions; nested <code>Schema</code> instances describe nested structure.</p> <pre><code>AddressSchema = Schema({\"city\": str, \"zip\": int | None})\nUserSchema = Schema({\"name\": str, \"address\": AddressSchema})\n\nuser_payload = Payload({\"name\": \"Ada\", \"address\": {\"city\": \"London\"}})\nUserSchema.validate_payload(user_payload) # no-op on success\nUserSchema.validate_update({\"address.city\": \"Paris\"})\n</code></pre> <ul> <li><code>validate_payload</code> \u2014 full structure check.</li> <li><code>validate_update</code> \u2014 path existence check for <code>fork()</code> updates.</li> </ul>"},{"location":"02_components/#schemaerror","title":"<code>SchemaError</code>","text":"<p>Raised when payload data violates the declared schema: invalid structure, undefined path, or invalid type. See Error Handling.</p>"},{"location":"02_components/#declarative-pipelines","title":"\ud83d\udcdc Declarative Pipelines","text":""},{"location":"02_components/#pipeline","title":"<code>Pipeline</code>","text":"<p>Dataclass wrapping <code>engine</code>, <code>state_cls</code>, and <code>initial_payload</code>. Executes with <code>run(payload_override=None)</code> and returns terminal states.</p>"},{"location":"02_components/#load_pipelinepath","title":"<code>load_pipeline(path)</code>","text":"<p>Factory that builds a <code>Pipeline</code> from one YAML file:</p> <pre><code>version: 1\nschema:\n value: int\ninitial:\n value: 1\nnodes:\n step1:\n class: mymod.Increment\ngraph:\n roots:\n - step1\n</code></pre> <p>See use case 03 for the full walkthrough.</p>"},{"location":"02_components/#progress-types","title":"\ud83d\udd01 Progress Types","text":"<ul> <li><code>StepResult</code> \u2014 <code>(index, node_id, states, completed)</code> for one executed step.</li> <li><code>ProgressMessage</code> \u2014 keyword-only, optional fields (<code>lines</code>, <code>blocks</code>, <code>count</code>, <code>unit</code>, <code>raw_ocr_line</code>, <code>error</code>, <code>step</code>, <code>status</code>) plus <code>as_dict()</code>. Passed to step hooks for progress reporting.</li> </ul> <p>Both are produced by <code>Engine.run_steps</code> / <code>run_steps_async</code> \u2014 see use case 07.</p>"},{"location":"02_components/#read-next","title":"\ud83d\udcda Read Next","text":"<ul> <li>Use case 04 \u2013 Schema and payload</li> <li>Use case 05 \u2013 State fork and lineage</li> <li>Best Practices</li> </ul>"},{"location":"04_best_practices/","title":"\ud83e\uddf9 Best Practices","text":"<p>Conventions adopted across dagpipe pipelines. Follow these to keep graphs deterministic, observable, and debuggable.</p>"},{"location":"04_best_practices/#node-design","title":"\ud83e\uddf1 Node design","text":"<ul> <li>Keep nodes pure. A node must never mutate its input <code>State</code> \u2014 fork a new one instead. Inputs are shared across branches, so mutation would corrupt siblings.</li> <li>Yield zero states to prune. <code>return ()</code> in <code>resolve</code> kills the branch: children never run and this path contributes no terminal state.</li> <li>One responsibility per node. Name nodes for what they do (<code>entity.resolve_numeric_merchant</code>, <code>text.normalize</code>), mirroring their <code>file.module</code> location.</li> <li>Prefer stateless subclasses. The singleton-per-subclass model means a stateless node can be reused everywhere for free. Only declare <code>__init__</code> when you need per-run dependencies (then inspect how instances are created \u2014 custom <code>__init__</code> opts out of the singleton).</li> <li>Use <code>Node.fork</code> inside <code>resolve</code> so the node ID is recorded in <code>history</code> automatically.</li> </ul>"},{"location":"04_best_practices/#state-schema-design","title":"\ud83c\udfaf State & schema design","text":"<ul> <li>Validate as early as possible. A root <code>State</code> validates its payload in <code>__post_init__</code> \u2014 construct roots inside a factory with a clear error path.</li> <li>Declare the schema once at the <code>Schema</code>-building layer and reuse it across state subclasses and YAML definitions.</li> <li>Use <code>object</code> sparingly. Free-form <code>object</code> fields defeat validation. Prefer explicit types or unions (<code>str | None</code>) wherever the contract is known.</li> <li>Evolve schemas deliberately. <code>fork()</code> updates are path-validated; adding a new path is a breaking change for existing payloads.</li> </ul>"},{"location":"04_best_practices/#graph-patterns","title":"\ud83d\udd00 Graph patterns","text":"<ul> <li>Multiple roots are independent. They receive the same root state and share nothing \u2014 don't expect cross-communication between roots.</li> <li>Merge = shared child. To join branches, point all of them at the same node. The merge node runs once per incoming state.</li> <li>Prune early with guards. <code>if not state.get(...): return</code> in <code>resolve</code> keeps downstream work minimal.</li> <li>Keep graphs acyclic by construction. Cycle detection exists, but a cycle is always a design bug. Lay out your graph like a DAG from the start.</li> <li>Prefer programmatic <code>Graph</code> for dynamic topologies; use YAML only for static pipelines you can diff in review.</li> </ul>"},{"location":"04_best_practices/#async-guidance","title":"\u26a1 Async guidance","text":"<ul> <li>Mix sync and async nodes freely \u2014 dispatch is per node, not per engine.</li> <li>Wrap per-node I/O in <code>AsyncNode.resolve_async</code>; keep the engine generic.</li> <li>Don't <code>await run_async(...)</code> for all-sync graphs; <code>run</code> is cheaper.</li> </ul>"},{"location":"04_best_practices/#step-wise-guidance","title":"\ud83e\ude9c Step-wise guidance","text":"<ul> <li><code>resume_from</code> is 0-based \u2014 persist <code>last_index + 1</code> and resume from it.</li> <li>Treat <code>completed=False</code> as \"no output\", not \"failure\".</li> <li>Route <code>ProgressMessage</code> through <code>on_step</code> hooks for tracing/progress bars without coupling the engine.</li> </ul>"},{"location":"04_best_practices/#testability","title":"\ud83e\uddea Testability","text":"<ul> <li>Build synthetic <code>State</code> via a small factory (see the <code>make_state</code> fixture pattern in Testing) \u2014 one call per scenario.</li> <li>Assert on terminal states and their <code>history</code>, not intermediate prints.</li> <li>For YAML pipelines, test with a tiny importable node module and a <code>tmp_path</code> fixture (as the integration suite does), keeping tests offline.</li> </ul>"},{"location":"04_best_practices/#anti-patterns","title":"\u274c Anti-patterns","text":"Pattern Why it's wrong Mutating <code>state.payload</code> or <code>state.metadata</code> Breaks immutability; shared across branches Building nodes with heavy <code>__init__</code> Skips singleton reuse; couples pipeline to instance state Logging inside <code>resolve</code> Hard to test; pollutes output. Use <code>on_step</code> hooks instead Reusing one <code>Graph</code> across concurrent runs Assumes nodes are stateless \u2014 they must be"},{"location":"04_best_practices/#read-next","title":"\ud83d\udcda Read Next","text":"<ul> <li>Error Handling \u2014 the exceptions you'll actually see.</li> <li>07 \u2013 Step-wise execution \u2014 resume & hooks.</li> <li>07 \u2013 Testing example \u2014 the fixture pattern.</li> </ul>"},{"location":"06_error_handling/","title":"\u26a0\ufe0f Error Handling","text":"<p>dagpipe raises a small, predictable set of exceptions. Know when they surface and how to react.</p>"},{"location":"06_error_handling/#exception-reference","title":"\ud83d\udccb Exception reference","text":"Exception Raised when Where <code>SchemaError</code> Payload violates the declared schema <code>State.__post_init__</code>, <code>Schema.validate_payload</code>, <code>Schema.validate_update</code>, <code>Schema._walk</code>, <code>Schema._check_type</code>, <code>Schema._validate_path</code> <code>TypeError</code> <code>resolve()</code>/<code>resolve_async()</code> yields a non-<code>State</code> <code>Node.run</code>, <code>AsyncNode.run_async</code> <code>TypeError</code> <code>Engine(...)</code> receives a non-<code>Node</code> element or a non-<code>Sequence</code>/<code>Graph</code> <code>Engine.__init__</code> <code>TypeError</code> <code>Engine.run</code>/<code>run_async</code> receives a non-<code>State</code> root <code>Engine.run</code>, <code>Engine.run_async</code> <code>TypeError</code> <code>Graph.add_edge</code>/<code>add_root</code> receives a non-<code>Node</code> <code>Graph.add_edge</code>, <code>Graph.add_root</code> <code>TypeError</code> A YAML node class path is not a <code>Node</code> subclass <code>_load_nodes</code> <code>ValueError</code> Adding an edge would create a cycle (incl. self-cycle) <code>Graph.add_edge</code> <code>ValueError</code> <code>Node.id</code> is not valid dotted snake_case <code>Node.clean_id_and_name</code> <code>RuntimeError</code> Engine mode is corrupt (should never happen) <code>Engine.run</code>"},{"location":"06_error_handling/#schema-failures","title":"\ud83e\uddca Schema failures","text":"<pre><code>from dagpipe import Payload, Schema, SchemaError, State\n\nclass UserState(State):\n schema = Schema({\"name\": str, \"zip\": int | None})\n\ntry:\n UserState(payload=Payload({\"name\": 42, \"zip\": \"abc\"}))\nexcept SchemaError as e:\n print(e) # Path 'name' must be str\n</code></pre> <p>Guidance:</p> <ul> <li>A <code>SchemaError</code> during construction means the root can never exist \u2014 fail the request/concept early.</li> <li><code>SchemaError</code> during <code>fork(payload_update=...)</code> means the update path was not declared \u2014 fix the schema or the update, don't swallow it.</li> </ul>"},{"location":"06_error_handling/#cycle-failures","title":"\ud83d\udd01 Cycle failures","text":"<pre><code>from dagpipe import Graph\n\ng = Graph()\ng.add_edge(a, b)\ntry:\n g.add_edge(b, a) # would create a cycle\nexcept ValueError as e:\n print(e) # Adding edge b \u2192 a would create cycle\n</code></pre> <p>Cycles (including <code>a \u2192 a</code>) always raise immediately at wiring time, never at execution. Treat them as a design error: redraw the topology.</p>"},{"location":"06_error_handling/#bad-node-output","title":"\ud83e\uddec Bad node output","text":"<pre><code>class BadNode(Node):\n id = \"bad.output\"\n\n def resolve(self, state):\n yield \"not a state\" # \u2190 not a State\n\nengine = Engine([BadNode()])\ntry:\n engine.run(root)\nexcept TypeError as e:\n print(e) # bad.output.resolve must yield State, got <class 'str'>\n</code></pre> <p>This check is applied per yielded object, so one bad element in a generator fails the whole <code>run</code>.</p>"},{"location":"06_error_handling/#step-wise-status-not-errors","title":"\ud83e\ude9c Step-wise status, not errors","text":"<p><code>run_steps</code> doesn't raise when a node produces no output \u2014 it reports <code>completed=False</code>:</p> <pre><code>for step in engine.run_steps(root):\n if not step.completed:\n log.info(\"%s produced no state (branch pruned)\", step.node_id)\n</code></pre> <p>Treat <code>completed=False</code> as a signal, not an exception. See use case 07.</p>"},{"location":"06_error_handling/#handling-strategies","title":"\ud83d\udca1 Handling strategies","text":"<ul> <li>At the boundary: construct states/pipelines in a wrapper that converts <code>SchemaError</code> into a user-facing <code>4xx</code> (API) or a descriptive <code>FAILED</code> status (batch jobs).</li> <li>At the graph: let <code>ValueError</code> from <code>add_edge</code> propagate during setup \u2014 it's a programming error you want to see in CI.</li> <li>Never catch <code>State</code>/<code>Payload</code> errors mid-pipeline and continue with a fallback payload \u2014 the pipeline was designed to enforce invariants.</li> </ul>"},{"location":"06_error_handling/#read-next","title":"\ud83d\udcda Read Next","text":"<ul> <li>Best Practices \u2014 design rules that avoid these errors.</li> <li>04 \u2013 Schema and payload \u2014 validation details.</li> <li>07 \u2013 Testing example \u2014 asserting on error paths.</li> </ul>"},{"location":"07_testing_example/","title":"\ud83e\uddea Testing Example","text":"<p>dagpipe's own test suite runs without any external services \u2014 nodes are the only things you need to fake. The pattern: small schema/state factories, then assert on terminal states and history.</p>"},{"location":"07_testing_example/#setup-state-factories","title":"\ud83c\udfaf Setup: state factories","text":"<p>Import from <code>dagpipe</code> and declare tiny schemas/states, exactly like the project's <code>conftest.py</code>:</p> <pre><code># conftest.py\nimport pytest\nfrom dagpipe import Payload, Schema, State\n\nTestSchema = Schema({\"value\": object})\nItemSchema = Schema({\n \"raw\": object,\n \"entity\": Schema({\"candidate\": str | None, \"name\": str | None}),\n \"tags\": list | None,\n})\n\nclass TestState(State):\n schema = TestSchema\n\nclass ItemState(State):\n schema = ItemSchema\n\n@pytest.fixture\ndef make_state():\n def _make(value=1):\n return TestState(payload=Payload({\"value\": value}))\n return _make\n\n@pytest.fixture\ndef make_item_state():\n def _make(**payload):\n base = {\"raw\": object(), \"tags\": []}\n base.update(payload)\n return ItemState(payload=Payload(base))\n return _make\n</code></pre>"},{"location":"07_testing_example/#a-node-to-test","title":"\ud83c\udfd7\ufe0f A node to test","text":"<pre><code>from dagpipe import Node\n\nclass Increment(Node):\n id = \"test.increment\"\n\n def resolve(self, state):\n yield self.fork(state, payload_update={\"value\": state.get(\"value\") + 1})\n</code></pre>"},{"location":"07_testing_example/#single-node-test","title":"\u2705 Single-node test","text":"<pre><code>def test_increment(make_state):\n node = Increment()\n (result,) = node.run(make_state(value=1))\n\n assert result.get(\"value\") == 2\n assert result.depth == 1\n assert result.history == (\"test.increment\",)\n</code></pre>"},{"location":"07_testing_example/#graph-behavior-test","title":"\ud83d\udd00 Graph behavior test","text":"<pre><code>from dagpipe import Engine, Graph\n\ndef test_branching_and_merge(make_item_state):\n class Extract(Node):\n id = \"entity.extract\"\n def resolve(self, state):\n yield self.fork(state, payload_update={\"entity.candidate\": \"amzn\"})\n yield self.fork(state, payload_update={\"entity.candidate\": \"flipkart\"})\n\n class Resolve(Node):\n id = \"entity.resolve\"\n def resolve(self, state):\n if state.get(\"entity.candidate\"):\n yield self.fork(state, payload_update={\n \"entity.name\": state.get(\"entity.candidate\").upper()})\n\n graph = Graph()\n graph.add_edge(Extract(), Resolve())\n\n results = Engine(graph).run(make_item_state())\n names = sorted(s.get(\"entity.name\") for s in results)\n\n assert names == [\"AMZN\", \"FLIPKART\"]\n assert all(len(s.history) == 2 for s in results) # extract \u2192 resolve\n</code></pre>"},{"location":"07_testing_example/#error-path-tests","title":"\u26a0\ufe0f Error-path tests","text":"<pre><code>import pytest\nfrom dagpipe import Payload, Schema, SchemaError, State\n\ndef test_invalid_payload_raises():\n class S(State):\n schema = Schema({\"name\": str})\n\n with pytest.raises(SchemaError, match=\"must be str\"):\n S(payload=Payload({\"name\": 42}))\n\ndef test_bad_yield_raises_type_error(make_state):\n class Bad(Node):\n id = \"bad.output\"\n def resolve(self, state):\n yield \"nope\"\n\n with pytest.raises(TypeError, match=\"must yield State\"):\n Bad().run(make_state())\n</code></pre>"},{"location":"07_testing_example/#testing-yaml-pipelines-offline","title":"\ud83e\uddf1 Testing YAML pipelines offline","text":"<p>Use a tiny importable node module + <code>tmp_path</code>, as the integration suite does:</p> <pre><code>def test_yaml_pipeline(tmp_path, monkeypatch):\n module_dir = tmp_path / \"testnodes\"\n module_dir.mkdir()\n (module_dir / \"__init__.py\").write_text(\"\"\"\nfrom dagpipe.node import Node\n\nclass Increment(Node):\n id = \"test.increment\"\n def resolve(self, state):\n yield self.fork(state, payload_update={\"value\": state.get(\"value\") + 1})\n\"\"\")\n\n monkeypatch.syspath_prepend(tmp_path)\n\n yaml_file = tmp_path / \"p.yaml\"\n yaml_file.write_text(\"\"\"\nschema:\n value: int\ninitial:\n value: 1\nnodes:\n inc:\n class: testnodes.Increment\ngraph:\n roots:\n - inc\n\"\"\")\n\n from dagpipe import load_pipeline\n results = load_pipeline(yaml_file).run()\n\n assert results[0].get(\"value\") == 2\n</code></pre>"},{"location":"07_testing_example/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Assert on terminal states + <code>history</code> \u2014 they capture the full behavior without brittle intermediate assertions.</li> <li>Factory fixtures keep tests readable: <code>make_item_state(entity={\"candidate\": \"x\"})</code> beats hand-building <code>Payload</code> dicts everywhere.</li> <li>Guard against side effects \u2014 nodes must be pure; tests catch leaks when the input state is still unchanged after <code>run</code> (assert on it explicitly).</li> <li>Run the suite with one command from the repo root:</li> </ul> <pre><code>pytest -q\n</code></pre>"},{"location":"07_testing_example/#read-next","title":"\ud83d\udcda Read Next","text":"<ul> <li>Best Practices \u2014 conventions these tests lock in.</li> <li>06 \u2013 Error Handling \u2014 the exceptions you assert on.</li> <li>02 \u2013 Graph execution \u2014 the topology under test.</li> </ul>"},{"location":"03_use_cases/01_basic_linear_pipeline/","title":"01 \u2013 Basic Linear Pipeline","text":"<p>A fixed sequence of steps: each step receives every state the previous step produced, and the last step's states are the terminal results.</p>"},{"location":"03_use_cases/01_basic_linear_pipeline/#goal","title":"\ud83c\udfaf Goal","text":"<p>Normalize and sum a list of numbers with a three-node chain: <code>Clean \u2192 Sum \u2192 Announce</code>.</p>"},{"location":"03_use_cases/01_basic_linear_pipeline/#nodes","title":"\ud83e\uddf1 Nodes","text":"<pre><code>from dagpipe import Node, State\n\nclass Clean(Node):\n id = \"math.clean\"\n\n def resolve(self, state: State):\n numbers = [n for n in state.get(\"numbers\", []) if isinstance(n, (int, float))]\n yield self.fork(state, payload_update={\"numbers\": numbers})\n\nclass Sum(Node):\n id = \"math.sum\"\n\n def resolve(self, state: State):\n yield self.fork(\n state,\n payload_update={\"total\": sum(state.get(\"numbers\", []))},\n )\n\nclass Announce(Node):\n id = \"math.announce\"\n\n def resolve(self, state: State):\n print(f\"total={state.get('total')}\")\n yield self.fork(state, payload_update={\"announced\": True})\n</code></pre>"},{"location":"03_use_cases/01_basic_linear_pipeline/#run-it","title":"\ud83c\udfc3 Run it","text":"<pre><code>from dagpipe import Engine, Payload, Schema, State\n\nclass CalcState(State):\n schema = Schema({\n \"numbers\": list,\n \"total\": int | float | None,\n \"announced\": bool | None,\n })\n\nengine = Engine([Clean(), Sum(), Announce()])\nresults = engine.run(CalcState(payload=Payload({\"numbers\": [1, \"x\", 2, 3.5]})))\n\nstate = results[0]\nassert state.get(\"numbers\") == [1, 2, 3.5]\nassert state.get(\"total\") == 6.5\nassert state.get(\"announced\") is True\nassert len(state.history) == 3 # clean \u2192 sum \u2192 announce\nassert state.depth == 3\n</code></pre>"},{"location":"03_use_cases/01_basic_linear_pipeline/#what-just-happened","title":"\ud83d\udd0d What just happened","text":"<ul> <li>The engine seeded <code>states = [root]</code>.</li> <li><code>Clean</code> yielded one new state \u2192 <code>states = [clean_state]</code>.</li> <li><code>Sum</code> consumed it \u2192 <code>states = [sum_state]</code>.</li> <li><code>Announce</code> consumed it \u2192 <code>states = [announce_state]</code> (terminal).</li> </ul> <p>If any node yields zero states, the engine breaks and returns an empty list \u2014 that is how a pipeline can terminate early.</p>"},{"location":"03_use_cases/01_basic_linear_pipeline/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Order in the sequence is the execution order \u2014 keep stateful order in mind.</li> <li>Every node sees all states produced by the previous node. If a node forks twice, the next step runs once per fork.</li> <li>Prefer <code>Node.fork</code> over <code>State.fork</code> inside <code>resolve()</code> so the node ID lands in <code>history</code> automatically.</li> </ul>"},{"location":"03_use_cases/01_basic_linear_pipeline/#related","title":"\ud83d\udcda Related","text":"<ul> <li>02 \u2013 Graph execution \u00b7 05 \u2013 State fork and lineage \u00b7 04 \u2013 Best Practices</li> </ul>"},{"location":"03_use_cases/02_graph_execution/","title":"02 \u2013 Graph Execution","text":"<p>A DAG lets states fan out (one node, many children) and merge (many nodes, one child). Both are plain <code>State</code> flow \u2014 the engine handles the plumbing.</p>"},{"location":"03_use_cases/02_graph_execution/#goal","title":"\ud83c\udfaf Goal","text":"<p>Resolve an entity candidate into a name through two alternative resolvers, then tag the result \u2014 while a <code>Direct</code> path bypasses resolution entirely.</p> <pre><code>\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 entity.extract \u2502 \u2500\u2500\u252c\u2500\u2500\u25b6 resolve_amzn \u2500\u2500\u2510\n\u2502 \u2502 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2510\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 \u251c\u2500\u2500\u25b6 entity.tag\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n\u2502 entity.direct \u2502 \u2500\u2500\u253c\u2500\u2500\u25b6 entity.direct \u2500\u2524 (no children \u2192 terminal)\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502 \u2502 \n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n\u2502 entity.kill \u2502 \u2500\u2500\u2518 (yields nothing \u2192 branch pruned)\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n</code></pre>"},{"location":"03_use_cases/02_graph_execution/#nodes","title":"\ud83e\uddf1 Nodes","text":"<pre><code>from dagpipe import Graph, Node\n\nclass ExtractCandidate(Node):\n id = \"entity.extract\"\n\n def resolve(self, state):\n # fan-out: two candidate states for one input\n yield self.fork(state, payload_update={\"entity.candidate\": \"amzn\"})\n yield self.fork(state, payload_update={\"entity.candidate\": \"flipkart\"})\n\nclass ResolveAmazon(Node):\n id = \"entity.resolve_amzn\"\n\n def resolve(self, state):\n if state.get(\"entity.candidate\") == \"amzn\":\n yield self.fork(state, payload_update={\"entity.name\": \"Amazon\"})\n\nclass ResolveFlipkart(Node):\n id = \"entity.resolve_fk\"\n\n def resolve(self, state):\n if state.get(\"entity.candidate\") == \"flipkart\":\n yield self.fork(state, payload_update={\"entity.name\": \"Flipkart\"})\n\nclass TagNode(Node):\n id = \"entity.tag\"\n\n def resolve(self, state):\n # only runs when a name was resolved\n if state.get(\"entity.name\"):\n tags = state.get(\"tags\") or []\n yield self.fork(state, payload_update={\"tags\": tags + [\"resolved\"]})\n\nclass DirectNode(Node):\n id = \"entity.direct\"\n\n def resolve(self, state):\n yield self.fork(state, payload_update={\"entity.name\": \"Direct\"})\n\nclass KillNode(Node):\n id = \"entity.kill\"\n\n def resolve(self, state):\n return () # branch pruned: no states, no children visited\n</code></pre>"},{"location":"03_use_cases/02_graph_execution/#wire-the-graph","title":"\ud83c\udfd7\ufe0f Wire the graph","text":"<pre><code>graph = Graph()\ngraph.add_edge(ExtractCandidate(), ResolveAmazon())\ngraph.add_edge(ExtractCandidate(), ResolveFlipkart())\ngraph.add_edge(ResolveAmazon(), TagNode())\ngraph.add_edge(ResolveFlipkart(), TagNode())\ngraph.add_root(DirectNode())\ngraph.add_root(KillNode())\n</code></pre>"},{"location":"03_use_cases/02_graph_execution/#run-it","title":"\ud83c\udfc3 Run it","text":"<pre><code>from dagpipe import Engine, Payload, Schema, State\n\nclass ItemState(State):\n schema = Schema({\n \"raw\": object,\n \"entity\": Schema({\"candidate\": str | None, \"name\": str | None}),\n \"tags\": list | None,\n })\n\nroot = ItemState(payload=Payload({\"raw\": object(), \"tags\": []}))\nresults = Engine(graph).run(root)\n\nnames = sorted(s.get(\"entity.name\") for s in results)\nassert names == [\"Amazon\", \"Direct\", \"Flipkart\"]\n\ntagged = {s.get(\"entity.name\"): s.get(\"tags\") for s in results}\nassert tagged[\"Amazon\"] == [\"resolved\"]\nassert tagged[\"Flipkart\"] == [\"resolved\"]\nassert tagged[\"Direct\"] is None # direct path never reached tag\n</code></pre>"},{"location":"03_use_cases/02_graph_execution/#what-just-happened","title":"\ud83d\udd0d What just happened","text":"<ul> <li>The engine enqueued all roots with the same root <code>State</code>.</li> <li><code>ExtractCandidate</code> forked two states \u2192 each resolver ran once per branch.</li> <li><code>ResolveAmazon</code>/<code>ResolveFlipkart</code> each produced at most one state (the wrong candidate branch pruned itself by yielding nothing).</li> <li><code>TagNode</code> merged both successful branches and appended <code>\"resolved\"</code>.</li> <li><code>DirectNode</code> had no children \u2192 its state is terminal immediately.</li> <li><code>KillNode</code> yielded nothing \u2192 pruned, never contributes.</li> </ul>"},{"location":"03_use_cases/02_graph_execution/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Root states are shared. Every root receives the same initial <code>State</code> \u2014 they don't interact.</li> <li>Merge = shared child. Both resolvers point at <code>TagNode</code>; the tag step runs once per incoming state.</li> <li>Pruning happens naturally: yield nothing and the branch dies.</li> <li>Cycle detection is automatic \u2014 <code>ValueError</code> on a self/indirect cycle at <code>add_edge</code> time.</li> </ul>"},{"location":"03_use_cases/02_graph_execution/#related","title":"\ud83d\udcda Related","text":"<ul> <li>01 \u2013 Basic linear pipeline \u00b7 03 \u2013 YAML pipelines \u00b7 06 \u2013 Async execution</li> </ul>"},{"location":"03_use_cases/03_yaml_pipeline/","title":"03 \u2013 YAML Pipelines","text":"<p><code>load_pipeline()</code> builds a complete pipeline \u2014 schema, state subclass, nodes, graph, engine, and initial payload \u2014 from one declarative YAML file.</p>"},{"location":"03_use_cases/03_yaml_pipeline/#goal","title":"\ud83c\udfaf Goal","text":"<p>Recreate the entity-resolution DAG from use case 02 purely in YAML.</p>"},{"location":"03_use_cases/03_yaml_pipeline/#the-yaml-file","title":"\ud83d\udcc4 The YAML file","text":"<pre><code>version: 1\n\nschema:\n raw: object\n tags: list | None\n entity:\n candidate: str | None\n name: str | None\n\ninitial:\n raw: {}\n\nnodes:\n extract:\n class: testnodes.ExtractCandidate\n resolve_amzn:\n class: testnodes.ResolveAmazon\n resolve_fk:\n class: testnodes.ResolveFlipkart\n tag:\n class: testnodes.TagNode\n direct:\n class: testnodes.DirectNode\n kill:\n class: testnodes.KillNode\n\ngraph:\n roots:\n - extract\n - direct\n - kill\n edges:\n extract:\n - resolve_amzn\n - resolve_fk\n resolve_amzn:\n - tag\n resolve_fk:\n - tag\n</code></pre> <p>Node class paths are fully qualified (<code>module.ClassName</code>). The modules are imported with <code>importlib</code> at load time, so they must be importable from your runtime environment.</p>"},{"location":"03_use_cases/03_yaml_pipeline/#run-it","title":"\ud83c\udfc3 Run it","text":"<pre><code>from dagpipe import load_pipeline\n\npipeline = load_pipeline(\"pipeline.yaml\")\nresults = pipeline.run()\n\nnames = sorted(s.get(\"entity.name\") for s in results)\nassert names == [\"Amazon\", \"Direct\", \"Flipkart\"]\n</code></pre>"},{"location":"03_use_cases/03_yaml_pipeline/#schema-short-hand","title":"\ud83d\udd0d Schema short-hand","text":"<p>Leaf values in <code>schema:</code> map to Python types:</p> YAML Python <code>str</code> <code>str</code> <code>int</code> / <code>float</code> <code>int</code> / <code>float</code> <code>bool</code> <code>bool</code> <code>dict</code> / <code>list</code> <code>dict</code> / <code>list</code> <code>bytes</code> <code>bytes</code> <code>object</code> <code>object</code> <code>None</code> <code>NoneType</code> <code>str | None</code> <code>str \\| None</code> (PEP-604 union) <p>Nested <code>schema</code> blocks become nested <code>Schema</code> instances.</p>"},{"location":"03_use_cases/03_yaml_pipeline/#overriding-the-initial-payload","title":"\ud83d\udd0d Overriding the initial payload","text":"<p><code>Pipeline.run(payload_override=None)</code> merges override values on top of <code>initial:</code> using <code>Payload.update</code> \u2014 the root payload is the override, not the default:</p> <pre><code>results = pipeline.run(payload_override={\"raw\": \"custom\"})\n</code></pre>"},{"location":"03_use_cases/03_yaml_pipeline/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Give node names that match their <code>class</code> intent \u2014 they become the dict keys used by <code>roots</code>/<code>edges</code>.</li> <li>Every node listed in <code>graph.roots</code>/<code>graph.edges</code> must exist in <code>nodes:</code> or you get a <code>KeyError</code>.</li> <li>YAML pipelines are best for static topologies. For dynamic graphs (nodes built at import time, conditional wiring) use the programmatic API.</li> </ul>"},{"location":"03_use_cases/03_yaml_pipeline/#related","title":"\ud83d\udcda Related","text":"<ul> <li>02 \u2013 Graph execution \u00b7 01 \u2013 Basic linear pipeline \u00b7 06 \u2013 Error Handling</li> </ul>"},{"location":"03_use_cases/04_schema_and_payload/","title":"04 \u2013 Schema and Payload","text":"<p><code>Schema</code> declares the allowed shape of a <code>Payload</code>, and is enforced whenever a <code>State</code> is constructed or forked.</p>"},{"location":"03_use_cases/04_schema_and_payload/#goal","title":"\ud83c\udfaf Goal","text":"<p>Validate a document with a nested address object, using unions for optional fields.</p>"},{"location":"03_use_cases/04_schema_and_payload/#the-schema","title":"\ud83e\uddf1 The schema","text":"<pre><code>from dagpipe import Payload, Schema\n\nAddressSchema = Schema({\n \"city\": str,\n \"zip\": int | None,\n})\n\nUserSchema = Schema({\n \"name\": str,\n \"address\": AddressSchema,\n})\n\nDeepItemSchema = Schema({\n \"raw\": object,\n \"user\": UserSchema,\n})\n</code></pre> <p>Outer keys inside a <code>Schema</code> tree point at one of:</p> <ul> <li>a type (<code>str</code>, <code>int</code>, ...) \u2014 single allowed type</li> <li>a union (<code>str | None</code>) \u2014 any of the members</li> <li>a nested <code>Schema</code> \u2014 the value must be a mapping following that sub-schema</li> <li><code>object</code> \u2014 unrestricted</li> </ul>"},{"location":"03_use_cases/04_schema_and_payload/#valid-payloads","title":"\u2705 Valid payloads","text":"<pre><code>ok = Payload({\n \"raw\": {\"anything\": \"goes\"},\n \"user\": {\n \"name\": \"John\",\n \"address\": {\"city\": \"Mumbai\", \"zip\": 400001},\n },\n})\nDeepItemSchema.validate_payload(ok) # no-op\n</code></pre>"},{"location":"03_use_cases/04_schema_and_payload/#violations","title":"\ud83d\udca5 Violations","text":"<p><code>validate_payload</code> raises <code>SchemaError</code> for any of:</p> Case Example Error Undefined key <code>\"phone\"</code> not declared <code>Invalid path 'phone' not defined in schema</code> Wrong scalar type <code>\"name\": 42</code> <code>Path 'user.name' must be str</code> Union violation <code>\"zip\": \"abc\"</code> <code>Path 'user.address.zip' must be one of (int, NoneType)</code> Non-container for nested schema <code>\"user\": \"joe\"</code> <code>Path 'user' must be a container</code> <pre><code>from dagpipe import SchemaError\n\ntry:\n DeepItemSchema.validate_payload(\n Payload({\"raw\": {}, \"user\": {\"name\": 42, \"address\": {\"city\": \"M\"}}})\n )\nexcept SchemaError as e:\n print(e) # Path 'user.name' must be str\n</code></pre>"},{"location":"03_use_cases/04_schema_and_payload/#updates-are-validated-too","title":"\ud83d\udd0d Updates are validated too","text":"<p><code>State.fork(payload_update=...)</code> calls <code>validate_update</code>, which checks that every dot-path in the update is declared \u2014 before anything is copied:</p> <pre><code>UserSchema.validate_update({\"address.city\": \"Paris\"}) # OK\nUserSchema.validate_update({\"address.country\": \"FR\"}) # SchemaError: not in schema\n</code></pre>"},{"location":"03_use_cases/04_schema_and_payload/#dot-path-access","title":"\ud83d\udd0d Dot-path access","text":"<p><code>Payload</code> gives typed, immutable access to nested values:</p> <pre><code>p = Payload({\"user\": {\"address\": {\"city\": \"Mumbai\", \"zip\": None}}})\n\np.get(\"user.address.city\") # 'Mumbai'\np.get(\"user.address.zip\") # None\np.has(\"user.address.zip\") # True\np.has(\"user.phone\") # False\np.update({\"user.phone\": \"123\"}) # Schema may reject this on the State level!\n</code></pre> <p><code>Payload.update</code> itself is schema-unaware \u2014 the shape check belongs to the <code>State</code>/<code>Schema</code> layer. Use <code>State.fork</code> when you want validation.</p>"},{"location":"03_use_cases/04_schema_and_payload/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li><code>Schema</code> is immutable and reusable across all state instances.</li> <li><code>SchemaError</code> raised during <code>__post_init__</code> means a <code>State</code> can never exist with an invalid payload \u2014 catch it early at construction.</li> <li>Keep union syntax consistent with your Python version (PEP-604 <code>str | None</code> for 3.10+, <code>Union[str, None]</code> otherwise).</li> </ul>"},{"location":"03_use_cases/04_schema_and_payload/#related","title":"\ud83d\udcda Related","text":"<ul> <li>05 \u2013 State fork and lineage \u00b7 02 \u2013 Core Components \u00b7 06 \u2013 Error Handling</li> </ul>"},{"location":"03_use_cases/05_state_fork_and_lineage/","title":"05 \u2013 State Fork and Lineage","text":"<p><code>State</code> is immutable: nodes never modify it. They produce new states via <code>fork()</code>, which carries lineage, confidence, and history forward.</p>"},{"location":"03_use_cases/05_state_fork_and_lineage/#goal","title":"\ud83c\udfaf Goal","text":"<p>Walk a branch tree and inspect how states are related, how confidence compounds, and how <code>history</code> records the exact node path taken.</p>"},{"location":"03_use_cases/05_state_fork_and_lineage/#setup","title":"\ud83e\uddf1 Setup","text":"<pre><code>from dagpipe import Payload, Schema, State, Node\n\nclass ScoreState(State):\n schema = Schema({\"score\": int | float, \"label\": str | None})\n\nclass AddBonus(Node):\n id = \"score.bonus\"\n\n def resolve(self, state):\n yield self.fork(state, confidence_delta=0.1,\n payload_update={\"score\": state.get(\"score\") + 10})\n</code></pre> <p><code>Node.fork</code> passes <code>node_id=self.id</code>, so <code>history</code> records the point of derivation automatically.</p>"},{"location":"03_use_cases/05_state_fork_and_lineage/#forking","title":"\ud83d\udd0d Forking","text":"<pre><code>root = ScoreState(payload=Payload({\"score\": 50}))\n\ns1 = root.fork(\n payload_update={\"score\": 60},\n confidence_delta=0.1,\n node_id=\"score.bonus\",\n metadata_update={\"source\": \"manual\"},\n)\n\ns2 = s1.fork(payload_update={\"label\": \"high\"})\n</code></pre> Field <code>root</code> <code>s1</code> <code>s2</code> <code>depth</code> <code>0</code> <code>1</code> <code>2</code> <code>score</code> <code>50</code> <code>60</code> <code>60</code> <code>confidence</code> <code>1.0</code> <code>1.1</code> <code>1.1</code> <code>history</code> <code>()</code> <code>('score.bonus',)</code> <code>('score.bonus',)</code> <code>parent</code> <code>None</code> <code>root</code> <code>s1</code>"},{"location":"03_use_cases/05_state_fork_and_lineage/#lineage","title":"\ud83d\udd0d Lineage","text":"<p><code>lineage()</code> walks parents back to the root, root-first:</p> <pre><code>states = s2.lineage()\nassert states == (root, s1, s2)\nassert states[0] is root\nassert states[-1] is s2\n</code></pre>"},{"location":"03_use_cases/05_state_fork_and_lineage/#immutability-guarantees","title":"\ud83d\udd0d Immutability guarantees","text":"<pre><code>assert root.get(\"label\") is None # s2's update didn't touch root\nassert s1.get(\"label\") is None # fork never mutates its input\nassert root.confidence == 1.0 # confidence travels forward only\nassert root is s1.parent # parent is by reference\n</code></pre>"},{"location":"03_use_cases/05_state_fork_and_lineage/#metadata-confidence","title":"\ud83d\udd0d Metadata & confidence","text":"<ul> <li><code>confidence_delta</code> is added (<code>parent.confidence + delta</code>), so a chain of bonuses compounds: 1.0 \u2192 1.1 \u2192 1.2.</li> <li><code>metadata_update</code> is shallow-merged into <code>metadata</code> \u2014 untouched states share the same metadata dict (no copy unless a fork changes it).</li> <li><code>payload_update</code> keys are dot-paths; they are validated by <code>validate_update</code> against the state's <code>schema</code> before copying.</li> </ul>"},{"location":"03_use_cases/05_state_fork_and_lineage/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Use <code>Node.fork</code> in <code>resolve()</code> \u2014 it records <code>node_id</code> into history for free.</li> <li>Fork parameters are keyword-only (<code>payload_update</code>, <code>confidence_delta</code>, <code>node_id</code>, <code>metadata_update</code>) \u2014 there is no positional API.</li> <li>A root state built directly (<code>ScoreState(payload=...)</code>) validates against <code>schema</code> in <code>__post_init__</code> \u2014 you can't create an invalid root.</li> <li>Since states are immutable and share structure, forking many branches is cheap \u2014 copy only the modified branch (see <code>Payload.update</code>).</li> </ul>"},{"location":"03_use_cases/05_state_fork_and_lineage/#related","title":"\ud83d\udcda Related","text":"<ul> <li>04 \u2013 Schema and payload \u00b7 01 \u2013 Basic linear pipeline \u00b7 04 \u2013 Best Practices</li> </ul>"},{"location":"03_use_cases/06_async_execution/","title":"06 \u2013 Async Execution","text":"<p><code>AsyncNode</code> subclasses run I/O-bound steps (HTTP, DB, file reads) via <code>Engine.run_async</code>. Sync and async nodes can be mixed in the same graph \u2014 the engine dispatches each node to the right path automatically.</p>"},{"location":"03_use_cases/06_async_execution/#goal","title":"\ud83c\udfaf Goal","text":"<p>Fetch a document from a remote service, then run a synchronous post-processor.</p> <pre><code>import asyncio\nfrom dagpipe import AsyncNode, Node, Schema, State, Payload, Graph, Engine\n\nclass FetchDoc(AsyncNode):\n id = \"web.fetch\"\n\n async def resolve_async(self, state: State):\n body = await fetch_body(state.get(\"url\")) # your async I/O\n yield self.fork(state, payload_update={\"body\": body, \"fetched\": True})\n\nclass Summarize(Node):\n id = \"text.summarize\"\n\n def resolve(self, state: State):\n words = len(state.get(\"body\", \"\").split())\n yield self.fork(state, payload_update={\"word_count\": words})\n\nclass DocState(State):\n schema = Schema({\n \"url\": str,\n \"body\": str | None,\n \"fetched\": bool | None,\n \"word_count\": int | None,\n })\n</code></pre>"},{"location":"03_use_cases/06_async_execution/#run-it","title":"\ud83c\udfc3 Run it","text":"<pre><code>async def main():\n graph = Graph()\n graph.add_edge(FetchDoc(), Summarize())\n\n engine = Engine(graph)\n results = await engine.run_async(DocState(payload=Payload({\"url\": \"https://example.com\"})))\n\n assert results[0].get(\"fetched\") is True\n assert results[0].get(\"word_count\") > 0\n\nasyncio.run(main())\n</code></pre>"},{"location":"03_use_cases/06_async_execution/#how-dispatch-works","title":"\ud83d\udd0d How dispatch works","text":"<p><code>Engine.run_async</code> checks each node at execution time:</p> <pre><code>node is AsyncNode ? \u2192 await node.run_async(state) # resolve_async()\n \u2192 node.run(state) # resolve()\n</code></pre> <ul> <li><code>AsyncNode.resolve()</code> is a no-op (returns <code>()</code>), so a sync engine (<code>run</code>) treats async nodes as if they pruned the branch.</li> <li><code>AsyncNode.resolve_async()</code> may be a plain <code>async def</code> returning an iterable, or an async generator \u2014 both are supported.</li> <li><code>run_async</code> validates that yielded objects are <code>State</code> (raises <code>TypeError</code> otherwise), exactly like the sync path.</li> </ul>"},{"location":"03_use_cases/06_async_execution/#async-step-wise-variant","title":"\ud83d\udd0d Async step-wise variant","text":"<p><code>run_steps_async</code> pairs with async nodes for progress + resume:</p> <pre><code>async for step in engine.run_steps_async(root):\n print(step.index, step.node_id, len(step.states), step.completed)\n</code></pre>"},{"location":"03_use_cases/06_async_execution/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Async nodes and sync nodes interleave freely \u2014 no need to split phases.</li> <li>For true concurrency across branches, combine async nodes with a library like <code>asyncio.gather</code> inside a single wrapper node.</li> <li>Don't call <code>run_async</code> on an engine whose nodes are all sync \u2014 you pay event loop overhead for nothing; <code>run</code> is fine there.</li> <li>Cancellation patterns (timeouts, retries) belong inside <code>resolve_async</code>, not the engine \u2014 keep the engine generic.</li> </ul>"},{"location":"03_use_cases/06_async_execution/#related","title":"\ud83d\udcda Related","text":"<ul> <li>02 \u2013 Graph execution \u00b7 07 \u2013 Step-wise execution \u00b7 02 \u2013 Core Components</li> </ul>"},{"location":"03_use_cases/07_step_execution/","title":"07 \u2013 Step-wise Execution","text":"<p><code>Engine.run_steps</code> runs a pipeline one step at a time, yielding a <code>StepResult</code> per executed node. It supports resume (skip already-done steps) and progress hooks (report <code>started</code> / <code>completed</code> / <code>skipped</code>).</p>"},{"location":"03_use_cases/07_step_execution/#goal","title":"\ud83c\udfaf Goal","text":"<p>Run a three-node pipeline, observe progress, and resume a partial run.</p>"},{"location":"03_use_cases/07_step_execution/#setup","title":"\ud83e\uddf1 Setup","text":"<pre><code>from dagpipe import Engine, Node, Payload, Schema, State\n\nclass StepA(Node):\n id = \"qa.step_a\"\n def resolve(self, state):\n yield self.fork(state, payload_update={\"a\": True})\n\nclass StepB(Node):\n id = \"qa.step_b\"\n def resolve(self, state):\n if not state.get(\"a\"):\n return # prune branch if precondition unmet\n yield self.fork(state, payload_update={\"b\": True})\n\nclass StepC(Node):\n id = \"qa.step_c\"\n def resolve(self, state):\n yield self.fork(state, payload_update={\"c\": True})\n\nclass QState(State):\n schema = Schema({\"a\": bool | None, \"b\": bool | None, \"c\": bool | None})\n\nengine = Engine([StepA(), StepB(), StepC()])\nroot = QState(payload=Payload({}))\n</code></pre>"},{"location":"03_use_cases/07_step_execution/#step-by-step","title":"\ud83c\udfc3 Step by step","text":"<pre><code>for step in engine.run_steps(root):\n print(f\"#{step.index} {step.node_id:10} completed={step.completed} states={len(step.states)}\")\n</code></pre> <p>Output:</p> <pre><code>#0 qa.step_a completed=True states=1\n#1 qa.step_b completed=True states=1\n#2 qa.step_c completed=True states=1\n</code></pre> <p>Each <code>StepResult</code> exposes:</p> Field Meaning <code>index</code> 0-based step ordinal (<code>resume_from</code> is relative to this) <code>node_id</code> The node that ran in this step <code>states</code> States produced by this step <code>completed</code> <code>True</code> if any state was produced"},{"location":"03_use_cases/07_step_execution/#resume-after-interruption","title":"\ud83d\udd0d Resume after interruption","text":"<p>If you already processed <code>#0</code> and <code>#1</code>, skip ahead:</p> <pre><code>for step in engine.run_steps(root, resume_from=2):\n print(step.node_id) # only qa.step_c\n</code></pre>"},{"location":"03_use_cases/07_step_execution/#progress-hooks","title":"\ud83d\udd0d Progress hooks","text":"<p>Pass <code>on_step</code> (or set it at engine construction) to be notified per step:</p> <pre><code>engine = Engine([StepA(), StepB(), StepC()],\n on_step=lambda step, status, msg: print(step, status))\n\nfor step in engine.run_steps(root):\n ...\n</code></pre> <p>Output:</p> <pre><code>qa.step_a started\nqa.step_a completed\nqa.step_b started\nqa.step_b completed\n...\n</code></pre> <p>A <code>ProgressMessage</code> can be passed as the third argument for richer progress (<code>lines</code>, <code>blocks</code>, <code>count</code>, <code>unit</code>, <code>error</code>, ...). Use <code>as_dict()</code> to surface it in your logging.</p> <p>The async variant <code>run_steps_async</code> mirrors this API with <code>on_step: AsyncStepHook</code> \u2014 see use case 06.</p>"},{"location":"03_use_cases/07_step_execution/#step-ordering-in-graph-mode","title":"\ud83d\udd0d Step ordering in graph mode","text":"<p>In graph mode, <code>run_steps</code> derives a deterministic topological order (DFS post-order, roots first), one <code>StepResult</code> per node. A node that yields no states marks that step <code>completed=False</code> and the branch stops there.</p>"},{"location":"03_use_cases/07_step_execution/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Use <code>resume_from</code> for idempotent retries: record the last consumed <code>index</code> (e.g. in a DB), then resume the next run from <code>index + 1</code>.</li> <li><code>completed=False</code> means \"no states produced\" (pruned/static node) \u2014 treat it as \"no work available\", not \"error\".</li> <li>Hooks are a good place to emit telemetry (counters, traces) without coupling the engine to your logging stack.</li> </ul>"},{"location":"03_use_cases/07_step_execution/#related","title":"\ud83d\udcda Related","text":"<ul> <li>06 \u2013 Async execution \u00b7 01 \u2013 Basic linear pipeline \u00b7 02 \u2013 Core Components</li> </ul>"}]} |