From 58544c452453ae969801b58c1421af9cb0ffb713 Mon Sep 17 00:00:00 2001 From: Vishesh 'ironeagle' Bangotra Date: Tue, 15 Sep 2026 19:34:33 +0530 Subject: [PATCH] docs: collect hexa wiki and re-collect dagpipe, hexa and omniread lib sites --- _index/index.html | 1 + config.yml | 1 + dagpipe/lib/index.html | 4 +- dagpipe/lib/search/search_index.json | 2 +- dagpipe/lib/sitemap.xml.gz | Bin 127 -> 127 bytes hexa/lib/design/index.html | 1353 ---- hexa/lib/index.html | 8 +- hexa/lib/model/index.html | 4 +- hexa/lib/parse_yaml/index.html | 2 +- hexa/lib/search/search_index.json | 2 +- hexa/wiki/01_overview/index.html | 1163 +++ hexa/wiki/02_components/index.html | 1255 +++ .../03_use_cases/01_yaml_first/index.html | 1110 +++ .../wiki/03_use_cases/02_abc_first/index.html | 1088 +++ .../03_use_cases/03_wiring_runtime/index.html | 1113 +++ hexa/wiki/04_design/index.html | 1373 ++++ .../05_philosophy}/index.html | 1083 +-- hexa/wiki/06_best_practices/index.html | 1072 +++ hexa/wiki/07_error_handling/index.html | 1126 +++ hexa/wiki/08_testing_example/index.html | 1092 +++ hexa/wiki/404.html | 860 +++ hexa/wiki/assets/images/favicon.png | Bin 0 -> 1870 bytes .../assets/javascripts/bundle.f55a23d4.min.js | 16 + .../javascripts/bundle.f55a23d4.min.js.map | 7 + .../javascripts/lunr/min/lunr.ar.min.js | 1 + .../javascripts/lunr/min/lunr.da.min.js | 18 + .../javascripts/lunr/min/lunr.de.min.js | 18 + .../javascripts/lunr/min/lunr.du.min.js | 18 + .../javascripts/lunr/min/lunr.el.min.js | 1 + .../javascripts/lunr/min/lunr.es.min.js | 18 + .../javascripts/lunr/min/lunr.fi.min.js | 18 + .../javascripts/lunr/min/lunr.fr.min.js | 18 + .../javascripts/lunr/min/lunr.he.min.js | 1 + .../javascripts/lunr/min/lunr.hi.min.js | 1 + .../javascripts/lunr/min/lunr.hu.min.js | 18 + .../javascripts/lunr/min/lunr.hy.min.js | 1 + .../javascripts/lunr/min/lunr.it.min.js | 18 + .../javascripts/lunr/min/lunr.ja.min.js | 1 + .../javascripts/lunr/min/lunr.jp.min.js | 1 + .../javascripts/lunr/min/lunr.kn.min.js | 1 + .../javascripts/lunr/min/lunr.ko.min.js | 1 + .../javascripts/lunr/min/lunr.multi.min.js | 1 + .../javascripts/lunr/min/lunr.nl.min.js | 18 + .../javascripts/lunr/min/lunr.no.min.js | 18 + .../javascripts/lunr/min/lunr.pt.min.js | 18 + .../javascripts/lunr/min/lunr.ro.min.js | 18 + .../javascripts/lunr/min/lunr.ru.min.js | 18 + .../javascripts/lunr/min/lunr.sa.min.js | 1 + .../lunr/min/lunr.stemmer.support.min.js | 1 + .../javascripts/lunr/min/lunr.sv.min.js | 18 + .../javascripts/lunr/min/lunr.ta.min.js | 1 + .../javascripts/lunr/min/lunr.te.min.js | 1 + .../javascripts/lunr/min/lunr.th.min.js | 1 + .../javascripts/lunr/min/lunr.tr.min.js | 18 + .../javascripts/lunr/min/lunr.vi.min.js | 1 + .../javascripts/lunr/min/lunr.zh.min.js | 1 + hexa/wiki/assets/javascripts/lunr/tinyseg.js | 206 + hexa/wiki/assets/javascripts/lunr/wordcut.js | 6708 +++++++++++++++++ .../workers/search.973d3a69.min.js | 42 + .../workers/search.973d3a69.min.js.map | 7 + .../assets/stylesheets/main.84d31ad4.min.css | 1 + .../stylesheets/main.84d31ad4.min.css.map | 1 + .../stylesheets/palette.06af60db.min.css | 1 + .../stylesheets/palette.06af60db.min.css.map | 1 + hexa/wiki/index.html | 1073 +++ hexa/wiki/search/search_index.json | 1 + hexa/wiki/sitemap.xml | 3 + hexa/wiki/sitemap.xml.gz | Bin 0 -> 127 bytes omniread/lib/index.html | 4 +- omniread/lib/search/search_index.json | 2 +- omniread/lib/sitemap.xml.gz | Bin 127 -> 127 bytes 71 files changed, 20194 insertions(+), 1878 deletions(-) delete mode 100644 hexa/lib/design/index.html create mode 100644 hexa/wiki/01_overview/index.html create mode 100644 hexa/wiki/02_components/index.html create mode 100644 hexa/wiki/03_use_cases/01_yaml_first/index.html create mode 100644 hexa/wiki/03_use_cases/02_abc_first/index.html create mode 100644 hexa/wiki/03_use_cases/03_wiring_runtime/index.html create mode 100644 hexa/wiki/04_design/index.html rename hexa/{lib/philosophy => wiki/05_philosophy}/index.html (54%) create mode 100644 hexa/wiki/06_best_practices/index.html create mode 100644 hexa/wiki/07_error_handling/index.html create mode 100644 hexa/wiki/08_testing_example/index.html create mode 100644 hexa/wiki/404.html create mode 100644 hexa/wiki/assets/images/favicon.png create mode 100644 hexa/wiki/assets/javascripts/bundle.f55a23d4.min.js create mode 100644 hexa/wiki/assets/javascripts/bundle.f55a23d4.min.js.map create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.ar.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.da.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.de.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.du.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.el.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.es.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.fi.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.fr.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.he.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.hi.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.hu.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.hy.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.it.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.ja.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.jp.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.kn.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.ko.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.multi.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.nl.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.no.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.pt.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.ro.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.ru.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.sa.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.stemmer.support.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.sv.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.ta.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.te.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.th.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.tr.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.vi.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/min/lunr.zh.min.js create mode 100644 hexa/wiki/assets/javascripts/lunr/tinyseg.js create mode 100644 hexa/wiki/assets/javascripts/lunr/wordcut.js create mode 100644 hexa/wiki/assets/javascripts/workers/search.973d3a69.min.js create mode 100644 hexa/wiki/assets/javascripts/workers/search.973d3a69.min.js.map create mode 100644 hexa/wiki/assets/stylesheets/main.84d31ad4.min.css create mode 100644 hexa/wiki/assets/stylesheets/main.84d31ad4.min.css.map create mode 100644 hexa/wiki/assets/stylesheets/palette.06af60db.min.css create mode 100644 hexa/wiki/assets/stylesheets/palette.06af60db.min.css.map create mode 100644 hexa/wiki/index.html create mode 100644 hexa/wiki/search/search_index.json create mode 100644 hexa/wiki/sitemap.xml create mode 100644 hexa/wiki/sitemap.xml.gz diff --git a/_index/index.html b/_index/index.html index d6dc111..e057ac1 100644 --- a/_index/index.html +++ b/_index/index.html @@ -170,6 +170,7 @@

Hexa

Hexagonal architecture framework and toolbox for Python services.

+ view wiki view lib
diff --git a/config.yml b/config.yml index 9742a43..79a19b5 100644 --- a/config.yml +++ b/config.yml @@ -82,6 +82,7 @@ repos: description: Hexagonal architecture framework and toolbox for Python services. section: libraries docs: + wiki: site lib: site - name: mongo-ops diff --git a/dagpipe/lib/index.html b/dagpipe/lib/index.html index 106c1ac..f0c557c 100644 --- a/dagpipe/lib/index.html +++ b/dagpipe/lib/index.html @@ -6782,9 +6782,7 @@ Ordered execution lineage (root first).

- + diff --git a/dagpipe/lib/search/search_index.json b/dagpipe/lib/search/search_index.json index 7f1543c..2b53a3d 100644 --- a/dagpipe/lib/search/search_index.json +++ b/dagpipe/lib/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"dagpipe","text":""},{"location":"#dagpipe","title":"dagpipe","text":""},{"location":"#dagpipe--summary","title":"Summary","text":"

Directed acyclic graph execution framework for deterministic state propagation.

dagpipe executes pipelines composed of nodes connected in a directed acyclic graph (DAG). Each node receives an immutable State and optionally produces derived states for downstream nodes.

"},{"location":"#dagpipe--installation","title":"Installation","text":"

Install using pip:

pip install dagpipe\n
"},{"location":"#dagpipe--quick-start","title":"Quick Start","text":"
from dagpipe import State, Payload, Schema, Graph, Engine\nfrom dagpipe.node import Node\n\nclass HelloNode(Node):\n    id = \"hello\"\n    def resolve(self, state):\n        yield self.fork(state, payload_update={\"msg\": \"hello\"})\n\n# Build and run\ngraph = Graph()\ngraph.add_root(HelloNode())\nengine = Engine(graph)\n\nclass MyState(State):\n    schema = Schema({})\n\nresults = engine.run(MyState(payload=Payload({})))\n
"},{"location":"#dagpipe--public-api","title":"Public API","text":"

This package re-exports the core pipeline components. Consumers should import from this namespace for standard usage.

"},{"location":"#dagpipe--execution-core","title":"Execution Core","text":""},{"location":"#dagpipe--state-data","title":"State & Data","text":""},{"location":"#dagpipe--declarative-pipelines","title":"Declarative Pipelines","text":""},{"location":"#dagpipe-classes","title":"Classes","text":""},{"location":"#dagpipe.AsyncNode","title":"AsyncNode","text":"

Bases: Node

Base class for nodes whose execution is asynchronous.

Subclasses implement resolve_async (an async generator yielding derived State objects). The engine dispatches to resolve_async when running an async traversal (see Engine.run_async).

Sync-only engines (and the base Node.run) treat an AsyncNode as a no-op consumer: calling run on an AsyncNode returns no states, signalling that an async engine is required.

"},{"location":"#dagpipe.AsyncNode-functions","title":"Functions","text":""},{"location":"#dagpipe.AsyncNode.__hash__","title":"__hash__","text":"
__hash__() -> int\n

Return stable hash based on node ID.

"},{"location":"#dagpipe.AsyncNode.__hash__--returns","title":"Returns","text":"

int

"},{"location":"#dagpipe.AsyncNode.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"#dagpipe.AsyncNode.__repr__--returns","title":"Returns","text":"

str

"},{"location":"#dagpipe.AsyncNode.__str__","title":"__str__","text":"
__str__() -> str\n

Return display representation.

"},{"location":"#dagpipe.AsyncNode.__str__--returns","title":"Returns","text":"

str

"},{"location":"#dagpipe.AsyncNode.clean_id_and_name","title":"clean_id_and_name classmethod","text":"
clean_id_and_name() -> None\n

Normalize and validate node ID and display name.

Raises:

Type Description TypeError

If ID is not a string.

ValueError

If ID format is invalid.

Notes

Guarantees:

- Generates ID from module and class name if missing.\n- Validates ID format.\n- Generates human-readable name if missing.\n
"},{"location":"#dagpipe.AsyncNode.fork","title":"fork","text":"
fork(\n    state: State,\n    *,\n    payload_update: Any = None,\n    confidence_delta: float = 0.0,\n    metadata_update: Any = None\n) -> State\n

Create a child State attributed to this node.

Parameters:

Name Type Description Default state State

Parent execution state.

required payload_update Any

Dot-path payload updates.

None confidence_delta float

Confidence adjustment.

0.0 metadata_update Any

Metadata updates.

None

Returns:

Name Type Description State State

New child execution state.

Notes

Responsibilities:

- Convenience wrapper around `State.fork()` that automatically\n  records this node's ID in state history.\n
"},{"location":"#dagpipe.AsyncNode.is_async","title":"is_async","text":"
is_async() -> bool\n

Return whether this node executes asynchronously.

"},{"location":"#dagpipe.AsyncNode.node_id_to_name","title":"node_id_to_name staticmethod","text":"
node_id_to_name(node_id: str) -> str\n

Convert a dotted snake_case node ID into a human-readable name.

Parameters:

Name Type Description Default node_id str

Unique node identifier (e.g., 'entity.resolve.numeric_merchant').

required

Returns:

Name Type Description str str

Human-readable display name (e.g., 'Entity \u203a Resolve \u203a Numeric Merchant').

"},{"location":"#dagpipe.AsyncNode.resolve_async","title":"resolve_async async","text":"
resolve_async(state: State) -> Iterable[State]\n

Execute node logic asynchronously.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description Iterable[State]

Iterable[State]: Derived execution state(s).

Notes

Subclasses implement this. Must not mutate the input state. Should use fork() to create child states.

"},{"location":"#dagpipe.AsyncNode.run","title":"run","text":"
run(state: State) -> tuple[State, ...]\n

Execute this node on a State.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Derived execution states.

Raises:

Type Description TypeError

If resolve() yields a non-State object.

"},{"location":"#dagpipe.AsyncNode.run_async","title":"run_async async","text":"
run_async(state: State) -> tuple[State, ...]\n

Execute this node asynchronously on a state, validating outputs.

"},{"location":"#dagpipe.Engine","title":"Engine","text":"
Engine(\n    nodes_or_graph: Sequence[Node] | Graph,\n    *,\n    on_step: StepHook | None = None\n)\n

Execution engine responsible for running pipeline logic.

Notes

Responsibilities:

- Accepts either a linear sequence of `Node` objects or a `Graph`\n  defining execution topology.\n- Propagates immutable `State` objects through `Node` objects and\n  collects terminal states.\n- Supports synchronous (`run`) and asynchronous (`run_async`)\n  execution, dispatching per-node.\n- Supports step-wise / resumable execution and progress hooks.\n

Guarantees:

- Never mutates `State`, `Node`, or `Graph` instances.\n- `State` objects are never modified in place; each branch produces\n  independent instances.\n- Execution order is deterministic and follows graph or pipeline topology.\n- Thread-safe for concurrent execution.\n
"},{"location":"#dagpipe.Engine-attributes","title":"Attributes","text":""},{"location":"#dagpipe.Engine.nodes","title":"nodes property","text":"
nodes: tuple[Node, ...]\n

Return nodes managed by this engine.

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Ordered sequence in linear mode or all nodes in graph mode.

"},{"location":"#dagpipe.Engine-functions","title":"Functions","text":""},{"location":"#dagpipe.Engine.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return the canonical string representation of the object.

Returns:

Name Type Description str str

Representation that uniquely identifies the object and its configuration.

"},{"location":"#dagpipe.Engine.run","title":"run","text":"
run(root: State) -> list[State]\n

Execute the pipeline starting from a root State.

Parameters:

Name Type Description Default root State

Initial execution state.

required

Returns:

Type Description list[State]

list[State]: Terminal execution states produced by the pipeline.

Raises:

Type Description TypeError

If root is not a State instance.

RuntimeError

If the engine execution mode is invalid.

Notes

Responsibilities:

- Selects execution mode, propagates state through nodes, creates\n  new instances for branches, and collects terminal states.\n
"},{"location":"#dagpipe.Engine.run_async","title":"run_async async","text":"
run_async(root: State) -> list[State]\n

Execute the pipeline starting from root, dispatching sync vs async nodes.

Parameters:

Name Type Description Default root State

Initial execution state.

required

Returns:

Type Description list[State]

list[State]: Terminal execution states produced by the pipeline.

Notes

Each node is executed with Node.run when synchronous and AsyncNode.run_async when asynchronous. Linear and graph topologies are both supported.

"},{"location":"#dagpipe.Engine.run_steps","title":"run_steps","text":"
run_steps(\n    root: State,\n    *,\n    resume_from: int | None = None,\n    on_step: StepHook | None = None\n) -> Iterator[StepResult]\n

Execute the pipeline step-by-step, yielding one StepResult per step.

Parameters:

Name Type Description Default root State

Initial execution state.

required resume_from int | None

Skip steps at index < resume_from (for resume-after-partial). Steps are 0-indexed.

None on_step StepHook | None

Callback (step, status, message) invoked per step; falls back to the engine-level hook when unset.

None

Yields:

Name Type Description StepResult StepResult

One per executed node/step, carrying the produced states.

Notes

This is a synchronous, generator-based checkpoint interface compatible with the imperative resume-by-step behaviour of the legacy orchestrator. Use run_steps_async for async nodes.

"},{"location":"#dagpipe.Engine.run_steps_async","title":"run_steps_async async","text":"
run_steps_async(\n    root: State,\n    *,\n    resume_from: int | None = None,\n    on_step: AsyncStepHook | None = None\n) -> AsyncIterator[StepResult]\n

Async variant of run_steps supporting AsyncNode execution.

Parameters:

Name Type Description Default root State

Initial execution state.

required resume_from int | None

Skip steps at index < resume_from.

None on_step AsyncStepHook | None

Async callback (step, status, message) invoked per step.

None

Yields:

Name Type Description StepResult AsyncIterator[StepResult]

One per executed node/step.

"},{"location":"#dagpipe.Graph","title":"Graph","text":"
Graph()\n

Directed Acyclic Graph defining execution topology of Node objects.

Notes

Responsibilities:

- Stores node connectivity and validates that the topology remains acyclic.\n- Structure determines how `State` flows between nodes during execution.\n

Guarantees:

- Topology is acyclic. Node relationships remain consistent.\n- Thread-safe for concurrent reads after construction.\n

Create an empty Graph.

"},{"location":"#dagpipe.Graph--initializes-node-registry-and-edge-mappings","title":"Initializes node registry and edge mappings.","text":""},{"location":"#dagpipe.Graph-functions","title":"Functions","text":""},{"location":"#dagpipe.Graph.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"#dagpipe.Graph.__repr__--returns","title":"Returns","text":"

str

"},{"location":"#dagpipe.Graph.add_edge","title":"add_edge","text":"
add_edge(src: Node, dst: Node) -> None\n

Add a directed edge from src to dst.

Parameters:

Name Type Description Default src Node

Source node.

required dst Node

Destination node.

required

Raises:

Type Description TypeError

If src or dst is not a Node.

ValueError

If the edge would create a cycle or if src and dst are common.

Notes "},{"location":"#dagpipe.Graph.add_root","title":"add_root","text":"
add_root(node: Node) -> None\n

Add a root node with no parents.

Parameters:

Name Type Description Default node Node

Node to add as a root.

required

Raises:

Type Description TypeError

If node is not a Node instance.

"},{"location":"#dagpipe.Graph.children","title":"children","text":"
children(node: Node) -> tuple[Node, ...]\n

Return child nodes of a node.

Parameters:

Name Type Description Default node Node

Node to query.

required

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Outgoing neighbors.

"},{"location":"#dagpipe.Graph.nodes","title":"nodes","text":"
nodes() -> tuple[Node, ...]\n

Return all nodes in the graph.

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: All registered nodes.

"},{"location":"#dagpipe.Graph.parents","title":"parents","text":"
parents(node: Node) -> tuple[Node, ...]\n

Return parent nodes of a node.

Parameters:

Name Type Description Default node Node

Node to query.

required

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Incoming neighbors.

"},{"location":"#dagpipe.Graph.roots","title":"roots","text":"
roots() -> tuple[Node, ...]\n

Return root nodes (nodes with no incoming edges).

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Entry point nodes.

"},{"location":"#dagpipe.Node","title":"Node","text":"

Bases: ABC

Base class for all dagpipe execution nodes.

Attributes:

Name Type Description id str

Unique identifier of the node (snake_case dotted format).

name str

Human-readable display name.

Notes

Responsibilities:

- Represents a deterministic unit of execution in the pipeline graph.\n- Consumes one `State` and produces zero, one, or many derived states.\n- Defines execution logic and enables branching, filtering, and transformation.\n

Guarantees:

- Nodes must never mutate the input `State`.\n- Instances are singletons per subclass and reused across executions.\n
"},{"location":"#dagpipe.Node-functions","title":"Functions","text":""},{"location":"#dagpipe.Node.__hash__","title":"__hash__","text":"
__hash__() -> int\n

Return stable hash based on node ID.

"},{"location":"#dagpipe.Node.__hash__--returns","title":"Returns","text":"

int

"},{"location":"#dagpipe.Node.__new__","title":"__new__","text":"
__new__(*args: Any, **kwargs: Any) -> Node\n

Create or reuse a Node instance.

Stateless subclasses (no parameterized __init__) share one singleton instance per class \u2014 matching the original dagpipe behaviour underpinning set_registry-style configuration. Subclasses that declare an __init__ requiring instance-state arguments get a fresh instance per construction so pipeline builders can inject per-run dependencies.

"},{"location":"#dagpipe.Node.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"#dagpipe.Node.__repr__--returns","title":"Returns","text":"

str

"},{"location":"#dagpipe.Node.__str__","title":"__str__","text":"
__str__() -> str\n

Return display representation.

"},{"location":"#dagpipe.Node.__str__--returns","title":"Returns","text":"

str

"},{"location":"#dagpipe.Node.clean_id_and_name","title":"clean_id_and_name classmethod","text":"
clean_id_and_name() -> None\n

Normalize and validate node ID and display name.

Raises:

Type Description TypeError

If ID is not a string.

ValueError

If ID format is invalid.

Notes

Guarantees:

- Generates ID from module and class name if missing.\n- Validates ID format.\n- Generates human-readable name if missing.\n
"},{"location":"#dagpipe.Node.fork","title":"fork","text":"
fork(\n    state: State,\n    *,\n    payload_update: Any = None,\n    confidence_delta: float = 0.0,\n    metadata_update: Any = None\n) -> State\n

Create a child State attributed to this node.

Parameters:

Name Type Description Default state State

Parent execution state.

required payload_update Any

Dot-path payload updates.

None confidence_delta float

Confidence adjustment.

0.0 metadata_update Any

Metadata updates.

None

Returns:

Name Type Description State State

New child execution state.

Notes

Responsibilities:

- Convenience wrapper around `State.fork()` that automatically\n  records this node's ID in state history.\n
"},{"location":"#dagpipe.Node.is_async","title":"is_async","text":"
is_async() -> bool\n

Return whether this node executes asynchronously.

"},{"location":"#dagpipe.Node.node_id_to_name","title":"node_id_to_name staticmethod","text":"
node_id_to_name(node_id: str) -> str\n

Convert a dotted snake_case node ID into a human-readable name.

Parameters:

Name Type Description Default node_id str

Unique node identifier (e.g., 'entity.resolve.numeric_merchant').

required

Returns:

Name Type Description str str

Human-readable display name (e.g., 'Entity \u203a Resolve \u203a Numeric Merchant').

"},{"location":"#dagpipe.Node.resolve","title":"resolve abstractmethod","text":"
resolve(state: State) -> Iterable[State]\n

Execute node logic.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description Iterable[State]

Iterable[State]: Derived execution state(s).

Notes

Responsibilities:

- Subclasses implement specific resolution behavior.\n- Must not mutate input state.\n- Should use `fork()` to create child states.\n- May yield zero states to terminate a branch.\n
"},{"location":"#dagpipe.Node.run","title":"run","text":"
run(state: State) -> tuple[State, ...]\n

Execute this node on a State.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Derived execution states.

Raises:

Type Description TypeError

If resolve() yields a non-State object.

"},{"location":"#dagpipe.Payload","title":"Payload dataclass","text":"
Payload(_data: Mapping[str, Any])\n

Immutable hierarchical container with dot-path access.

Attributes:

Name Type Description _data Mapping[str, Any]

Immutable hierarchical data structure.

Notes

Responsibilities:

- Stores execution data used by `State`.\n- Supports efficient atomic updates without modifying existing instances.\n- `Payload` instances are fully thread-safe due to immutability.\n
"},{"location":"#dagpipe.Payload-functions","title":"Functions","text":""},{"location":"#dagpipe.Payload.as_dict","title":"as_dict","text":"
as_dict() -> Mapping[str, Any]\n

Return underlying mapping.

Returns:

Type Description Mapping[str, Any]

Mapping[str, Any]: Read-only view of the underlying data.

"},{"location":"#dagpipe.Payload.get","title":"get","text":"
get(path: str, default: Any = None) -> Any\n

Retrieve value using dot-path.

Parameters:

Name Type Description Default path str

Dot-separated path to the value.

required default Any

Default value if path doesn't exist.

None

Returns:

Name Type Description Any Any

The retrieved value or default.

"},{"location":"#dagpipe.Payload.has","title":"has","text":"
has(path: str) -> bool\n

Return True if path exists.

Parameters:

Name Type Description Default path str

Dot-separated path to check.

required

Returns:

Name Type Description bool bool

Existence of the path.

"},{"location":"#dagpipe.Payload.iter_paths","title":"iter_paths classmethod","text":"
iter_paths(\n    data: Mapping[str, Any], prefix: str = \"\"\n) -> Iterable[str]\n

Recursively yield dot-paths for all leaf nodes.

Parameters:

Name Type Description Default data Mapping[str, Any]

The mapping to iterate over.

required prefix str

Current path prefix.

''

Yields:

Name Type Description str Iterable[str]

Dot-path for each leaf node.

"},{"location":"#dagpipe.Payload.keys","title":"keys","text":"
keys() -> Iterable[str]\n

Return top-level keys.

Returns:

Type Description Iterable[str]

Iterable[str]: Iterator over top-level keys.

"},{"location":"#dagpipe.Payload.update","title":"update","text":"
update(updates: Mapping[str, Any]) -> Payload\n

Create a new Payload with dot-path updates applied.

Parameters:

Name Type Description Default updates Mapping[str, Any]

Dot-path to value mapping.

required

Returns:

Name Type Description Payload Payload

New immutable payload instance with updates.

Notes

Guarantees:

- Preserves existing data by copying only modified branches.\n- Returns a new immutable `Payload`.\n
"},{"location":"#dagpipe.Pipeline","title":"Pipeline dataclass","text":"
Pipeline(\n    engine: Engine,\n    state_cls: type[State],\n    initial_payload: Payload,\n)\n

Executable pipeline created from YAML configuration.

Attributes:

Name Type Description engine Engine

Execution engine responsible for running the pipeline.

state_cls Type[State]

Dynamically created State subclass with configured schema.

initial_payload Payload

Default payload used when execution begins.

Notes

Responsibilities:

- Encapsulates engine, state type, and initial payload.\n- Provides a simplified interface for executing configured pipelines.\n- Safe for concurrent execution if underlying nodes are thread-safe.\n
"},{"location":"#dagpipe.Pipeline-functions","title":"Functions","text":""},{"location":"#dagpipe.Pipeline.run","title":"run","text":"
run(\n    payload_override: Mapping[str, Any] | None = None,\n) -> list[State]\n

Execute the pipeline.

Parameters:

Name Type Description Default payload_override Mapping[str, Any] | None

Payload values overriding initial payload.

None

Returns:

Type Description list[State]

list[State]: Terminal execution states.

Notes

Responsibilities:

- Merges override payload with initial payload.\n- Creates root `State` and executes engine.\n
"},{"location":"#dagpipe.ProgressMessage","title":"ProgressMessage","text":"
ProgressMessage(\n    *,\n    lines: int | None = None,\n    blocks: int | None = None,\n    count: int | None = None,\n    unit: str | None = None,\n    raw_ocr_line: str | None = None,\n    error: str | None = None,\n    step: str = \"\",\n    status: str = \"\"\n)\n

Lightweight progress payload emitted by engine step hooks.

Mirrors the imperative ProgressMessage used by the legacy orchestrator so callers can surface counts/lines/errors without coupling the engine to pydantic.

"},{"location":"#dagpipe.Schema","title":"Schema dataclass","text":"
Schema(tree: Mapping[str, SchemaNode])\n

Immutable hierarchical schema defining allowed payload structure.

Attributes:

Name Type Description tree Mapping[str, SchemaNode]

Hierarchical schema definition.

Notes

Responsibilities:

- Validates `State` payloads and updates.\n- Reusable across all `State` instances.\n- Fully thread-safe due to immutability.\n
"},{"location":"#dagpipe.Schema-functions","title":"Functions","text":""},{"location":"#dagpipe.Schema.validate_payload","title":"validate_payload","text":"
validate_payload(payload: Payload) -> None\n

Validate complete payload structure.

Parameters:

Name Type Description Default payload Payload

Payload to validate.

required

Raises:

Type Description SchemaError

If payload violates schema.

"},{"location":"#dagpipe.Schema.validate_update","title":"validate_update","text":"
validate_update(updates: Mapping[str, Any]) -> None\n

Validate payload update paths.

Parameters:

Name Type Description Default updates Mapping[str, Any]

Dot-path updates to validate.

required

Raises:

Type Description SchemaError

If any path is invalid according to the schema.

"},{"location":"#dagpipe.SchemaError","title":"SchemaError","text":"

Bases: Exception

Raised when payload data violates the declared schema.

"},{"location":"#dagpipe.SchemaError--indicates-invalid-structure-invalid-path-or-invalid-type","title":"Indicates invalid structure, invalid path, or invalid type.","text":""},{"location":"#dagpipe.State","title":"State dataclass","text":"
State(\n    payload: Payload,\n    confidence: float = 1.0,\n    parent: State | None = None,\n    depth: int = 0,\n    history: tuple[str, ...] = tuple(),\n    metadata: dict[str, Any] = dict(),\n)\n

Immutable execution state propagated through dagpipe pipeline.

Attributes:

Name Type Description payload Payload

Execution data container.

schema ClassVar[Schema]

Payload validation schema.

confidence float

Execution confidence score.

parent Optional[State]

Parent state reference.

depth int

Execution depth.

history Tuple[str, ...]

Ordered node execution lineage.

metadata Dict[str, Any]

Execution metadata.

Notes

Responsibilities:

- Represents a complete execution snapshot at a specific point in\n  pipeline traversal.\n- Fundamental unit of execution in `dagpipe`.\n- Fully thread-safe due to immutability.\n
"},{"location":"#dagpipe.State-functions","title":"Functions","text":""},{"location":"#dagpipe.State.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Concise debug representation.

Avoids printing full data for large states.

"},{"location":"#dagpipe.State.fork","title":"fork","text":"
fork(\n    *,\n    payload_update: Mapping[str, Any] | None = None,\n    confidence_delta: float = 0.0,\n    node_id: str | None = None,\n    metadata_update: Mapping[str, Any] | None = None\n) -> State\n

Create a new child State derived from this state.

Parameters:

Name Type Description Default payload_update Mapping[str, Any] | None

Dot-path updates applied to the payload.

None confidence_delta float

Adjustment applied to current confidence.

0.0 node_id str | None

Identifier of the node creating this state.

None metadata_update Mapping[str, Any] | None

Updates merged into state metadata.

None

Returns:

Name Type Description State State

A new immutable State instance.

Notes

Guarantees:

- This is the only supported mechanism for modifying execution data.\n- Validates payload updates, preserves lineage, increments depth,\n  and appends to history.\n
"},{"location":"#dagpipe.State.get","title":"get","text":"
get(key: str, default: Any = None) -> Any\n

Retrieve payload value.

Parameters:

Name Type Description Default key str

Dot-path key.

required default Any

Fallback value.

None

Returns:

Name Type Description Any Any

Stored value or default.

"},{"location":"#dagpipe.State.has","title":"has","text":"
has(key: str) -> bool\n

Check whether payload contains key.

Parameters:

Name Type Description Default key str

Dot-path key.

required

Returns:

Name Type Description bool bool

Existence of the key.

"},{"location":"#dagpipe.State.lineage","title":"lineage","text":"
lineage() -> tuple[State, ...]\n

Return lineage from root to this State.

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Ordered execution lineage (root first).

"},{"location":"#dagpipe.StepResult","title":"StepResult","text":"
StepResult(\n    index: int,\n    node_id: str,\n    states: tuple[State, ...],\n    completed: bool,\n)\n

A single checkpointed step within an async/resumable engine run.

Attributes:

Name Type Description index int

Ordinal index of the step.

node_id str

Identifier of the node associated with this step.

states tuple[State, ...]

States produced by running this step.

completed bool

Whether this step succeeded (vs. paused/interrupted).

Initialise StepResult.

Parameters:

Name Type Description Default index int

Ordinal index of the step.

required node_id str

Identifier of the node associated with this step.

required states tuple[State, ...]

States produced by running this step.

required completed bool

Whether this step succeeded (vs. paused/interrupted).

required"},{"location":"#dagpipe.StepResult-functions","title":"Functions","text":""},{"location":"#dagpipe-functions","title":"Functions","text":""},{"location":"#dagpipe.load_pipeline","title":"load_pipeline","text":"
load_pipeline(path: str) -> Pipeline\n

Load pipeline from YAML file.

Parameters:

Name Type Description Default path str

Path to YAML configuration file.

required

Returns:

Name Type Description Pipeline Pipeline

Executable pipeline instance.

Notes

Responsibilities:

- Loads YAML configuration and builds schema.\n- Creates `State` subclass and loads `Node` instances.\n- Builds `Graph` topology and initializes `Engine`.\n
"},{"location":"engine/","title":"Engine","text":""},{"location":"engine/#dagpipe.engine","title":"dagpipe.engine","text":""},{"location":"engine/#dagpipe.engine--summary","title":"Summary","text":"

Execution engine responsible for running pipelines and graphs.

The Engine executes Node objects and propagates immutable State instances through either a linear sequence or a directed acyclic graph (Graph). It orchestrates execution order, branching, and state propagation.

"},{"location":"engine/#dagpipe.engine--guarantees","title":"Guarantees","text":""},{"location":"engine/#dagpipe.engine-classes","title":"Classes","text":""},{"location":"engine/#dagpipe.engine.Engine","title":"Engine","text":"
Engine(\n    nodes_or_graph: Sequence[Node] | Graph,\n    *,\n    on_step: StepHook | None = None\n)\n

Execution engine responsible for running pipeline logic.

Notes

Responsibilities:

- Accepts either a linear sequence of `Node` objects or a `Graph`\n  defining execution topology.\n- Propagates immutable `State` objects through `Node` objects and\n  collects terminal states.\n- Supports synchronous (`run`) and asynchronous (`run_async`)\n  execution, dispatching per-node.\n- Supports step-wise / resumable execution and progress hooks.\n

Guarantees:

- Never mutates `State`, `Node`, or `Graph` instances.\n- `State` objects are never modified in place; each branch produces\n  independent instances.\n- Execution order is deterministic and follows graph or pipeline topology.\n- Thread-safe for concurrent execution.\n
"},{"location":"engine/#dagpipe.engine.Engine-attributes","title":"Attributes","text":""},{"location":"engine/#dagpipe.engine.Engine.nodes","title":"nodes property","text":"
nodes: tuple[Node, ...]\n

Return nodes managed by this engine.

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Ordered sequence in linear mode or all nodes in graph mode.

"},{"location":"engine/#dagpipe.engine.Engine-functions","title":"Functions","text":""},{"location":"engine/#dagpipe.engine.Engine.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return the canonical string representation of the object.

Returns:

Name Type Description str str

Representation that uniquely identifies the object and its configuration.

"},{"location":"engine/#dagpipe.engine.Engine.run","title":"run","text":"
run(root: State) -> list[State]\n

Execute the pipeline starting from a root State.

Parameters:

Name Type Description Default root State

Initial execution state.

required

Returns:

Type Description list[State]

list[State]: Terminal execution states produced by the pipeline.

Raises:

Type Description TypeError

If root is not a State instance.

RuntimeError

If the engine execution mode is invalid.

Notes

Responsibilities:

- Selects execution mode, propagates state through nodes, creates\n  new instances for branches, and collects terminal states.\n
"},{"location":"engine/#dagpipe.engine.Engine.run_async","title":"run_async async","text":"
run_async(root: State) -> list[State]\n

Execute the pipeline starting from root, dispatching sync vs async nodes.

Parameters:

Name Type Description Default root State

Initial execution state.

required

Returns:

Type Description list[State]

list[State]: Terminal execution states produced by the pipeline.

Notes

Each node is executed with Node.run when synchronous and AsyncNode.run_async when asynchronous. Linear and graph topologies are both supported.

"},{"location":"engine/#dagpipe.engine.Engine.run_steps","title":"run_steps","text":"
run_steps(\n    root: State,\n    *,\n    resume_from: int | None = None,\n    on_step: StepHook | None = None\n) -> Iterator[StepResult]\n

Execute the pipeline step-by-step, yielding one StepResult per step.

Parameters:

Name Type Description Default root State

Initial execution state.

required resume_from int | None

Skip steps at index < resume_from (for resume-after-partial). Steps are 0-indexed.

None on_step StepHook | None

Callback (step, status, message) invoked per step; falls back to the engine-level hook when unset.

None

Yields:

Name Type Description StepResult StepResult

One per executed node/step, carrying the produced states.

Notes

This is a synchronous, generator-based checkpoint interface compatible with the imperative resume-by-step behaviour of the legacy orchestrator. Use run_steps_async for async nodes.

"},{"location":"engine/#dagpipe.engine.Engine.run_steps_async","title":"run_steps_async async","text":"
run_steps_async(\n    root: State,\n    *,\n    resume_from: int | None = None,\n    on_step: AsyncStepHook | None = None\n) -> AsyncIterator[StepResult]\n

Async variant of run_steps supporting AsyncNode execution.

Parameters:

Name Type Description Default root State

Initial execution state.

required resume_from int | None

Skip steps at index < resume_from.

None on_step AsyncStepHook | None

Async callback (step, status, message) invoked per step.

None

Yields:

Name Type Description StepResult AsyncIterator[StepResult]

One per executed node/step.

"},{"location":"engine/#dagpipe.engine.ProgressMessage","title":"ProgressMessage","text":"
ProgressMessage(\n    *,\n    lines: int | None = None,\n    blocks: int | None = None,\n    count: int | None = None,\n    unit: str | None = None,\n    raw_ocr_line: str | None = None,\n    error: str | None = None,\n    step: str = \"\",\n    status: str = \"\"\n)\n

Lightweight progress payload emitted by engine step hooks.

Mirrors the imperative ProgressMessage used by the legacy orchestrator so callers can surface counts/lines/errors without coupling the engine to pydantic.

"},{"location":"engine/#dagpipe.engine.StepResult","title":"StepResult","text":"
StepResult(\n    index: int,\n    node_id: str,\n    states: tuple[State, ...],\n    completed: bool,\n)\n

A single checkpointed step within an async/resumable engine run.

Attributes:

Name Type Description index int

Ordinal index of the step.

node_id str

Identifier of the node associated with this step.

states tuple[State, ...]

States produced by running this step.

completed bool

Whether this step succeeded (vs. paused/interrupted).

Initialise StepResult.

Parameters:

Name Type Description Default index int

Ordinal index of the step.

required node_id str

Identifier of the node associated with this step.

required states tuple[State, ...]

States produced by running this step.

required completed bool

Whether this step succeeded (vs. paused/interrupted).

required"},{"location":"engine/#dagpipe.engine.StepResult-functions","title":"Functions","text":""},{"location":"graph/","title":"Graph","text":""},{"location":"graph/#dagpipe.graph","title":"dagpipe.graph","text":""},{"location":"graph/#dagpipe.graph--summary","title":"Summary","text":"

Defines DAG structure connecting nodes.

A Graph describes execution topology only. It does not execute nodes or manage State. Execution is handled by an Engine.

"},{"location":"graph/#dagpipe.graph--responsibilities","title":"Responsibilities","text":""},{"location":"graph/#dagpipe.graph-classes","title":"Classes","text":""},{"location":"graph/#dagpipe.graph.Graph","title":"Graph","text":"
Graph()\n

Directed Acyclic Graph defining execution topology of Node objects.

Notes

Responsibilities:

- Stores node connectivity and validates that the topology remains acyclic.\n- Structure determines how `State` flows between nodes during execution.\n

Guarantees:

- Topology is acyclic. Node relationships remain consistent.\n- Thread-safe for concurrent reads after construction.\n

Create an empty Graph.

"},{"location":"graph/#dagpipe.graph.Graph--initializes-node-registry-and-edge-mappings","title":"Initializes node registry and edge mappings.","text":""},{"location":"graph/#dagpipe.graph.Graph-functions","title":"Functions","text":""},{"location":"graph/#dagpipe.graph.Graph.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"graph/#dagpipe.graph.Graph.__repr__--returns","title":"Returns","text":"

str

"},{"location":"graph/#dagpipe.graph.Graph.add_edge","title":"add_edge","text":"
add_edge(src: Node, dst: Node) -> None\n

Add a directed edge from src to dst.

Parameters:

Name Type Description Default src Node

Source node.

required dst Node

Destination node.

required

Raises:

Type Description TypeError

If src or dst is not a Node.

ValueError

If the edge would create a cycle or if src and dst are common.

Notes "},{"location":"graph/#dagpipe.graph.Graph.add_root","title":"add_root","text":"
add_root(node: Node) -> None\n

Add a root node with no parents.

Parameters:

Name Type Description Default node Node

Node to add as a root.

required

Raises:

Type Description TypeError

If node is not a Node instance.

"},{"location":"graph/#dagpipe.graph.Graph.children","title":"children","text":"
children(node: Node) -> tuple[Node, ...]\n

Return child nodes of a node.

Parameters:

Name Type Description Default node Node

Node to query.

required

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Outgoing neighbors.

"},{"location":"graph/#dagpipe.graph.Graph.nodes","title":"nodes","text":"
nodes() -> tuple[Node, ...]\n

Return all nodes in the graph.

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: All registered nodes.

"},{"location":"graph/#dagpipe.graph.Graph.parents","title":"parents","text":"
parents(node: Node) -> tuple[Node, ...]\n

Return parent nodes of a node.

Parameters:

Name Type Description Default node Node

Node to query.

required

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Incoming neighbors.

"},{"location":"graph/#dagpipe.graph.Graph.roots","title":"roots","text":"
roots() -> tuple[Node, ...]\n

Return root nodes (nodes with no incoming edges).

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Entry point nodes.

"},{"location":"node/","title":"Node","text":""},{"location":"node/#dagpipe.node","title":"dagpipe.node","text":""},{"location":"node/#dagpipe.node--summary","title":"Summary","text":"

Defines the Node abstraction used by dagpipe.

A node represents a single unit of pipeline execution logic. It consumes one State and produces zero, one, or many new State objects.

Nodes are connected using a Graph and executed by an Engine.

"},{"location":"node/#dagpipe.node--design-principles","title":"Design principles","text":""},{"location":"node/#dagpipe.node-classes","title":"Classes","text":""},{"location":"node/#dagpipe.node.AsyncNode","title":"AsyncNode","text":"

Bases: Node

Base class for nodes whose execution is asynchronous.

Subclasses implement resolve_async (an async generator yielding derived State objects). The engine dispatches to resolve_async when running an async traversal (see Engine.run_async).

Sync-only engines (and the base Node.run) treat an AsyncNode as a no-op consumer: calling run on an AsyncNode returns no states, signalling that an async engine is required.

"},{"location":"node/#dagpipe.node.AsyncNode-functions","title":"Functions","text":""},{"location":"node/#dagpipe.node.AsyncNode.__hash__","title":"__hash__","text":"
__hash__() -> int\n

Return stable hash based on node ID.

"},{"location":"node/#dagpipe.node.AsyncNode.__hash__--returns","title":"Returns","text":"

int

"},{"location":"node/#dagpipe.node.AsyncNode.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"node/#dagpipe.node.AsyncNode.__repr__--returns","title":"Returns","text":"

str

"},{"location":"node/#dagpipe.node.AsyncNode.__str__","title":"__str__","text":"
__str__() -> str\n

Return display representation.

"},{"location":"node/#dagpipe.node.AsyncNode.__str__--returns","title":"Returns","text":"

str

"},{"location":"node/#dagpipe.node.AsyncNode.clean_id_and_name","title":"clean_id_and_name classmethod","text":"
clean_id_and_name() -> None\n

Normalize and validate node ID and display name.

Raises:

Type Description TypeError

If ID is not a string.

ValueError

If ID format is invalid.

Notes

Guarantees:

- Generates ID from module and class name if missing.\n- Validates ID format.\n- Generates human-readable name if missing.\n
"},{"location":"node/#dagpipe.node.AsyncNode.fork","title":"fork","text":"
fork(\n    state: State,\n    *,\n    payload_update: Any = None,\n    confidence_delta: float = 0.0,\n    metadata_update: Any = None\n) -> State\n

Create a child State attributed to this node.

Parameters:

Name Type Description Default state State

Parent execution state.

required payload_update Any

Dot-path payload updates.

None confidence_delta float

Confidence adjustment.

0.0 metadata_update Any

Metadata updates.

None

Returns:

Name Type Description State State

New child execution state.

Notes

Responsibilities:

- Convenience wrapper around `State.fork()` that automatically\n  records this node's ID in state history.\n
"},{"location":"node/#dagpipe.node.AsyncNode.is_async","title":"is_async","text":"
is_async() -> bool\n

Return whether this node executes asynchronously.

"},{"location":"node/#dagpipe.node.AsyncNode.node_id_to_name","title":"node_id_to_name staticmethod","text":"
node_id_to_name(node_id: str) -> str\n

Convert a dotted snake_case node ID into a human-readable name.

Parameters:

Name Type Description Default node_id str

Unique node identifier (e.g., 'entity.resolve.numeric_merchant').

required

Returns:

Name Type Description str str

Human-readable display name (e.g., 'Entity \u203a Resolve \u203a Numeric Merchant').

"},{"location":"node/#dagpipe.node.AsyncNode.resolve_async","title":"resolve_async async","text":"
resolve_async(state: State) -> Iterable[State]\n

Execute node logic asynchronously.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description Iterable[State]

Iterable[State]: Derived execution state(s).

Notes

Subclasses implement this. Must not mutate the input state. Should use fork() to create child states.

"},{"location":"node/#dagpipe.node.AsyncNode.run","title":"run","text":"
run(state: State) -> tuple[State, ...]\n

Execute this node on a State.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Derived execution states.

Raises:

Type Description TypeError

If resolve() yields a non-State object.

"},{"location":"node/#dagpipe.node.AsyncNode.run_async","title":"run_async async","text":"
run_async(state: State) -> tuple[State, ...]\n

Execute this node asynchronously on a state, validating outputs.

"},{"location":"node/#dagpipe.node.Node","title":"Node","text":"

Bases: ABC

Base class for all dagpipe execution nodes.

Attributes:

Name Type Description id str

Unique identifier of the node (snake_case dotted format).

name str

Human-readable display name.

Notes

Responsibilities:

- Represents a deterministic unit of execution in the pipeline graph.\n- Consumes one `State` and produces zero, one, or many derived states.\n- Defines execution logic and enables branching, filtering, and transformation.\n

Guarantees:

- Nodes must never mutate the input `State`.\n- Instances are singletons per subclass and reused across executions.\n
"},{"location":"node/#dagpipe.node.Node-functions","title":"Functions","text":""},{"location":"node/#dagpipe.node.Node.__hash__","title":"__hash__","text":"
__hash__() -> int\n

Return stable hash based on node ID.

"},{"location":"node/#dagpipe.node.Node.__hash__--returns","title":"Returns","text":"

int

"},{"location":"node/#dagpipe.node.Node.__new__","title":"__new__","text":"
__new__(*args: Any, **kwargs: Any) -> Node\n

Create or reuse a Node instance.

Stateless subclasses (no parameterized __init__) share one singleton instance per class \u2014 matching the original dagpipe behaviour underpinning set_registry-style configuration. Subclasses that declare an __init__ requiring instance-state arguments get a fresh instance per construction so pipeline builders can inject per-run dependencies.

"},{"location":"node/#dagpipe.node.Node.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"node/#dagpipe.node.Node.__repr__--returns","title":"Returns","text":"

str

"},{"location":"node/#dagpipe.node.Node.__str__","title":"__str__","text":"
__str__() -> str\n

Return display representation.

"},{"location":"node/#dagpipe.node.Node.__str__--returns","title":"Returns","text":"

str

"},{"location":"node/#dagpipe.node.Node.clean_id_and_name","title":"clean_id_and_name classmethod","text":"
clean_id_and_name() -> None\n

Normalize and validate node ID and display name.

Raises:

Type Description TypeError

If ID is not a string.

ValueError

If ID format is invalid.

Notes

Guarantees:

- Generates ID from module and class name if missing.\n- Validates ID format.\n- Generates human-readable name if missing.\n
"},{"location":"node/#dagpipe.node.Node.fork","title":"fork","text":"
fork(\n    state: State,\n    *,\n    payload_update: Any = None,\n    confidence_delta: float = 0.0,\n    metadata_update: Any = None\n) -> State\n

Create a child State attributed to this node.

Parameters:

Name Type Description Default state State

Parent execution state.

required payload_update Any

Dot-path payload updates.

None confidence_delta float

Confidence adjustment.

0.0 metadata_update Any

Metadata updates.

None

Returns:

Name Type Description State State

New child execution state.

Notes

Responsibilities:

- Convenience wrapper around `State.fork()` that automatically\n  records this node's ID in state history.\n
"},{"location":"node/#dagpipe.node.Node.is_async","title":"is_async","text":"
is_async() -> bool\n

Return whether this node executes asynchronously.

"},{"location":"node/#dagpipe.node.Node.node_id_to_name","title":"node_id_to_name staticmethod","text":"
node_id_to_name(node_id: str) -> str\n

Convert a dotted snake_case node ID into a human-readable name.

Parameters:

Name Type Description Default node_id str

Unique node identifier (e.g., 'entity.resolve.numeric_merchant').

required

Returns:

Name Type Description str str

Human-readable display name (e.g., 'Entity \u203a Resolve \u203a Numeric Merchant').

"},{"location":"node/#dagpipe.node.Node.resolve","title":"resolve abstractmethod","text":"
resolve(state: State) -> Iterable[State]\n

Execute node logic.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description Iterable[State]

Iterable[State]: Derived execution state(s).

Notes

Responsibilities:

- Subclasses implement specific resolution behavior.\n- Must not mutate input state.\n- Should use `fork()` to create child states.\n- May yield zero states to terminate a branch.\n
"},{"location":"node/#dagpipe.node.Node.run","title":"run","text":"
run(state: State) -> tuple[State, ...]\n

Execute this node on a State.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Derived execution states.

Raises:

Type Description TypeError

If resolve() yields a non-State object.

"},{"location":"state/","title":"State","text":""},{"location":"state/#dagpipe.state","title":"dagpipe.state","text":""},{"location":"state/#dagpipe.state--summary","title":"Summary","text":"

Defines the core State object used by dagpipe.

The State represents a single point in pipeline execution. It contains arbitrary data and metadata and is designed to be immutable. Instead of modifying an existing state, nodes create new child states via fork().

"},{"location":"state/#dagpipe.state--design-principles","title":"Design principles","text":""},{"location":"state/#dagpipe.state-classes","title":"Classes","text":""},{"location":"state/#dagpipe.state.Payload","title":"Payload dataclass","text":"
Payload(_data: Mapping[str, Any])\n

Immutable hierarchical container with dot-path access.

Attributes:

Name Type Description _data Mapping[str, Any]

Immutable hierarchical data structure.

Notes

Responsibilities:

- Stores execution data used by `State`.\n- Supports efficient atomic updates without modifying existing instances.\n- `Payload` instances are fully thread-safe due to immutability.\n
"},{"location":"state/#dagpipe.state.Payload-functions","title":"Functions","text":""},{"location":"state/#dagpipe.state.Payload.as_dict","title":"as_dict","text":"
as_dict() -> Mapping[str, Any]\n

Return underlying mapping.

Returns:

Type Description Mapping[str, Any]

Mapping[str, Any]: Read-only view of the underlying data.

"},{"location":"state/#dagpipe.state.Payload.get","title":"get","text":"
get(path: str, default: Any = None) -> Any\n

Retrieve value using dot-path.

Parameters:

Name Type Description Default path str

Dot-separated path to the value.

required default Any

Default value if path doesn't exist.

None

Returns:

Name Type Description Any Any

The retrieved value or default.

"},{"location":"state/#dagpipe.state.Payload.has","title":"has","text":"
has(path: str) -> bool\n

Return True if path exists.

Parameters:

Name Type Description Default path str

Dot-separated path to check.

required

Returns:

Name Type Description bool bool

Existence of the path.

"},{"location":"state/#dagpipe.state.Payload.iter_paths","title":"iter_paths classmethod","text":"
iter_paths(\n    data: Mapping[str, Any], prefix: str = \"\"\n) -> Iterable[str]\n

Recursively yield dot-paths for all leaf nodes.

Parameters:

Name Type Description Default data Mapping[str, Any]

The mapping to iterate over.

required prefix str

Current path prefix.

''

Yields:

Name Type Description str Iterable[str]

Dot-path for each leaf node.

"},{"location":"state/#dagpipe.state.Payload.keys","title":"keys","text":"
keys() -> Iterable[str]\n

Return top-level keys.

Returns:

Type Description Iterable[str]

Iterable[str]: Iterator over top-level keys.

"},{"location":"state/#dagpipe.state.Payload.update","title":"update","text":"
update(updates: Mapping[str, Any]) -> Payload\n

Create a new Payload with dot-path updates applied.

Parameters:

Name Type Description Default updates Mapping[str, Any]

Dot-path to value mapping.

required

Returns:

Name Type Description Payload Payload

New immutable payload instance with updates.

Notes

Guarantees:

- Preserves existing data by copying only modified branches.\n- Returns a new immutable `Payload`.\n
"},{"location":"state/#dagpipe.state.Schema","title":"Schema dataclass","text":"
Schema(tree: Mapping[str, SchemaNode])\n

Immutable hierarchical schema defining allowed payload structure.

Attributes:

Name Type Description tree Mapping[str, SchemaNode]

Hierarchical schema definition.

Notes

Responsibilities:

- Validates `State` payloads and updates.\n- Reusable across all `State` instances.\n- Fully thread-safe due to immutability.\n
"},{"location":"state/#dagpipe.state.Schema-functions","title":"Functions","text":""},{"location":"state/#dagpipe.state.Schema.validate_payload","title":"validate_payload","text":"
validate_payload(payload: Payload) -> None\n

Validate complete payload structure.

Parameters:

Name Type Description Default payload Payload

Payload to validate.

required

Raises:

Type Description SchemaError

If payload violates schema.

"},{"location":"state/#dagpipe.state.Schema.validate_update","title":"validate_update","text":"
validate_update(updates: Mapping[str, Any]) -> None\n

Validate payload update paths.

Parameters:

Name Type Description Default updates Mapping[str, Any]

Dot-path updates to validate.

required

Raises:

Type Description SchemaError

If any path is invalid according to the schema.

"},{"location":"state/#dagpipe.state.SchemaError","title":"SchemaError","text":"

Bases: Exception

Raised when payload data violates the declared schema.

"},{"location":"state/#dagpipe.state.SchemaError--indicates-invalid-structure-invalid-path-or-invalid-type","title":"Indicates invalid structure, invalid path, or invalid type.","text":""},{"location":"state/#dagpipe.state.State","title":"State dataclass","text":"
State(\n    payload: Payload,\n    confidence: float = 1.0,\n    parent: State | None = None,\n    depth: int = 0,\n    history: tuple[str, ...] = tuple(),\n    metadata: dict[str, Any] = dict(),\n)\n

Immutable execution state propagated through dagpipe pipeline.

Attributes:

Name Type Description payload Payload

Execution data container.

schema ClassVar[Schema]

Payload validation schema.

confidence float

Execution confidence score.

parent Optional[State]

Parent state reference.

depth int

Execution depth.

history Tuple[str, ...]

Ordered node execution lineage.

metadata Dict[str, Any]

Execution metadata.

Notes

Responsibilities:

- Represents a complete execution snapshot at a specific point in\n  pipeline traversal.\n- Fundamental unit of execution in `dagpipe`.\n- Fully thread-safe due to immutability.\n
"},{"location":"state/#dagpipe.state.State-functions","title":"Functions","text":""},{"location":"state/#dagpipe.state.State.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Concise debug representation.

Avoids printing full data for large states.

"},{"location":"state/#dagpipe.state.State.fork","title":"fork","text":"
fork(\n    *,\n    payload_update: Mapping[str, Any] | None = None,\n    confidence_delta: float = 0.0,\n    node_id: str | None = None,\n    metadata_update: Mapping[str, Any] | None = None\n) -> State\n

Create a new child State derived from this state.

Parameters:

Name Type Description Default payload_update Mapping[str, Any] | None

Dot-path updates applied to the payload.

None confidence_delta float

Adjustment applied to current confidence.

0.0 node_id str | None

Identifier of the node creating this state.

None metadata_update Mapping[str, Any] | None

Updates merged into state metadata.

None

Returns:

Name Type Description State State

A new immutable State instance.

Notes

Guarantees:

- This is the only supported mechanism for modifying execution data.\n- Validates payload updates, preserves lineage, increments depth,\n  and appends to history.\n
"},{"location":"state/#dagpipe.state.State.get","title":"get","text":"
get(key: str, default: Any = None) -> Any\n

Retrieve payload value.

Parameters:

Name Type Description Default key str

Dot-path key.

required default Any

Fallback value.

None

Returns:

Name Type Description Any Any

Stored value or default.

"},{"location":"state/#dagpipe.state.State.has","title":"has","text":"
has(key: str) -> bool\n

Check whether payload contains key.

Parameters:

Name Type Description Default key str

Dot-path key.

required

Returns:

Name Type Description bool bool

Existence of the key.

"},{"location":"state/#dagpipe.state.State.lineage","title":"lineage","text":"
lineage() -> tuple[State, ...]\n

Return lineage from root to this State.

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Ordered execution lineage (root first).

"},{"location":"yaml_loader/","title":"Yaml Loader","text":""},{"location":"yaml_loader/#dagpipe.yaml_loader","title":"dagpipe.yaml_loader","text":""},{"location":"yaml_loader/#dagpipe.yaml_loader--summary","title":"Summary","text":"

Loads dagpipe pipelines from YAML configuration.

Creates fully configured pipeline objects from declarative YAML definitions, including Schema, State subclasses, Node instances, Graph topology, and initial payloads.

"},{"location":"yaml_loader/#dagpipe.yaml_loader-classes","title":"Classes","text":""},{"location":"yaml_loader/#dagpipe.yaml_loader.Pipeline","title":"Pipeline dataclass","text":"
Pipeline(\n    engine: Engine,\n    state_cls: type[State],\n    initial_payload: Payload,\n)\n

Executable pipeline created from YAML configuration.

Attributes:

Name Type Description engine Engine

Execution engine responsible for running the pipeline.

state_cls Type[State]

Dynamically created State subclass with configured schema.

initial_payload Payload

Default payload used when execution begins.

Notes

Responsibilities:

- Encapsulates engine, state type, and initial payload.\n- Provides a simplified interface for executing configured pipelines.\n- Safe for concurrent execution if underlying nodes are thread-safe.\n
"},{"location":"yaml_loader/#dagpipe.yaml_loader.Pipeline-functions","title":"Functions","text":""},{"location":"yaml_loader/#dagpipe.yaml_loader.Pipeline.run","title":"run","text":"
run(\n    payload_override: Mapping[str, Any] | None = None,\n) -> list[State]\n

Execute the pipeline.

Parameters:

Name Type Description Default payload_override Mapping[str, Any] | None

Payload values overriding initial payload.

None

Returns:

Type Description list[State]

list[State]: Terminal execution states.

Notes

Responsibilities:

- Merges override payload with initial payload.\n- Creates root `State` and executes engine.\n
"},{"location":"yaml_loader/#dagpipe.yaml_loader-functions","title":"Functions","text":""},{"location":"yaml_loader/#dagpipe.yaml_loader.load_pipeline","title":"load_pipeline","text":"
load_pipeline(path: str) -> Pipeline\n

Load pipeline from YAML file.

Parameters:

Name Type Description Default path str

Path to YAML configuration file.

required

Returns:

Name Type Description Pipeline Pipeline

Executable pipeline instance.

Notes

Responsibilities:

- Loads YAML configuration and builds schema.\n- Creates `State` subclass and loads `Node` instances.\n- Builds `Graph` topology and initializes `Engine`.\n
"},{"location":"dagpipe/","title":"Dagpipe","text":""},{"location":"dagpipe/#dagpipe","title":"dagpipe","text":""},{"location":"dagpipe/#dagpipe--summary","title":"Summary","text":"

Directed acyclic graph execution framework for deterministic state propagation.

dagpipe executes pipelines composed of nodes connected in a directed acyclic graph (DAG). Each node receives an immutable State and optionally produces derived states for downstream nodes.

"},{"location":"dagpipe/#dagpipe--installation","title":"Installation","text":"

Install using pip:

pip install dagpipe\n
"},{"location":"dagpipe/#dagpipe--quick-start","title":"Quick Start","text":"
from dagpipe import State, Payload, Schema, Graph, Engine\nfrom dagpipe.node import Node\n\nclass HelloNode(Node):\n    id = \"hello\"\n    def resolve(self, state):\n        yield self.fork(state, payload_update={\"msg\": \"hello\"})\n\n# Build and run\ngraph = Graph()\ngraph.add_root(HelloNode())\nengine = Engine(graph)\n\nclass MyState(State):\n    schema = Schema({})\n\nresults = engine.run(MyState(payload=Payload({})))\n
"},{"location":"dagpipe/#dagpipe--public-api","title":"Public API","text":"

This package re-exports the core pipeline components. Consumers should import from this namespace for standard usage.

"},{"location":"dagpipe/#dagpipe--execution-core","title":"Execution Core","text":""},{"location":"dagpipe/#dagpipe--state-data","title":"State & Data","text":""},{"location":"dagpipe/#dagpipe--declarative-pipelines","title":"Declarative Pipelines","text":""},{"location":"dagpipe/#dagpipe-classes","title":"Classes","text":""},{"location":"dagpipe/#dagpipe.AsyncNode","title":"AsyncNode","text":"

Bases: Node

Base class for nodes whose execution is asynchronous.

Subclasses implement resolve_async (an async generator yielding derived State objects). The engine dispatches to resolve_async when running an async traversal (see Engine.run_async).

Sync-only engines (and the base Node.run) treat an AsyncNode as a no-op consumer: calling run on an AsyncNode returns no states, signalling that an async engine is required.

"},{"location":"dagpipe/#dagpipe.AsyncNode-functions","title":"Functions","text":""},{"location":"dagpipe/#dagpipe.AsyncNode.__hash__","title":"__hash__","text":"
__hash__() -> int\n

Return stable hash based on node ID.

"},{"location":"dagpipe/#dagpipe.AsyncNode.__hash__--returns","title":"Returns","text":"

int

"},{"location":"dagpipe/#dagpipe.AsyncNode.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"dagpipe/#dagpipe.AsyncNode.__repr__--returns","title":"Returns","text":"

str

"},{"location":"dagpipe/#dagpipe.AsyncNode.__str__","title":"__str__","text":"
__str__() -> str\n

Return display representation.

"},{"location":"dagpipe/#dagpipe.AsyncNode.__str__--returns","title":"Returns","text":"

str

"},{"location":"dagpipe/#dagpipe.AsyncNode.clean_id_and_name","title":"clean_id_and_name classmethod","text":"
clean_id_and_name() -> None\n

Normalize and validate node ID and display name.

Raises:

Type Description TypeError

If ID is not a string.

ValueError

If ID format is invalid.

Notes

Guarantees:

- Generates ID from module and class name if missing.\n- Validates ID format.\n- Generates human-readable name if missing.\n
"},{"location":"dagpipe/#dagpipe.AsyncNode.fork","title":"fork","text":"
fork(\n    state: State,\n    *,\n    payload_update: Any = None,\n    confidence_delta: float = 0.0,\n    metadata_update: Any = None\n) -> State\n

Create a child State attributed to this node.

Parameters:

Name Type Description Default state State

Parent execution state.

required payload_update Any

Dot-path payload updates.

None confidence_delta float

Confidence adjustment.

0.0 metadata_update Any

Metadata updates.

None

Returns:

Name Type Description State State

New child execution state.

Notes

Responsibilities:

- Convenience wrapper around `State.fork()` that automatically\n  records this node's ID in state history.\n
"},{"location":"dagpipe/#dagpipe.AsyncNode.is_async","title":"is_async","text":"
is_async() -> bool\n

Return whether this node executes asynchronously.

"},{"location":"dagpipe/#dagpipe.AsyncNode.node_id_to_name","title":"node_id_to_name staticmethod","text":"
node_id_to_name(node_id: str) -> str\n

Convert a dotted snake_case node ID into a human-readable name.

Parameters:

Name Type Description Default node_id str

Unique node identifier (e.g., 'entity.resolve.numeric_merchant').

required

Returns:

Name Type Description str str

Human-readable display name (e.g., 'Entity \u203a Resolve \u203a Numeric Merchant').

"},{"location":"dagpipe/#dagpipe.AsyncNode.resolve_async","title":"resolve_async async","text":"
resolve_async(state: State) -> Iterable[State]\n

Execute node logic asynchronously.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description Iterable[State]

Iterable[State]: Derived execution state(s).

Notes

Subclasses implement this. Must not mutate the input state. Should use fork() to create child states.

"},{"location":"dagpipe/#dagpipe.AsyncNode.run","title":"run","text":"
run(state: State) -> tuple[State, ...]\n

Execute this node on a State.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Derived execution states.

Raises:

Type Description TypeError

If resolve() yields a non-State object.

"},{"location":"dagpipe/#dagpipe.AsyncNode.run_async","title":"run_async async","text":"
run_async(state: State) -> tuple[State, ...]\n

Execute this node asynchronously on a state, validating outputs.

"},{"location":"dagpipe/#dagpipe.Engine","title":"Engine","text":"
Engine(\n    nodes_or_graph: Sequence[Node] | Graph,\n    *,\n    on_step: StepHook | None = None\n)\n

Execution engine responsible for running pipeline logic.

Notes

Responsibilities:

- Accepts either a linear sequence of `Node` objects or a `Graph`\n  defining execution topology.\n- Propagates immutable `State` objects through `Node` objects and\n  collects terminal states.\n- Supports synchronous (`run`) and asynchronous (`run_async`)\n  execution, dispatching per-node.\n- Supports step-wise / resumable execution and progress hooks.\n

Guarantees:

- Never mutates `State`, `Node`, or `Graph` instances.\n- `State` objects are never modified in place; each branch produces\n  independent instances.\n- Execution order is deterministic and follows graph or pipeline topology.\n- Thread-safe for concurrent execution.\n
"},{"location":"dagpipe/#dagpipe.Engine-attributes","title":"Attributes","text":""},{"location":"dagpipe/#dagpipe.Engine.nodes","title":"nodes property","text":"
nodes: tuple[Node, ...]\n

Return nodes managed by this engine.

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Ordered sequence in linear mode or all nodes in graph mode.

"},{"location":"dagpipe/#dagpipe.Engine-functions","title":"Functions","text":""},{"location":"dagpipe/#dagpipe.Engine.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return the canonical string representation of the object.

Returns:

Name Type Description str str

Representation that uniquely identifies the object and its configuration.

"},{"location":"dagpipe/#dagpipe.Engine.run","title":"run","text":"
run(root: State) -> list[State]\n

Execute the pipeline starting from a root State.

Parameters:

Name Type Description Default root State

Initial execution state.

required

Returns:

Type Description list[State]

list[State]: Terminal execution states produced by the pipeline.

Raises:

Type Description TypeError

If root is not a State instance.

RuntimeError

If the engine execution mode is invalid.

Notes

Responsibilities:

- Selects execution mode, propagates state through nodes, creates\n  new instances for branches, and collects terminal states.\n
"},{"location":"dagpipe/#dagpipe.Engine.run_async","title":"run_async async","text":"
run_async(root: State) -> list[State]\n

Execute the pipeline starting from root, dispatching sync vs async nodes.

Parameters:

Name Type Description Default root State

Initial execution state.

required

Returns:

Type Description list[State]

list[State]: Terminal execution states produced by the pipeline.

Notes

Each node is executed with Node.run when synchronous and AsyncNode.run_async when asynchronous. Linear and graph topologies are both supported.

"},{"location":"dagpipe/#dagpipe.Engine.run_steps","title":"run_steps","text":"
run_steps(\n    root: State,\n    *,\n    resume_from: int | None = None,\n    on_step: StepHook | None = None\n) -> Iterator[StepResult]\n

Execute the pipeline step-by-step, yielding one StepResult per step.

Parameters:

Name Type Description Default root State

Initial execution state.

required resume_from int | None

Skip steps at index < resume_from (for resume-after-partial). Steps are 0-indexed.

None on_step StepHook | None

Callback (step, status, message) invoked per step; falls back to the engine-level hook when unset.

None

Yields:

Name Type Description StepResult StepResult

One per executed node/step, carrying the produced states.

Notes

This is a synchronous, generator-based checkpoint interface compatible with the imperative resume-by-step behaviour of the legacy orchestrator. Use run_steps_async for async nodes.

"},{"location":"dagpipe/#dagpipe.Engine.run_steps_async","title":"run_steps_async async","text":"
run_steps_async(\n    root: State,\n    *,\n    resume_from: int | None = None,\n    on_step: AsyncStepHook | None = None\n) -> AsyncIterator[StepResult]\n

Async variant of run_steps supporting AsyncNode execution.

Parameters:

Name Type Description Default root State

Initial execution state.

required resume_from int | None

Skip steps at index < resume_from.

None on_step AsyncStepHook | None

Async callback (step, status, message) invoked per step.

None

Yields:

Name Type Description StepResult AsyncIterator[StepResult]

One per executed node/step.

"},{"location":"dagpipe/#dagpipe.Graph","title":"Graph","text":"
Graph()\n

Directed Acyclic Graph defining execution topology of Node objects.

Notes

Responsibilities:

- Stores node connectivity and validates that the topology remains acyclic.\n- Structure determines how `State` flows between nodes during execution.\n

Guarantees:

- Topology is acyclic. Node relationships remain consistent.\n- Thread-safe for concurrent reads after construction.\n

Create an empty Graph.

"},{"location":"dagpipe/#dagpipe.Graph--initializes-node-registry-and-edge-mappings","title":"Initializes node registry and edge mappings.","text":""},{"location":"dagpipe/#dagpipe.Graph-functions","title":"Functions","text":""},{"location":"dagpipe/#dagpipe.Graph.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"dagpipe/#dagpipe.Graph.__repr__--returns","title":"Returns","text":"

str

"},{"location":"dagpipe/#dagpipe.Graph.add_edge","title":"add_edge","text":"
add_edge(src: Node, dst: Node) -> None\n

Add a directed edge from src to dst.

Parameters:

Name Type Description Default src Node

Source node.

required dst Node

Destination node.

required

Raises:

Type Description TypeError

If src or dst is not a Node.

ValueError

If the edge would create a cycle or if src and dst are common.

Notes "},{"location":"dagpipe/#dagpipe.Graph.add_root","title":"add_root","text":"
add_root(node: Node) -> None\n

Add a root node with no parents.

Parameters:

Name Type Description Default node Node

Node to add as a root.

required

Raises:

Type Description TypeError

If node is not a Node instance.

"},{"location":"dagpipe/#dagpipe.Graph.children","title":"children","text":"
children(node: Node) -> tuple[Node, ...]\n

Return child nodes of a node.

Parameters:

Name Type Description Default node Node

Node to query.

required

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Outgoing neighbors.

"},{"location":"dagpipe/#dagpipe.Graph.nodes","title":"nodes","text":"
nodes() -> tuple[Node, ...]\n

Return all nodes in the graph.

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: All registered nodes.

"},{"location":"dagpipe/#dagpipe.Graph.parents","title":"parents","text":"
parents(node: Node) -> tuple[Node, ...]\n

Return parent nodes of a node.

Parameters:

Name Type Description Default node Node

Node to query.

required

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Incoming neighbors.

"},{"location":"dagpipe/#dagpipe.Graph.roots","title":"roots","text":"
roots() -> tuple[Node, ...]\n

Return root nodes (nodes with no incoming edges).

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Entry point nodes.

"},{"location":"dagpipe/#dagpipe.Node","title":"Node","text":"

Bases: ABC

Base class for all dagpipe execution nodes.

Attributes:

Name Type Description id str

Unique identifier of the node (snake_case dotted format).

name str

Human-readable display name.

Notes

Responsibilities:

- Represents a deterministic unit of execution in the pipeline graph.\n- Consumes one `State` and produces zero, one, or many derived states.\n- Defines execution logic and enables branching, filtering, and transformation.\n

Guarantees:

- Nodes must never mutate the input `State`.\n- Instances are singletons per subclass and reused across executions.\n
"},{"location":"dagpipe/#dagpipe.Node-functions","title":"Functions","text":""},{"location":"dagpipe/#dagpipe.Node.__hash__","title":"__hash__","text":"
__hash__() -> int\n

Return stable hash based on node ID.

"},{"location":"dagpipe/#dagpipe.Node.__hash__--returns","title":"Returns","text":"

int

"},{"location":"dagpipe/#dagpipe.Node.__new__","title":"__new__","text":"
__new__(*args: Any, **kwargs: Any) -> Node\n

Create or reuse a Node instance.

Stateless subclasses (no parameterized __init__) share one singleton instance per class \u2014 matching the original dagpipe behaviour underpinning set_registry-style configuration. Subclasses that declare an __init__ requiring instance-state arguments get a fresh instance per construction so pipeline builders can inject per-run dependencies.

"},{"location":"dagpipe/#dagpipe.Node.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"dagpipe/#dagpipe.Node.__repr__--returns","title":"Returns","text":"

str

"},{"location":"dagpipe/#dagpipe.Node.__str__","title":"__str__","text":"
__str__() -> str\n

Return display representation.

"},{"location":"dagpipe/#dagpipe.Node.__str__--returns","title":"Returns","text":"

str

"},{"location":"dagpipe/#dagpipe.Node.clean_id_and_name","title":"clean_id_and_name classmethod","text":"
clean_id_and_name() -> None\n

Normalize and validate node ID and display name.

Raises:

Type Description TypeError

If ID is not a string.

ValueError

If ID format is invalid.

Notes

Guarantees:

- Generates ID from module and class name if missing.\n- Validates ID format.\n- Generates human-readable name if missing.\n
"},{"location":"dagpipe/#dagpipe.Node.fork","title":"fork","text":"
fork(\n    state: State,\n    *,\n    payload_update: Any = None,\n    confidence_delta: float = 0.0,\n    metadata_update: Any = None\n) -> State\n

Create a child State attributed to this node.

Parameters:

Name Type Description Default state State

Parent execution state.

required payload_update Any

Dot-path payload updates.

None confidence_delta float

Confidence adjustment.

0.0 metadata_update Any

Metadata updates.

None

Returns:

Name Type Description State State

New child execution state.

Notes

Responsibilities:

- Convenience wrapper around `State.fork()` that automatically\n  records this node's ID in state history.\n
"},{"location":"dagpipe/#dagpipe.Node.is_async","title":"is_async","text":"
is_async() -> bool\n

Return whether this node executes asynchronously.

"},{"location":"dagpipe/#dagpipe.Node.node_id_to_name","title":"node_id_to_name staticmethod","text":"
node_id_to_name(node_id: str) -> str\n

Convert a dotted snake_case node ID into a human-readable name.

Parameters:

Name Type Description Default node_id str

Unique node identifier (e.g., 'entity.resolve.numeric_merchant').

required

Returns:

Name Type Description str str

Human-readable display name (e.g., 'Entity \u203a Resolve \u203a Numeric Merchant').

"},{"location":"dagpipe/#dagpipe.Node.resolve","title":"resolve abstractmethod","text":"
resolve(state: State) -> Iterable[State]\n

Execute node logic.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description Iterable[State]

Iterable[State]: Derived execution state(s).

Notes

Responsibilities:

- Subclasses implement specific resolution behavior.\n- Must not mutate input state.\n- Should use `fork()` to create child states.\n- May yield zero states to terminate a branch.\n
"},{"location":"dagpipe/#dagpipe.Node.run","title":"run","text":"
run(state: State) -> tuple[State, ...]\n

Execute this node on a State.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Derived execution states.

Raises:

Type Description TypeError

If resolve() yields a non-State object.

"},{"location":"dagpipe/#dagpipe.Payload","title":"Payload dataclass","text":"
Payload(_data: Mapping[str, Any])\n

Immutable hierarchical container with dot-path access.

Attributes:

Name Type Description _data Mapping[str, Any]

Immutable hierarchical data structure.

Notes

Responsibilities:

- Stores execution data used by `State`.\n- Supports efficient atomic updates without modifying existing instances.\n- `Payload` instances are fully thread-safe due to immutability.\n
"},{"location":"dagpipe/#dagpipe.Payload-functions","title":"Functions","text":""},{"location":"dagpipe/#dagpipe.Payload.as_dict","title":"as_dict","text":"
as_dict() -> Mapping[str, Any]\n

Return underlying mapping.

Returns:

Type Description Mapping[str, Any]

Mapping[str, Any]: Read-only view of the underlying data.

"},{"location":"dagpipe/#dagpipe.Payload.get","title":"get","text":"
get(path: str, default: Any = None) -> Any\n

Retrieve value using dot-path.

Parameters:

Name Type Description Default path str

Dot-separated path to the value.

required default Any

Default value if path doesn't exist.

None

Returns:

Name Type Description Any Any

The retrieved value or default.

"},{"location":"dagpipe/#dagpipe.Payload.has","title":"has","text":"
has(path: str) -> bool\n

Return True if path exists.

Parameters:

Name Type Description Default path str

Dot-separated path to check.

required

Returns:

Name Type Description bool bool

Existence of the path.

"},{"location":"dagpipe/#dagpipe.Payload.iter_paths","title":"iter_paths classmethod","text":"
iter_paths(\n    data: Mapping[str, Any], prefix: str = \"\"\n) -> Iterable[str]\n

Recursively yield dot-paths for all leaf nodes.

Parameters:

Name Type Description Default data Mapping[str, Any]

The mapping to iterate over.

required prefix str

Current path prefix.

''

Yields:

Name Type Description str Iterable[str]

Dot-path for each leaf node.

"},{"location":"dagpipe/#dagpipe.Payload.keys","title":"keys","text":"
keys() -> Iterable[str]\n

Return top-level keys.

Returns:

Type Description Iterable[str]

Iterable[str]: Iterator over top-level keys.

"},{"location":"dagpipe/#dagpipe.Payload.update","title":"update","text":"
update(updates: Mapping[str, Any]) -> Payload\n

Create a new Payload with dot-path updates applied.

Parameters:

Name Type Description Default updates Mapping[str, Any]

Dot-path to value mapping.

required

Returns:

Name Type Description Payload Payload

New immutable payload instance with updates.

Notes

Guarantees:

- Preserves existing data by copying only modified branches.\n- Returns a new immutable `Payload`.\n
"},{"location":"dagpipe/#dagpipe.Pipeline","title":"Pipeline dataclass","text":"
Pipeline(\n    engine: Engine,\n    state_cls: type[State],\n    initial_payload: Payload,\n)\n

Executable pipeline created from YAML configuration.

Attributes:

Name Type Description engine Engine

Execution engine responsible for running the pipeline.

state_cls Type[State]

Dynamically created State subclass with configured schema.

initial_payload Payload

Default payload used when execution begins.

Notes

Responsibilities:

- Encapsulates engine, state type, and initial payload.\n- Provides a simplified interface for executing configured pipelines.\n- Safe for concurrent execution if underlying nodes are thread-safe.\n
"},{"location":"dagpipe/#dagpipe.Pipeline-functions","title":"Functions","text":""},{"location":"dagpipe/#dagpipe.Pipeline.run","title":"run","text":"
run(\n    payload_override: Mapping[str, Any] | None = None,\n) -> list[State]\n

Execute the pipeline.

Parameters:

Name Type Description Default payload_override Mapping[str, Any] | None

Payload values overriding initial payload.

None

Returns:

Type Description list[State]

list[State]: Terminal execution states.

Notes

Responsibilities:

- Merges override payload with initial payload.\n- Creates root `State` and executes engine.\n
"},{"location":"dagpipe/#dagpipe.ProgressMessage","title":"ProgressMessage","text":"
ProgressMessage(\n    *,\n    lines: int | None = None,\n    blocks: int | None = None,\n    count: int | None = None,\n    unit: str | None = None,\n    raw_ocr_line: str | None = None,\n    error: str | None = None,\n    step: str = \"\",\n    status: str = \"\"\n)\n

Lightweight progress payload emitted by engine step hooks.

Mirrors the imperative ProgressMessage used by the legacy orchestrator so callers can surface counts/lines/errors without coupling the engine to pydantic.

"},{"location":"dagpipe/#dagpipe.Schema","title":"Schema dataclass","text":"
Schema(tree: Mapping[str, SchemaNode])\n

Immutable hierarchical schema defining allowed payload structure.

Attributes:

Name Type Description tree Mapping[str, SchemaNode]

Hierarchical schema definition.

Notes

Responsibilities:

- Validates `State` payloads and updates.\n- Reusable across all `State` instances.\n- Fully thread-safe due to immutability.\n
"},{"location":"dagpipe/#dagpipe.Schema-functions","title":"Functions","text":""},{"location":"dagpipe/#dagpipe.Schema.validate_payload","title":"validate_payload","text":"
validate_payload(payload: Payload) -> None\n

Validate complete payload structure.

Parameters:

Name Type Description Default payload Payload

Payload to validate.

required

Raises:

Type Description SchemaError

If payload violates schema.

"},{"location":"dagpipe/#dagpipe.Schema.validate_update","title":"validate_update","text":"
validate_update(updates: Mapping[str, Any]) -> None\n

Validate payload update paths.

Parameters:

Name Type Description Default updates Mapping[str, Any]

Dot-path updates to validate.

required

Raises:

Type Description SchemaError

If any path is invalid according to the schema.

"},{"location":"dagpipe/#dagpipe.SchemaError","title":"SchemaError","text":"

Bases: Exception

Raised when payload data violates the declared schema.

"},{"location":"dagpipe/#dagpipe.SchemaError--indicates-invalid-structure-invalid-path-or-invalid-type","title":"Indicates invalid structure, invalid path, or invalid type.","text":""},{"location":"dagpipe/#dagpipe.State","title":"State dataclass","text":"
State(\n    payload: Payload,\n    confidence: float = 1.0,\n    parent: State | None = None,\n    depth: int = 0,\n    history: tuple[str, ...] = tuple(),\n    metadata: dict[str, Any] = dict(),\n)\n

Immutable execution state propagated through dagpipe pipeline.

Attributes:

Name Type Description payload Payload

Execution data container.

schema ClassVar[Schema]

Payload validation schema.

confidence float

Execution confidence score.

parent Optional[State]

Parent state reference.

depth int

Execution depth.

history Tuple[str, ...]

Ordered node execution lineage.

metadata Dict[str, Any]

Execution metadata.

Notes

Responsibilities:

- Represents a complete execution snapshot at a specific point in\n  pipeline traversal.\n- Fundamental unit of execution in `dagpipe`.\n- Fully thread-safe due to immutability.\n
"},{"location":"dagpipe/#dagpipe.State-functions","title":"Functions","text":""},{"location":"dagpipe/#dagpipe.State.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Concise debug representation.

Avoids printing full data for large states.

"},{"location":"dagpipe/#dagpipe.State.fork","title":"fork","text":"
fork(\n    *,\n    payload_update: Mapping[str, Any] | None = None,\n    confidence_delta: float = 0.0,\n    node_id: str | None = None,\n    metadata_update: Mapping[str, Any] | None = None\n) -> State\n

Create a new child State derived from this state.

Parameters:

Name Type Description Default payload_update Mapping[str, Any] | None

Dot-path updates applied to the payload.

None confidence_delta float

Adjustment applied to current confidence.

0.0 node_id str | None

Identifier of the node creating this state.

None metadata_update Mapping[str, Any] | None

Updates merged into state metadata.

None

Returns:

Name Type Description State State

A new immutable State instance.

Notes

Guarantees:

- This is the only supported mechanism for modifying execution data.\n- Validates payload updates, preserves lineage, increments depth,\n  and appends to history.\n
"},{"location":"dagpipe/#dagpipe.State.get","title":"get","text":"
get(key: str, default: Any = None) -> Any\n

Retrieve payload value.

Parameters:

Name Type Description Default key str

Dot-path key.

required default Any

Fallback value.

None

Returns:

Name Type Description Any Any

Stored value or default.

"},{"location":"dagpipe/#dagpipe.State.has","title":"has","text":"
has(key: str) -> bool\n

Check whether payload contains key.

Parameters:

Name Type Description Default key str

Dot-path key.

required

Returns:

Name Type Description bool bool

Existence of the key.

"},{"location":"dagpipe/#dagpipe.State.lineage","title":"lineage","text":"
lineage() -> tuple[State, ...]\n

Return lineage from root to this State.

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Ordered execution lineage (root first).

"},{"location":"dagpipe/#dagpipe.StepResult","title":"StepResult","text":"
StepResult(\n    index: int,\n    node_id: str,\n    states: tuple[State, ...],\n    completed: bool,\n)\n

A single checkpointed step within an async/resumable engine run.

Attributes:

Name Type Description index int

Ordinal index of the step.

node_id str

Identifier of the node associated with this step.

states tuple[State, ...]

States produced by running this step.

completed bool

Whether this step succeeded (vs. paused/interrupted).

Initialise StepResult.

Parameters:

Name Type Description Default index int

Ordinal index of the step.

required node_id str

Identifier of the node associated with this step.

required states tuple[State, ...]

States produced by running this step.

required completed bool

Whether this step succeeded (vs. paused/interrupted).

required"},{"location":"dagpipe/#dagpipe.StepResult-functions","title":"Functions","text":""},{"location":"dagpipe/#dagpipe-functions","title":"Functions","text":""},{"location":"dagpipe/#dagpipe.load_pipeline","title":"load_pipeline","text":"
load_pipeline(path: str) -> Pipeline\n

Load pipeline from YAML file.

Parameters:

Name Type Description Default path str

Path to YAML configuration file.

required

Returns:

Name Type Description Pipeline Pipeline

Executable pipeline instance.

Notes

Responsibilities:

- Loads YAML configuration and builds schema.\n- Creates `State` subclass and loads `Node` instances.\n- Builds `Graph` topology and initializes `Engine`.\n
"},{"location":"dagpipe/engine/","title":"Engine","text":""},{"location":"dagpipe/engine/#dagpipe.engine","title":"dagpipe.engine","text":""},{"location":"dagpipe/engine/#dagpipe.engine--summary","title":"Summary","text":"

Execution engine responsible for running pipelines and graphs.

The Engine executes Node objects and propagates immutable State instances through either a linear sequence or a directed acyclic graph (Graph). It orchestrates execution order, branching, and state propagation.

"},{"location":"dagpipe/engine/#dagpipe.engine--guarantees","title":"Guarantees","text":""},{"location":"dagpipe/engine/#dagpipe.engine-classes","title":"Classes","text":""},{"location":"dagpipe/engine/#dagpipe.engine.Engine","title":"Engine","text":"
Engine(\n    nodes_or_graph: Sequence[Node] | Graph,\n    *,\n    on_step: StepHook | None = None\n)\n

Execution engine responsible for running pipeline logic.

Notes

Responsibilities:

- Accepts either a linear sequence of `Node` objects or a `Graph`\n  defining execution topology.\n- Propagates immutable `State` objects through `Node` objects and\n  collects terminal states.\n- Supports synchronous (`run`) and asynchronous (`run_async`)\n  execution, dispatching per-node.\n- Supports step-wise / resumable execution and progress hooks.\n

Guarantees:

- Never mutates `State`, `Node`, or `Graph` instances.\n- `State` objects are never modified in place; each branch produces\n  independent instances.\n- Execution order is deterministic and follows graph or pipeline topology.\n- Thread-safe for concurrent execution.\n
"},{"location":"dagpipe/engine/#dagpipe.engine.Engine-attributes","title":"Attributes","text":""},{"location":"dagpipe/engine/#dagpipe.engine.Engine.nodes","title":"nodes property","text":"
nodes: tuple[Node, ...]\n

Return nodes managed by this engine.

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Ordered sequence in linear mode or all nodes in graph mode.

"},{"location":"dagpipe/engine/#dagpipe.engine.Engine-functions","title":"Functions","text":""},{"location":"dagpipe/engine/#dagpipe.engine.Engine.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return the canonical string representation of the object.

Returns:

Name Type Description str str

Representation that uniquely identifies the object and its configuration.

"},{"location":"dagpipe/engine/#dagpipe.engine.Engine.run","title":"run","text":"
run(root: State) -> list[State]\n

Execute the pipeline starting from a root State.

Parameters:

Name Type Description Default root State

Initial execution state.

required

Returns:

Type Description list[State]

list[State]: Terminal execution states produced by the pipeline.

Raises:

Type Description TypeError

If root is not a State instance.

RuntimeError

If the engine execution mode is invalid.

Notes

Responsibilities:

- Selects execution mode, propagates state through nodes, creates\n  new instances for branches, and collects terminal states.\n
"},{"location":"dagpipe/engine/#dagpipe.engine.Engine.run_async","title":"run_async async","text":"
run_async(root: State) -> list[State]\n

Execute the pipeline starting from root, dispatching sync vs async nodes.

Parameters:

Name Type Description Default root State

Initial execution state.

required

Returns:

Type Description list[State]

list[State]: Terminal execution states produced by the pipeline.

Notes

Each node is executed with Node.run when synchronous and AsyncNode.run_async when asynchronous. Linear and graph topologies are both supported.

"},{"location":"dagpipe/engine/#dagpipe.engine.Engine.run_steps","title":"run_steps","text":"
run_steps(\n    root: State,\n    *,\n    resume_from: int | None = None,\n    on_step: StepHook | None = None\n) -> Iterator[StepResult]\n

Execute the pipeline step-by-step, yielding one StepResult per step.

Parameters:

Name Type Description Default root State

Initial execution state.

required resume_from int | None

Skip steps at index < resume_from (for resume-after-partial). Steps are 0-indexed.

None on_step StepHook | None

Callback (step, status, message) invoked per step; falls back to the engine-level hook when unset.

None

Yields:

Name Type Description StepResult StepResult

One per executed node/step, carrying the produced states.

Notes

This is a synchronous, generator-based checkpoint interface compatible with the imperative resume-by-step behaviour of the legacy orchestrator. Use run_steps_async for async nodes.

"},{"location":"dagpipe/engine/#dagpipe.engine.Engine.run_steps_async","title":"run_steps_async async","text":"
run_steps_async(\n    root: State,\n    *,\n    resume_from: int | None = None,\n    on_step: AsyncStepHook | None = None\n) -> AsyncIterator[StepResult]\n

Async variant of run_steps supporting AsyncNode execution.

Parameters:

Name Type Description Default root State

Initial execution state.

required resume_from int | None

Skip steps at index < resume_from.

None on_step AsyncStepHook | None

Async callback (step, status, message) invoked per step.

None

Yields:

Name Type Description StepResult AsyncIterator[StepResult]

One per executed node/step.

"},{"location":"dagpipe/engine/#dagpipe.engine.ProgressMessage","title":"ProgressMessage","text":"
ProgressMessage(\n    *,\n    lines: int | None = None,\n    blocks: int | None = None,\n    count: int | None = None,\n    unit: str | None = None,\n    raw_ocr_line: str | None = None,\n    error: str | None = None,\n    step: str = \"\",\n    status: str = \"\"\n)\n

Lightweight progress payload emitted by engine step hooks.

Mirrors the imperative ProgressMessage used by the legacy orchestrator so callers can surface counts/lines/errors without coupling the engine to pydantic.

"},{"location":"dagpipe/engine/#dagpipe.engine.StepResult","title":"StepResult","text":"
StepResult(\n    index: int,\n    node_id: str,\n    states: tuple[State, ...],\n    completed: bool,\n)\n

A single checkpointed step within an async/resumable engine run.

Attributes:

Name Type Description index int

Ordinal index of the step.

node_id str

Identifier of the node associated with this step.

states tuple[State, ...]

States produced by running this step.

completed bool

Whether this step succeeded (vs. paused/interrupted).

Initialise StepResult.

Parameters:

Name Type Description Default index int

Ordinal index of the step.

required node_id str

Identifier of the node associated with this step.

required states tuple[State, ...]

States produced by running this step.

required completed bool

Whether this step succeeded (vs. paused/interrupted).

required"},{"location":"dagpipe/engine/#dagpipe.engine.StepResult-functions","title":"Functions","text":""},{"location":"dagpipe/graph/","title":"Graph","text":""},{"location":"dagpipe/graph/#dagpipe.graph","title":"dagpipe.graph","text":""},{"location":"dagpipe/graph/#dagpipe.graph--summary","title":"Summary","text":"

Defines DAG structure connecting nodes.

A Graph describes execution topology only. It does not execute nodes or manage State. Execution is handled by an Engine.

"},{"location":"dagpipe/graph/#dagpipe.graph--responsibilities","title":"Responsibilities","text":""},{"location":"dagpipe/graph/#dagpipe.graph-classes","title":"Classes","text":""},{"location":"dagpipe/graph/#dagpipe.graph.Graph","title":"Graph","text":"
Graph()\n

Directed Acyclic Graph defining execution topology of Node objects.

Notes

Responsibilities:

- Stores node connectivity and validates that the topology remains acyclic.\n- Structure determines how `State` flows between nodes during execution.\n

Guarantees:

- Topology is acyclic. Node relationships remain consistent.\n- Thread-safe for concurrent reads after construction.\n

Create an empty Graph.

"},{"location":"dagpipe/graph/#dagpipe.graph.Graph--initializes-node-registry-and-edge-mappings","title":"Initializes node registry and edge mappings.","text":""},{"location":"dagpipe/graph/#dagpipe.graph.Graph-functions","title":"Functions","text":""},{"location":"dagpipe/graph/#dagpipe.graph.Graph.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"dagpipe/graph/#dagpipe.graph.Graph.__repr__--returns","title":"Returns","text":"

str

"},{"location":"dagpipe/graph/#dagpipe.graph.Graph.add_edge","title":"add_edge","text":"
add_edge(src: Node, dst: Node) -> None\n

Add a directed edge from src to dst.

Parameters:

Name Type Description Default src Node

Source node.

required dst Node

Destination node.

required

Raises:

Type Description TypeError

If src or dst is not a Node.

ValueError

If the edge would create a cycle or if src and dst are common.

Notes "},{"location":"dagpipe/graph/#dagpipe.graph.Graph.add_root","title":"add_root","text":"
add_root(node: Node) -> None\n

Add a root node with no parents.

Parameters:

Name Type Description Default node Node

Node to add as a root.

required

Raises:

Type Description TypeError

If node is not a Node instance.

"},{"location":"dagpipe/graph/#dagpipe.graph.Graph.children","title":"children","text":"
children(node: Node) -> tuple[Node, ...]\n

Return child nodes of a node.

Parameters:

Name Type Description Default node Node

Node to query.

required

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Outgoing neighbors.

"},{"location":"dagpipe/graph/#dagpipe.graph.Graph.nodes","title":"nodes","text":"
nodes() -> tuple[Node, ...]\n

Return all nodes in the graph.

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: All registered nodes.

"},{"location":"dagpipe/graph/#dagpipe.graph.Graph.parents","title":"parents","text":"
parents(node: Node) -> tuple[Node, ...]\n

Return parent nodes of a node.

Parameters:

Name Type Description Default node Node

Node to query.

required

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Incoming neighbors.

"},{"location":"dagpipe/graph/#dagpipe.graph.Graph.roots","title":"roots","text":"
roots() -> tuple[Node, ...]\n

Return root nodes (nodes with no incoming edges).

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Entry point nodes.

"},{"location":"dagpipe/node/","title":"Node","text":""},{"location":"dagpipe/node/#dagpipe.node","title":"dagpipe.node","text":""},{"location":"dagpipe/node/#dagpipe.node--summary","title":"Summary","text":"

Defines the Node abstraction used by dagpipe.

A node represents a single unit of pipeline execution logic. It consumes one State and produces zero, one, or many new State objects.

Nodes are connected using a Graph and executed by an Engine.

"},{"location":"dagpipe/node/#dagpipe.node--design-principles","title":"Design principles","text":""},{"location":"dagpipe/node/#dagpipe.node-classes","title":"Classes","text":""},{"location":"dagpipe/node/#dagpipe.node.AsyncNode","title":"AsyncNode","text":"

Bases: Node

Base class for nodes whose execution is asynchronous.

Subclasses implement resolve_async (an async generator yielding derived State objects). The engine dispatches to resolve_async when running an async traversal (see Engine.run_async).

Sync-only engines (and the base Node.run) treat an AsyncNode as a no-op consumer: calling run on an AsyncNode returns no states, signalling that an async engine is required.

"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode-functions","title":"Functions","text":""},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.__hash__","title":"__hash__","text":"
__hash__() -> int\n

Return stable hash based on node ID.

"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.__hash__--returns","title":"Returns","text":"

int

"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.__repr__--returns","title":"Returns","text":"

str

"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.__str__","title":"__str__","text":"
__str__() -> str\n

Return display representation.

"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.__str__--returns","title":"Returns","text":"

str

"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.clean_id_and_name","title":"clean_id_and_name classmethod","text":"
clean_id_and_name() -> None\n

Normalize and validate node ID and display name.

Raises:

Type Description TypeError

If ID is not a string.

ValueError

If ID format is invalid.

Notes

Guarantees:

- Generates ID from module and class name if missing.\n- Validates ID format.\n- Generates human-readable name if missing.\n
"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.fork","title":"fork","text":"
fork(\n    state: State,\n    *,\n    payload_update: Any = None,\n    confidence_delta: float = 0.0,\n    metadata_update: Any = None\n) -> State\n

Create a child State attributed to this node.

Parameters:

Name Type Description Default state State

Parent execution state.

required payload_update Any

Dot-path payload updates.

None confidence_delta float

Confidence adjustment.

0.0 metadata_update Any

Metadata updates.

None

Returns:

Name Type Description State State

New child execution state.

Notes

Responsibilities:

- Convenience wrapper around `State.fork()` that automatically\n  records this node's ID in state history.\n
"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.is_async","title":"is_async","text":"
is_async() -> bool\n

Return whether this node executes asynchronously.

"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.node_id_to_name","title":"node_id_to_name staticmethod","text":"
node_id_to_name(node_id: str) -> str\n

Convert a dotted snake_case node ID into a human-readable name.

Parameters:

Name Type Description Default node_id str

Unique node identifier (e.g., 'entity.resolve.numeric_merchant').

required

Returns:

Name Type Description str str

Human-readable display name (e.g., 'Entity \u203a Resolve \u203a Numeric Merchant').

"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.resolve_async","title":"resolve_async async","text":"
resolve_async(state: State) -> Iterable[State]\n

Execute node logic asynchronously.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description Iterable[State]

Iterable[State]: Derived execution state(s).

Notes

Subclasses implement this. Must not mutate the input state. Should use fork() to create child states.

"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.run","title":"run","text":"
run(state: State) -> tuple[State, ...]\n

Execute this node on a State.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Derived execution states.

Raises:

Type Description TypeError

If resolve() yields a non-State object.

"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.run_async","title":"run_async async","text":"
run_async(state: State) -> tuple[State, ...]\n

Execute this node asynchronously on a state, validating outputs.

"},{"location":"dagpipe/node/#dagpipe.node.Node","title":"Node","text":"

Bases: ABC

Base class for all dagpipe execution nodes.

Attributes:

Name Type Description id str

Unique identifier of the node (snake_case dotted format).

name str

Human-readable display name.

Notes

Responsibilities:

- Represents a deterministic unit of execution in the pipeline graph.\n- Consumes one `State` and produces zero, one, or many derived states.\n- Defines execution logic and enables branching, filtering, and transformation.\n

Guarantees:

- Nodes must never mutate the input `State`.\n- Instances are singletons per subclass and reused across executions.\n
"},{"location":"dagpipe/node/#dagpipe.node.Node-functions","title":"Functions","text":""},{"location":"dagpipe/node/#dagpipe.node.Node.__hash__","title":"__hash__","text":"
__hash__() -> int\n

Return stable hash based on node ID.

"},{"location":"dagpipe/node/#dagpipe.node.Node.__hash__--returns","title":"Returns","text":"

int

"},{"location":"dagpipe/node/#dagpipe.node.Node.__new__","title":"__new__","text":"
__new__(*args: Any, **kwargs: Any) -> Node\n

Create or reuse a Node instance.

Stateless subclasses (no parameterized __init__) share one singleton instance per class \u2014 matching the original dagpipe behaviour underpinning set_registry-style configuration. Subclasses that declare an __init__ requiring instance-state arguments get a fresh instance per construction so pipeline builders can inject per-run dependencies.

"},{"location":"dagpipe/node/#dagpipe.node.Node.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"dagpipe/node/#dagpipe.node.Node.__repr__--returns","title":"Returns","text":"

str

"},{"location":"dagpipe/node/#dagpipe.node.Node.__str__","title":"__str__","text":"
__str__() -> str\n

Return display representation.

"},{"location":"dagpipe/node/#dagpipe.node.Node.__str__--returns","title":"Returns","text":"

str

"},{"location":"dagpipe/node/#dagpipe.node.Node.clean_id_and_name","title":"clean_id_and_name classmethod","text":"
clean_id_and_name() -> None\n

Normalize and validate node ID and display name.

Raises:

Type Description TypeError

If ID is not a string.

ValueError

If ID format is invalid.

Notes

Guarantees:

- Generates ID from module and class name if missing.\n- Validates ID format.\n- Generates human-readable name if missing.\n
"},{"location":"dagpipe/node/#dagpipe.node.Node.fork","title":"fork","text":"
fork(\n    state: State,\n    *,\n    payload_update: Any = None,\n    confidence_delta: float = 0.0,\n    metadata_update: Any = None\n) -> State\n

Create a child State attributed to this node.

Parameters:

Name Type Description Default state State

Parent execution state.

required payload_update Any

Dot-path payload updates.

None confidence_delta float

Confidence adjustment.

0.0 metadata_update Any

Metadata updates.

None

Returns:

Name Type Description State State

New child execution state.

Notes

Responsibilities:

- Convenience wrapper around `State.fork()` that automatically\n  records this node's ID in state history.\n
"},{"location":"dagpipe/node/#dagpipe.node.Node.is_async","title":"is_async","text":"
is_async() -> bool\n

Return whether this node executes asynchronously.

"},{"location":"dagpipe/node/#dagpipe.node.Node.node_id_to_name","title":"node_id_to_name staticmethod","text":"
node_id_to_name(node_id: str) -> str\n

Convert a dotted snake_case node ID into a human-readable name.

Parameters:

Name Type Description Default node_id str

Unique node identifier (e.g., 'entity.resolve.numeric_merchant').

required

Returns:

Name Type Description str str

Human-readable display name (e.g., 'Entity \u203a Resolve \u203a Numeric Merchant').

"},{"location":"dagpipe/node/#dagpipe.node.Node.resolve","title":"resolve abstractmethod","text":"
resolve(state: State) -> Iterable[State]\n

Execute node logic.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description Iterable[State]

Iterable[State]: Derived execution state(s).

Notes

Responsibilities:

- Subclasses implement specific resolution behavior.\n- Must not mutate input state.\n- Should use `fork()` to create child states.\n- May yield zero states to terminate a branch.\n
"},{"location":"dagpipe/node/#dagpipe.node.Node.run","title":"run","text":"
run(state: State) -> tuple[State, ...]\n

Execute this node on a State.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Derived execution states.

Raises:

Type Description TypeError

If resolve() yields a non-State object.

"},{"location":"dagpipe/state/","title":"State","text":""},{"location":"dagpipe/state/#dagpipe.state","title":"dagpipe.state","text":""},{"location":"dagpipe/state/#dagpipe.state--summary","title":"Summary","text":"

Defines the core State object used by dagpipe.

The State represents a single point in pipeline execution. It contains arbitrary data and metadata and is designed to be immutable. Instead of modifying an existing state, nodes create new child states via fork().

"},{"location":"dagpipe/state/#dagpipe.state--design-principles","title":"Design principles","text":""},{"location":"dagpipe/state/#dagpipe.state-classes","title":"Classes","text":""},{"location":"dagpipe/state/#dagpipe.state.Payload","title":"Payload dataclass","text":"
Payload(_data: Mapping[str, Any])\n

Immutable hierarchical container with dot-path access.

Attributes:

Name Type Description _data Mapping[str, Any]

Immutable hierarchical data structure.

Notes

Responsibilities:

- Stores execution data used by `State`.\n- Supports efficient atomic updates without modifying existing instances.\n- `Payload` instances are fully thread-safe due to immutability.\n
"},{"location":"dagpipe/state/#dagpipe.state.Payload-functions","title":"Functions","text":""},{"location":"dagpipe/state/#dagpipe.state.Payload.as_dict","title":"as_dict","text":"
as_dict() -> Mapping[str, Any]\n

Return underlying mapping.

Returns:

Type Description Mapping[str, Any]

Mapping[str, Any]: Read-only view of the underlying data.

"},{"location":"dagpipe/state/#dagpipe.state.Payload.get","title":"get","text":"
get(path: str, default: Any = None) -> Any\n

Retrieve value using dot-path.

Parameters:

Name Type Description Default path str

Dot-separated path to the value.

required default Any

Default value if path doesn't exist.

None

Returns:

Name Type Description Any Any

The retrieved value or default.

"},{"location":"dagpipe/state/#dagpipe.state.Payload.has","title":"has","text":"
has(path: str) -> bool\n

Return True if path exists.

Parameters:

Name Type Description Default path str

Dot-separated path to check.

required

Returns:

Name Type Description bool bool

Existence of the path.

"},{"location":"dagpipe/state/#dagpipe.state.Payload.iter_paths","title":"iter_paths classmethod","text":"
iter_paths(\n    data: Mapping[str, Any], prefix: str = \"\"\n) -> Iterable[str]\n

Recursively yield dot-paths for all leaf nodes.

Parameters:

Name Type Description Default data Mapping[str, Any]

The mapping to iterate over.

required prefix str

Current path prefix.

''

Yields:

Name Type Description str Iterable[str]

Dot-path for each leaf node.

"},{"location":"dagpipe/state/#dagpipe.state.Payload.keys","title":"keys","text":"
keys() -> Iterable[str]\n

Return top-level keys.

Returns:

Type Description Iterable[str]

Iterable[str]: Iterator over top-level keys.

"},{"location":"dagpipe/state/#dagpipe.state.Payload.update","title":"update","text":"
update(updates: Mapping[str, Any]) -> Payload\n

Create a new Payload with dot-path updates applied.

Parameters:

Name Type Description Default updates Mapping[str, Any]

Dot-path to value mapping.

required

Returns:

Name Type Description Payload Payload

New immutable payload instance with updates.

Notes

Guarantees:

- Preserves existing data by copying only modified branches.\n- Returns a new immutable `Payload`.\n
"},{"location":"dagpipe/state/#dagpipe.state.Schema","title":"Schema dataclass","text":"
Schema(tree: Mapping[str, SchemaNode])\n

Immutable hierarchical schema defining allowed payload structure.

Attributes:

Name Type Description tree Mapping[str, SchemaNode]

Hierarchical schema definition.

Notes

Responsibilities:

- Validates `State` payloads and updates.\n- Reusable across all `State` instances.\n- Fully thread-safe due to immutability.\n
"},{"location":"dagpipe/state/#dagpipe.state.Schema-functions","title":"Functions","text":""},{"location":"dagpipe/state/#dagpipe.state.Schema.validate_payload","title":"validate_payload","text":"
validate_payload(payload: Payload) -> None\n

Validate complete payload structure.

Parameters:

Name Type Description Default payload Payload

Payload to validate.

required

Raises:

Type Description SchemaError

If payload violates schema.

"},{"location":"dagpipe/state/#dagpipe.state.Schema.validate_update","title":"validate_update","text":"
validate_update(updates: Mapping[str, Any]) -> None\n

Validate payload update paths.

Parameters:

Name Type Description Default updates Mapping[str, Any]

Dot-path updates to validate.

required

Raises:

Type Description SchemaError

If any path is invalid according to the schema.

"},{"location":"dagpipe/state/#dagpipe.state.SchemaError","title":"SchemaError","text":"

Bases: Exception

Raised when payload data violates the declared schema.

"},{"location":"dagpipe/state/#dagpipe.state.SchemaError--indicates-invalid-structure-invalid-path-or-invalid-type","title":"Indicates invalid structure, invalid path, or invalid type.","text":""},{"location":"dagpipe/state/#dagpipe.state.State","title":"State dataclass","text":"
State(\n    payload: Payload,\n    confidence: float = 1.0,\n    parent: State | None = None,\n    depth: int = 0,\n    history: tuple[str, ...] = tuple(),\n    metadata: dict[str, Any] = dict(),\n)\n

Immutable execution state propagated through dagpipe pipeline.

Attributes:

Name Type Description payload Payload

Execution data container.

schema ClassVar[Schema]

Payload validation schema.

confidence float

Execution confidence score.

parent Optional[State]

Parent state reference.

depth int

Execution depth.

history Tuple[str, ...]

Ordered node execution lineage.

metadata Dict[str, Any]

Execution metadata.

Notes

Responsibilities:

- Represents a complete execution snapshot at a specific point in\n  pipeline traversal.\n- Fundamental unit of execution in `dagpipe`.\n- Fully thread-safe due to immutability.\n
"},{"location":"dagpipe/state/#dagpipe.state.State-functions","title":"Functions","text":""},{"location":"dagpipe/state/#dagpipe.state.State.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Concise debug representation.

Avoids printing full data for large states.

"},{"location":"dagpipe/state/#dagpipe.state.State.fork","title":"fork","text":"
fork(\n    *,\n    payload_update: Mapping[str, Any] | None = None,\n    confidence_delta: float = 0.0,\n    node_id: str | None = None,\n    metadata_update: Mapping[str, Any] | None = None\n) -> State\n

Create a new child State derived from this state.

Parameters:

Name Type Description Default payload_update Mapping[str, Any] | None

Dot-path updates applied to the payload.

None confidence_delta float

Adjustment applied to current confidence.

0.0 node_id str | None

Identifier of the node creating this state.

None metadata_update Mapping[str, Any] | None

Updates merged into state metadata.

None

Returns:

Name Type Description State State

A new immutable State instance.

Notes

Guarantees:

- This is the only supported mechanism for modifying execution data.\n- Validates payload updates, preserves lineage, increments depth,\n  and appends to history.\n
"},{"location":"dagpipe/state/#dagpipe.state.State.get","title":"get","text":"
get(key: str, default: Any = None) -> Any\n

Retrieve payload value.

Parameters:

Name Type Description Default key str

Dot-path key.

required default Any

Fallback value.

None

Returns:

Name Type Description Any Any

Stored value or default.

"},{"location":"dagpipe/state/#dagpipe.state.State.has","title":"has","text":"
has(key: str) -> bool\n

Check whether payload contains key.

Parameters:

Name Type Description Default key str

Dot-path key.

required

Returns:

Name Type Description bool bool

Existence of the key.

"},{"location":"dagpipe/state/#dagpipe.state.State.lineage","title":"lineage","text":"
lineage() -> tuple[State, ...]\n

Return lineage from root to this State.

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Ordered execution lineage (root first).

"},{"location":"dagpipe/yaml_loader/","title":"Yaml Loader","text":""},{"location":"dagpipe/yaml_loader/#dagpipe.yaml_loader","title":"dagpipe.yaml_loader","text":""},{"location":"dagpipe/yaml_loader/#dagpipe.yaml_loader--summary","title":"Summary","text":"

Loads dagpipe pipelines from YAML configuration.

Creates fully configured pipeline objects from declarative YAML definitions, including Schema, State subclasses, Node instances, Graph topology, and initial payloads.

"},{"location":"dagpipe/yaml_loader/#dagpipe.yaml_loader-classes","title":"Classes","text":""},{"location":"dagpipe/yaml_loader/#dagpipe.yaml_loader.Pipeline","title":"Pipeline dataclass","text":"
Pipeline(\n    engine: Engine,\n    state_cls: type[State],\n    initial_payload: Payload,\n)\n

Executable pipeline created from YAML configuration.

Attributes:

Name Type Description engine Engine

Execution engine responsible for running the pipeline.

state_cls Type[State]

Dynamically created State subclass with configured schema.

initial_payload Payload

Default payload used when execution begins.

Notes

Responsibilities:

- Encapsulates engine, state type, and initial payload.\n- Provides a simplified interface for executing configured pipelines.\n- Safe for concurrent execution if underlying nodes are thread-safe.\n
"},{"location":"dagpipe/yaml_loader/#dagpipe.yaml_loader.Pipeline-functions","title":"Functions","text":""},{"location":"dagpipe/yaml_loader/#dagpipe.yaml_loader.Pipeline.run","title":"run","text":"
run(\n    payload_override: Mapping[str, Any] | None = None,\n) -> list[State]\n

Execute the pipeline.

Parameters:

Name Type Description Default payload_override Mapping[str, Any] | None

Payload values overriding initial payload.

None

Returns:

Type Description list[State]

list[State]: Terminal execution states.

Notes

Responsibilities:

- Merges override payload with initial payload.\n- Creates root `State` and executes engine.\n
"},{"location":"dagpipe/yaml_loader/#dagpipe.yaml_loader-functions","title":"Functions","text":""},{"location":"dagpipe/yaml_loader/#dagpipe.yaml_loader.load_pipeline","title":"load_pipeline","text":"
load_pipeline(path: str) -> Pipeline\n

Load pipeline from YAML file.

Parameters:

Name Type Description Default path str

Path to YAML configuration file.

required

Returns:

Name Type Description Pipeline Pipeline

Executable pipeline instance.

Notes

Responsibilities:

- Loads YAML configuration and builds schema.\n- Creates `State` subclass and loads `Node` instances.\n- Builds `Graph` topology and initializes `Engine`.\n
"}]} \ No newline at end of file +{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"dagpipe","text":""},{"location":"#dagpipe","title":"dagpipe","text":""},{"location":"#dagpipe--summary","title":"Summary","text":"

Directed acyclic graph execution framework for deterministic state propagation.

dagpipe executes pipelines composed of nodes connected in a directed acyclic graph (DAG). Each node receives an immutable State and optionally produces derived states for downstream nodes.

"},{"location":"#dagpipe--installation","title":"Installation","text":"

Install using pip:

pip install dagpipe\n
"},{"location":"#dagpipe--quick-start","title":"Quick Start","text":"
from dagpipe import State, Payload, Schema, Graph, Engine\nfrom dagpipe.node import Node\n\nclass HelloNode(Node):\n    id = \"hello\"\n    def resolve(self, state):\n        yield self.fork(state, payload_update={\"msg\": \"hello\"})\n\n# Build and run\ngraph = Graph()\ngraph.add_root(HelloNode())\nengine = Engine(graph)\n\nclass MyState(State):\n    schema = Schema({})\n\nresults = engine.run(MyState(payload=Payload({})))\n
"},{"location":"#dagpipe--public-api","title":"Public API","text":"

This package re-exports the core pipeline components. Consumers should import from this namespace for standard usage.

"},{"location":"#dagpipe--execution-core","title":"Execution Core","text":""},{"location":"#dagpipe--state-data","title":"State & Data","text":""},{"location":"#dagpipe--declarative-pipelines","title":"Declarative Pipelines","text":""},{"location":"#dagpipe-classes","title":"Classes","text":""},{"location":"#dagpipe.AsyncNode","title":"AsyncNode","text":"

Bases: Node

Base class for nodes whose execution is asynchronous.

Subclasses implement resolve_async (an async generator yielding derived State objects). The engine dispatches to resolve_async when running an async traversal (see Engine.run_async).

Sync-only engines (and the base Node.run) treat an AsyncNode as a no-op consumer: calling run on an AsyncNode returns no states, signalling that an async engine is required.

"},{"location":"#dagpipe.AsyncNode-functions","title":"Functions","text":""},{"location":"#dagpipe.AsyncNode.__hash__","title":"__hash__","text":"
__hash__() -> int\n

Return stable hash based on node ID.

"},{"location":"#dagpipe.AsyncNode.__hash__--returns","title":"Returns","text":"

int

"},{"location":"#dagpipe.AsyncNode.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"#dagpipe.AsyncNode.__repr__--returns","title":"Returns","text":"

str

"},{"location":"#dagpipe.AsyncNode.__str__","title":"__str__","text":"
__str__() -> str\n

Return display representation.

"},{"location":"#dagpipe.AsyncNode.__str__--returns","title":"Returns","text":"

str

"},{"location":"#dagpipe.AsyncNode.clean_id_and_name","title":"clean_id_and_name classmethod","text":"
clean_id_and_name() -> None\n

Normalize and validate node ID and display name.

Raises:

Type Description TypeError

If ID is not a string.

ValueError

If ID format is invalid.

Notes

Guarantees:

- Generates ID from module and class name if missing.\n- Validates ID format.\n- Generates human-readable name if missing.\n
"},{"location":"#dagpipe.AsyncNode.fork","title":"fork","text":"
fork(\n    state: State,\n    *,\n    payload_update: Any = None,\n    confidence_delta: float = 0.0,\n    metadata_update: Any = None\n) -> State\n

Create a child State attributed to this node.

Parameters:

Name Type Description Default state State

Parent execution state.

required payload_update Any

Dot-path payload updates.

None confidence_delta float

Confidence adjustment.

0.0 metadata_update Any

Metadata updates.

None

Returns:

Name Type Description State State

New child execution state.

Notes

Responsibilities:

- Convenience wrapper around `State.fork()` that automatically\n  records this node's ID in state history.\n
"},{"location":"#dagpipe.AsyncNode.is_async","title":"is_async","text":"
is_async() -> bool\n

Return whether this node executes asynchronously.

"},{"location":"#dagpipe.AsyncNode.node_id_to_name","title":"node_id_to_name staticmethod","text":"
node_id_to_name(node_id: str) -> str\n

Convert a dotted snake_case node ID into a human-readable name.

Parameters:

Name Type Description Default node_id str

Unique node identifier (e.g., 'entity.resolve.numeric_merchant').

required

Returns:

Name Type Description str str

Human-readable display name (e.g., 'Entity \u203a Resolve \u203a Numeric Merchant').

"},{"location":"#dagpipe.AsyncNode.resolve_async","title":"resolve_async async","text":"
resolve_async(state: State) -> Iterable[State]\n

Execute node logic asynchronously.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description Iterable[State]

Iterable[State]: Derived execution state(s).

Notes

Subclasses implement this. Must not mutate the input state. Should use fork() to create child states.

"},{"location":"#dagpipe.AsyncNode.run","title":"run","text":"
run(state: State) -> tuple[State, ...]\n

Execute this node on a State.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Derived execution states.

Raises:

Type Description TypeError

If resolve() yields a non-State object.

"},{"location":"#dagpipe.AsyncNode.run_async","title":"run_async async","text":"
run_async(state: State) -> tuple[State, ...]\n

Execute this node asynchronously on a state, validating outputs.

"},{"location":"#dagpipe.Engine","title":"Engine","text":"
Engine(\n    nodes_or_graph: Sequence[Node] | Graph,\n    *,\n    on_step: StepHook | None = None\n)\n

Execution engine responsible for running pipeline logic.

Notes

Responsibilities:

- Accepts either a linear sequence of `Node` objects or a `Graph`\n  defining execution topology.\n- Propagates immutable `State` objects through `Node` objects and\n  collects terminal states.\n- Supports synchronous (`run`) and asynchronous (`run_async`)\n  execution, dispatching per-node.\n- Supports step-wise / resumable execution and progress hooks.\n

Guarantees:

- Never mutates `State`, `Node`, or `Graph` instances.\n- `State` objects are never modified in place; each branch produces\n  independent instances.\n- Execution order is deterministic and follows graph or pipeline topology.\n- Thread-safe for concurrent execution.\n
"},{"location":"#dagpipe.Engine-attributes","title":"Attributes","text":""},{"location":"#dagpipe.Engine.nodes","title":"nodes property","text":"
nodes: tuple[Node, ...]\n

Return nodes managed by this engine.

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Ordered sequence in linear mode or all nodes in graph mode.

"},{"location":"#dagpipe.Engine-functions","title":"Functions","text":""},{"location":"#dagpipe.Engine.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return the canonical string representation of the object.

Returns:

Name Type Description str str

Representation that uniquely identifies the object and its configuration.

"},{"location":"#dagpipe.Engine.run","title":"run","text":"
run(root: State) -> list[State]\n

Execute the pipeline starting from a root State.

Parameters:

Name Type Description Default root State

Initial execution state.

required

Returns:

Type Description list[State]

list[State]: Terminal execution states produced by the pipeline.

Raises:

Type Description TypeError

If root is not a State instance.

RuntimeError

If the engine execution mode is invalid.

Notes

Responsibilities:

- Selects execution mode, propagates state through nodes, creates\n  new instances for branches, and collects terminal states.\n
"},{"location":"#dagpipe.Engine.run_async","title":"run_async async","text":"
run_async(root: State) -> list[State]\n

Execute the pipeline starting from root, dispatching sync vs async nodes.

Parameters:

Name Type Description Default root State

Initial execution state.

required

Returns:

Type Description list[State]

list[State]: Terminal execution states produced by the pipeline.

Notes

Each node is executed with Node.run when synchronous and AsyncNode.run_async when asynchronous. Linear and graph topologies are both supported.

"},{"location":"#dagpipe.Engine.run_steps","title":"run_steps","text":"
run_steps(\n    root: State,\n    *,\n    resume_from: int | None = None,\n    on_step: StepHook | None = None\n) -> Iterator[StepResult]\n

Execute the pipeline step-by-step, yielding one StepResult per step.

Parameters:

Name Type Description Default root State

Initial execution state.

required resume_from int | None

Skip steps at index < resume_from (for resume-after-partial). Steps are 0-indexed.

None on_step StepHook | None

Callback (step, status, message) invoked per step; falls back to the engine-level hook when unset.

None

Yields:

Name Type Description StepResult StepResult

One per executed node/step, carrying the produced states.

Notes

This is a synchronous, generator-based checkpoint interface compatible with the imperative resume-by-step behaviour of the legacy orchestrator. Use run_steps_async for async nodes.

"},{"location":"#dagpipe.Engine.run_steps_async","title":"run_steps_async async","text":"
run_steps_async(\n    root: State,\n    *,\n    resume_from: int | None = None,\n    on_step: AsyncStepHook | None = None\n) -> AsyncIterator[StepResult]\n

Async variant of run_steps supporting AsyncNode execution.

Parameters:

Name Type Description Default root State

Initial execution state.

required resume_from int | None

Skip steps at index < resume_from.

None on_step AsyncStepHook | None

Async callback (step, status, message) invoked per step.

None

Yields:

Name Type Description StepResult AsyncIterator[StepResult]

One per executed node/step.

"},{"location":"#dagpipe.Graph","title":"Graph","text":"
Graph()\n

Directed Acyclic Graph defining execution topology of Node objects.

Notes

Responsibilities:

- Stores node connectivity and validates that the topology remains acyclic.\n- Structure determines how `State` flows between nodes during execution.\n

Guarantees:

- Topology is acyclic. Node relationships remain consistent.\n- Thread-safe for concurrent reads after construction.\n

Create an empty Graph.

"},{"location":"#dagpipe.Graph--initializes-node-registry-and-edge-mappings","title":"Initializes node registry and edge mappings.","text":""},{"location":"#dagpipe.Graph-functions","title":"Functions","text":""},{"location":"#dagpipe.Graph.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"#dagpipe.Graph.__repr__--returns","title":"Returns","text":"

str

"},{"location":"#dagpipe.Graph.add_edge","title":"add_edge","text":"
add_edge(src: Node, dst: Node) -> None\n

Add a directed edge from src to dst.

Parameters:

Name Type Description Default src Node

Source node.

required dst Node

Destination node.

required

Raises:

Type Description TypeError

If src or dst is not a Node.

ValueError

If the edge would create a cycle or if src and dst are common.

Notes "},{"location":"#dagpipe.Graph.add_root","title":"add_root","text":"
add_root(node: Node) -> None\n

Add a root node with no parents.

Parameters:

Name Type Description Default node Node

Node to add as a root.

required

Raises:

Type Description TypeError

If node is not a Node instance.

"},{"location":"#dagpipe.Graph.children","title":"children","text":"
children(node: Node) -> tuple[Node, ...]\n

Return child nodes of a node.

Parameters:

Name Type Description Default node Node

Node to query.

required

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Outgoing neighbors.

"},{"location":"#dagpipe.Graph.nodes","title":"nodes","text":"
nodes() -> tuple[Node, ...]\n

Return all nodes in the graph.

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: All registered nodes.

"},{"location":"#dagpipe.Graph.parents","title":"parents","text":"
parents(node: Node) -> tuple[Node, ...]\n

Return parent nodes of a node.

Parameters:

Name Type Description Default node Node

Node to query.

required

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Incoming neighbors.

"},{"location":"#dagpipe.Graph.roots","title":"roots","text":"
roots() -> tuple[Node, ...]\n

Return root nodes (nodes with no incoming edges).

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Entry point nodes.

"},{"location":"#dagpipe.Node","title":"Node","text":"

Bases: ABC

Base class for all dagpipe execution nodes.

Attributes:

Name Type Description id str

Unique identifier of the node (snake_case dotted format).

name str

Human-readable display name.

Notes

Responsibilities:

- Represents a deterministic unit of execution in the pipeline graph.\n- Consumes one `State` and produces zero, one, or many derived states.\n- Defines execution logic and enables branching, filtering, and transformation.\n

Guarantees:

- Nodes must never mutate the input `State`.\n- Instances are singletons per subclass and reused across executions.\n
"},{"location":"#dagpipe.Node-functions","title":"Functions","text":""},{"location":"#dagpipe.Node.__hash__","title":"__hash__","text":"
__hash__() -> int\n

Return stable hash based on node ID.

"},{"location":"#dagpipe.Node.__hash__--returns","title":"Returns","text":"

int

"},{"location":"#dagpipe.Node.__new__","title":"__new__","text":"
__new__(*args: Any, **kwargs: Any) -> Node\n

Create or reuse a Node instance.

Stateless subclasses (no parameterized __init__) share one singleton instance per class \u2014 matching the original dagpipe behaviour underpinning set_registry-style configuration. Subclasses that declare an __init__ requiring instance-state arguments get a fresh instance per construction so pipeline builders can inject per-run dependencies.

"},{"location":"#dagpipe.Node.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"#dagpipe.Node.__repr__--returns","title":"Returns","text":"

str

"},{"location":"#dagpipe.Node.__str__","title":"__str__","text":"
__str__() -> str\n

Return display representation.

"},{"location":"#dagpipe.Node.__str__--returns","title":"Returns","text":"

str

"},{"location":"#dagpipe.Node.clean_id_and_name","title":"clean_id_and_name classmethod","text":"
clean_id_and_name() -> None\n

Normalize and validate node ID and display name.

Raises:

Type Description TypeError

If ID is not a string.

ValueError

If ID format is invalid.

Notes

Guarantees:

- Generates ID from module and class name if missing.\n- Validates ID format.\n- Generates human-readable name if missing.\n
"},{"location":"#dagpipe.Node.fork","title":"fork","text":"
fork(\n    state: State,\n    *,\n    payload_update: Any = None,\n    confidence_delta: float = 0.0,\n    metadata_update: Any = None\n) -> State\n

Create a child State attributed to this node.

Parameters:

Name Type Description Default state State

Parent execution state.

required payload_update Any

Dot-path payload updates.

None confidence_delta float

Confidence adjustment.

0.0 metadata_update Any

Metadata updates.

None

Returns:

Name Type Description State State

New child execution state.

Notes

Responsibilities:

- Convenience wrapper around `State.fork()` that automatically\n  records this node's ID in state history.\n
"},{"location":"#dagpipe.Node.is_async","title":"is_async","text":"
is_async() -> bool\n

Return whether this node executes asynchronously.

"},{"location":"#dagpipe.Node.node_id_to_name","title":"node_id_to_name staticmethod","text":"
node_id_to_name(node_id: str) -> str\n

Convert a dotted snake_case node ID into a human-readable name.

Parameters:

Name Type Description Default node_id str

Unique node identifier (e.g., 'entity.resolve.numeric_merchant').

required

Returns:

Name Type Description str str

Human-readable display name (e.g., 'Entity \u203a Resolve \u203a Numeric Merchant').

"},{"location":"#dagpipe.Node.resolve","title":"resolve abstractmethod","text":"
resolve(state: State) -> Iterable[State]\n

Execute node logic.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description Iterable[State]

Iterable[State]: Derived execution state(s).

Notes

Responsibilities:

- Subclasses implement specific resolution behavior.\n- Must not mutate input state.\n- Should use `fork()` to create child states.\n- May yield zero states to terminate a branch.\n
"},{"location":"#dagpipe.Node.run","title":"run","text":"
run(state: State) -> tuple[State, ...]\n

Execute this node on a State.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Derived execution states.

Raises:

Type Description TypeError

If resolve() yields a non-State object.

"},{"location":"#dagpipe.Payload","title":"Payload dataclass","text":"
Payload(_data: Mapping[str, Any])\n

Immutable hierarchical container with dot-path access.

Attributes:

Name Type Description _data Mapping[str, Any]

Immutable hierarchical data structure.

Notes

Responsibilities:

- Stores execution data used by `State`.\n- Supports efficient atomic updates without modifying existing instances.\n- `Payload` instances are fully thread-safe due to immutability.\n
"},{"location":"#dagpipe.Payload-functions","title":"Functions","text":""},{"location":"#dagpipe.Payload.as_dict","title":"as_dict","text":"
as_dict() -> Mapping[str, Any]\n

Return underlying mapping.

Returns:

Type Description Mapping[str, Any]

Mapping[str, Any]: Read-only view of the underlying data.

"},{"location":"#dagpipe.Payload.get","title":"get","text":"
get(path: str, default: Any = None) -> Any\n

Retrieve value using dot-path.

Parameters:

Name Type Description Default path str

Dot-separated path to the value.

required default Any

Default value if path doesn't exist.

None

Returns:

Name Type Description Any Any

The retrieved value or default.

"},{"location":"#dagpipe.Payload.has","title":"has","text":"
has(path: str) -> bool\n

Return True if path exists.

Parameters:

Name Type Description Default path str

Dot-separated path to check.

required

Returns:

Name Type Description bool bool

Existence of the path.

"},{"location":"#dagpipe.Payload.iter_paths","title":"iter_paths classmethod","text":"
iter_paths(\n    data: Mapping[str, Any], prefix: str = \"\"\n) -> Iterable[str]\n

Recursively yield dot-paths for all leaf nodes.

Parameters:

Name Type Description Default data Mapping[str, Any]

The mapping to iterate over.

required prefix str

Current path prefix.

''

Yields:

Name Type Description str Iterable[str]

Dot-path for each leaf node.

"},{"location":"#dagpipe.Payload.keys","title":"keys","text":"
keys() -> Iterable[str]\n

Return top-level keys.

Returns:

Type Description Iterable[str]

Iterable[str]: Iterator over top-level keys.

"},{"location":"#dagpipe.Payload.update","title":"update","text":"
update(updates: Mapping[str, Any]) -> Payload\n

Create a new Payload with dot-path updates applied.

Parameters:

Name Type Description Default updates Mapping[str, Any]

Dot-path to value mapping.

required

Returns:

Name Type Description Payload Payload

New immutable payload instance with updates.

Notes

Guarantees:

- Preserves existing data by copying only modified branches.\n- Returns a new immutable `Payload`.\n
"},{"location":"#dagpipe.Pipeline","title":"Pipeline dataclass","text":"
Pipeline(\n    engine: Engine,\n    state_cls: type[State],\n    initial_payload: Payload,\n)\n

Executable pipeline created from YAML configuration.

Attributes:

Name Type Description engine Engine

Execution engine responsible for running the pipeline.

state_cls Type[State]

Dynamically created State subclass with configured schema.

initial_payload Payload

Default payload used when execution begins.

Notes

Responsibilities:

- Encapsulates engine, state type, and initial payload.\n- Provides a simplified interface for executing configured pipelines.\n- Safe for concurrent execution if underlying nodes are thread-safe.\n
"},{"location":"#dagpipe.Pipeline-functions","title":"Functions","text":""},{"location":"#dagpipe.Pipeline.run","title":"run","text":"
run(\n    payload_override: Mapping[str, Any] | None = None,\n) -> list[State]\n

Execute the pipeline.

Parameters:

Name Type Description Default payload_override Mapping[str, Any] | None

Payload values overriding initial payload.

None

Returns:

Type Description list[State]

list[State]: Terminal execution states.

Notes

Responsibilities:

- Merges override payload with initial payload.\n- Creates root `State` and executes engine.\n
"},{"location":"#dagpipe.ProgressMessage","title":"ProgressMessage","text":"
ProgressMessage(\n    *,\n    lines: int | None = None,\n    blocks: int | None = None,\n    count: int | None = None,\n    unit: str | None = None,\n    raw_ocr_line: str | None = None,\n    error: str | None = None,\n    step: str = \"\",\n    status: str = \"\"\n)\n

Lightweight progress payload emitted by engine step hooks.

Mirrors the imperative ProgressMessage used by the legacy orchestrator so callers can surface counts/lines/errors without coupling the engine to pydantic.

"},{"location":"#dagpipe.Schema","title":"Schema dataclass","text":"
Schema(tree: Mapping[str, SchemaNode])\n

Immutable hierarchical schema defining allowed payload structure.

Attributes:

Name Type Description tree Mapping[str, SchemaNode]

Hierarchical schema definition.

Notes

Responsibilities:

- Validates `State` payloads and updates.\n- Reusable across all `State` instances.\n- Fully thread-safe due to immutability.\n
"},{"location":"#dagpipe.Schema-functions","title":"Functions","text":""},{"location":"#dagpipe.Schema.validate_payload","title":"validate_payload","text":"
validate_payload(payload: Payload) -> None\n

Validate complete payload structure.

Parameters:

Name Type Description Default payload Payload

Payload to validate.

required

Raises:

Type Description SchemaError

If payload violates schema.

"},{"location":"#dagpipe.Schema.validate_update","title":"validate_update","text":"
validate_update(updates: Mapping[str, Any]) -> None\n

Validate payload update paths.

Parameters:

Name Type Description Default updates Mapping[str, Any]

Dot-path updates to validate.

required

Raises:

Type Description SchemaError

If any path is invalid according to the schema.

"},{"location":"#dagpipe.SchemaError","title":"SchemaError","text":"

Bases: Exception

Raised when payload data violates the declared schema.

"},{"location":"#dagpipe.SchemaError--indicates-invalid-structure-invalid-path-or-invalid-type","title":"Indicates invalid structure, invalid path, or invalid type.","text":""},{"location":"#dagpipe.State","title":"State dataclass","text":"
State(\n    payload: Payload,\n    confidence: float = 1.0,\n    parent: State | None = None,\n    depth: int = 0,\n    history: tuple[str, ...] = tuple(),\n    metadata: dict[str, Any] = dict(),\n)\n

Immutable execution state propagated through dagpipe pipeline.

Attributes:

Name Type Description payload Payload

Execution data container.

schema ClassVar[Schema]

Payload validation schema.

confidence float

Execution confidence score.

parent Optional[State]

Parent state reference.

depth int

Execution depth.

history Tuple[str, ...]

Ordered node execution lineage.

metadata Dict[str, Any]

Execution metadata.

Notes

Responsibilities:

- Represents a complete execution snapshot at a specific point in\n  pipeline traversal.\n- Fundamental unit of execution in `dagpipe`.\n- Fully thread-safe due to immutability.\n
"},{"location":"#dagpipe.State-functions","title":"Functions","text":""},{"location":"#dagpipe.State.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Concise debug representation.

Avoids printing full data for large states.

"},{"location":"#dagpipe.State.fork","title":"fork","text":"
fork(\n    *,\n    payload_update: Mapping[str, Any] | None = None,\n    confidence_delta: float = 0.0,\n    node_id: str | None = None,\n    metadata_update: Mapping[str, Any] | None = None\n) -> State\n

Create a new child State derived from this state.

Parameters:

Name Type Description Default payload_update Mapping[str, Any] | None

Dot-path updates applied to the payload.

None confidence_delta float

Adjustment applied to current confidence.

0.0 node_id str | None

Identifier of the node creating this state.

None metadata_update Mapping[str, Any] | None

Updates merged into state metadata.

None

Returns:

Name Type Description State State

A new immutable State instance.

Notes

Guarantees:

- This is the only supported mechanism for modifying execution data.\n- Validates payload updates, preserves lineage, increments depth,\n  and appends to history.\n
"},{"location":"#dagpipe.State.get","title":"get","text":"
get(key: str, default: Any = None) -> Any\n

Retrieve payload value.

Parameters:

Name Type Description Default key str

Dot-path key.

required default Any

Fallback value.

None

Returns:

Name Type Description Any Any

Stored value or default.

"},{"location":"#dagpipe.State.has","title":"has","text":"
has(key: str) -> bool\n

Check whether payload contains key.

Parameters:

Name Type Description Default key str

Dot-path key.

required

Returns:

Name Type Description bool bool

Existence of the key.

"},{"location":"#dagpipe.State.lineage","title":"lineage","text":"
lineage() -> tuple[State, ...]\n

Return lineage from root to this State.

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Ordered execution lineage (root first).

"},{"location":"#dagpipe.StepResult","title":"StepResult","text":"
StepResult(\n    index: int,\n    node_id: str,\n    states: tuple[State, ...],\n    completed: bool,\n)\n

A single checkpointed step within an async/resumable engine run.

Attributes:

Name Type Description index int

Ordinal index of the step.

node_id str

Identifier of the node associated with this step.

states tuple[State, ...]

States produced by running this step.

completed bool

Whether this step succeeded (vs. paused/interrupted).

Initialise StepResult.

Parameters:

Name Type Description Default index int

Ordinal index of the step.

required node_id str

Identifier of the node associated with this step.

required states tuple[State, ...]

States produced by running this step.

required completed bool

Whether this step succeeded (vs. paused/interrupted).

required"},{"location":"#dagpipe.StepResult-functions","title":"Functions","text":""},{"location":"#dagpipe-functions","title":"Functions","text":""},{"location":"#dagpipe.load_pipeline","title":"load_pipeline","text":"
load_pipeline(path: str) -> Pipeline\n

Load pipeline from YAML file.

Parameters:

Name Type Description Default path str

Path to YAML configuration file.

required

Returns:

Name Type Description Pipeline Pipeline

Executable pipeline instance.

Notes

Responsibilities:

- Loads YAML configuration and builds schema.\n- Creates `State` subclass and loads `Node` instances.\n- Builds `Graph` topology and initializes `Engine`.\n
"},{"location":"engine/","title":"Engine","text":""},{"location":"engine/#dagpipe.engine","title":"dagpipe.engine","text":""},{"location":"engine/#dagpipe.engine--summary","title":"Summary","text":"

Execution engine responsible for running pipelines and graphs.

The Engine executes Node objects and propagates immutable State instances through either a linear sequence or a directed acyclic graph (Graph). It orchestrates execution order, branching, and state propagation.

"},{"location":"engine/#dagpipe.engine--guarantees","title":"Guarantees","text":""},{"location":"engine/#dagpipe.engine-classes","title":"Classes","text":""},{"location":"engine/#dagpipe.engine.Engine","title":"Engine","text":"
Engine(\n    nodes_or_graph: Sequence[Node] | Graph,\n    *,\n    on_step: StepHook | None = None\n)\n

Execution engine responsible for running pipeline logic.

Notes

Responsibilities:

- Accepts either a linear sequence of `Node` objects or a `Graph`\n  defining execution topology.\n- Propagates immutable `State` objects through `Node` objects and\n  collects terminal states.\n- Supports synchronous (`run`) and asynchronous (`run_async`)\n  execution, dispatching per-node.\n- Supports step-wise / resumable execution and progress hooks.\n

Guarantees:

- Never mutates `State`, `Node`, or `Graph` instances.\n- `State` objects are never modified in place; each branch produces\n  independent instances.\n- Execution order is deterministic and follows graph or pipeline topology.\n- Thread-safe for concurrent execution.\n
"},{"location":"engine/#dagpipe.engine.Engine-attributes","title":"Attributes","text":""},{"location":"engine/#dagpipe.engine.Engine.nodes","title":"nodes property","text":"
nodes: tuple[Node, ...]\n

Return nodes managed by this engine.

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Ordered sequence in linear mode or all nodes in graph mode.

"},{"location":"engine/#dagpipe.engine.Engine-functions","title":"Functions","text":""},{"location":"engine/#dagpipe.engine.Engine.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return the canonical string representation of the object.

Returns:

Name Type Description str str

Representation that uniquely identifies the object and its configuration.

"},{"location":"engine/#dagpipe.engine.Engine.run","title":"run","text":"
run(root: State) -> list[State]\n

Execute the pipeline starting from a root State.

Parameters:

Name Type Description Default root State

Initial execution state.

required

Returns:

Type Description list[State]

list[State]: Terminal execution states produced by the pipeline.

Raises:

Type Description TypeError

If root is not a State instance.

RuntimeError

If the engine execution mode is invalid.

Notes

Responsibilities:

- Selects execution mode, propagates state through nodes, creates\n  new instances for branches, and collects terminal states.\n
"},{"location":"engine/#dagpipe.engine.Engine.run_async","title":"run_async async","text":"
run_async(root: State) -> list[State]\n

Execute the pipeline starting from root, dispatching sync vs async nodes.

Parameters:

Name Type Description Default root State

Initial execution state.

required

Returns:

Type Description list[State]

list[State]: Terminal execution states produced by the pipeline.

Notes

Each node is executed with Node.run when synchronous and AsyncNode.run_async when asynchronous. Linear and graph topologies are both supported.

"},{"location":"engine/#dagpipe.engine.Engine.run_steps","title":"run_steps","text":"
run_steps(\n    root: State,\n    *,\n    resume_from: int | None = None,\n    on_step: StepHook | None = None\n) -> Iterator[StepResult]\n

Execute the pipeline step-by-step, yielding one StepResult per step.

Parameters:

Name Type Description Default root State

Initial execution state.

required resume_from int | None

Skip steps at index < resume_from (for resume-after-partial). Steps are 0-indexed.

None on_step StepHook | None

Callback (step, status, message) invoked per step; falls back to the engine-level hook when unset.

None

Yields:

Name Type Description StepResult StepResult

One per executed node/step, carrying the produced states.

Notes

This is a synchronous, generator-based checkpoint interface compatible with the imperative resume-by-step behaviour of the legacy orchestrator. Use run_steps_async for async nodes.

"},{"location":"engine/#dagpipe.engine.Engine.run_steps_async","title":"run_steps_async async","text":"
run_steps_async(\n    root: State,\n    *,\n    resume_from: int | None = None,\n    on_step: AsyncStepHook | None = None\n) -> AsyncIterator[StepResult]\n

Async variant of run_steps supporting AsyncNode execution.

Parameters:

Name Type Description Default root State

Initial execution state.

required resume_from int | None

Skip steps at index < resume_from.

None on_step AsyncStepHook | None

Async callback (step, status, message) invoked per step.

None

Yields:

Name Type Description StepResult AsyncIterator[StepResult]

One per executed node/step.

"},{"location":"engine/#dagpipe.engine.ProgressMessage","title":"ProgressMessage","text":"
ProgressMessage(\n    *,\n    lines: int | None = None,\n    blocks: int | None = None,\n    count: int | None = None,\n    unit: str | None = None,\n    raw_ocr_line: str | None = None,\n    error: str | None = None,\n    step: str = \"\",\n    status: str = \"\"\n)\n

Lightweight progress payload emitted by engine step hooks.

Mirrors the imperative ProgressMessage used by the legacy orchestrator so callers can surface counts/lines/errors without coupling the engine to pydantic.

"},{"location":"engine/#dagpipe.engine.StepResult","title":"StepResult","text":"
StepResult(\n    index: int,\n    node_id: str,\n    states: tuple[State, ...],\n    completed: bool,\n)\n

A single checkpointed step within an async/resumable engine run.

Attributes:

Name Type Description index int

Ordinal index of the step.

node_id str

Identifier of the node associated with this step.

states tuple[State, ...]

States produced by running this step.

completed bool

Whether this step succeeded (vs. paused/interrupted).

Initialise StepResult.

Parameters:

Name Type Description Default index int

Ordinal index of the step.

required node_id str

Identifier of the node associated with this step.

required states tuple[State, ...]

States produced by running this step.

required completed bool

Whether this step succeeded (vs. paused/interrupted).

required"},{"location":"engine/#dagpipe.engine.StepResult-functions","title":"Functions","text":""},{"location":"graph/","title":"Graph","text":""},{"location":"graph/#dagpipe.graph","title":"dagpipe.graph","text":""},{"location":"graph/#dagpipe.graph--summary","title":"Summary","text":"

Defines DAG structure connecting nodes.

A Graph describes execution topology only. It does not execute nodes or manage State. Execution is handled by an Engine.

"},{"location":"graph/#dagpipe.graph--responsibilities","title":"Responsibilities","text":""},{"location":"graph/#dagpipe.graph-classes","title":"Classes","text":""},{"location":"graph/#dagpipe.graph.Graph","title":"Graph","text":"
Graph()\n

Directed Acyclic Graph defining execution topology of Node objects.

Notes

Responsibilities:

- Stores node connectivity and validates that the topology remains acyclic.\n- Structure determines how `State` flows between nodes during execution.\n

Guarantees:

- Topology is acyclic. Node relationships remain consistent.\n- Thread-safe for concurrent reads after construction.\n

Create an empty Graph.

"},{"location":"graph/#dagpipe.graph.Graph--initializes-node-registry-and-edge-mappings","title":"Initializes node registry and edge mappings.","text":""},{"location":"graph/#dagpipe.graph.Graph-functions","title":"Functions","text":""},{"location":"graph/#dagpipe.graph.Graph.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"graph/#dagpipe.graph.Graph.__repr__--returns","title":"Returns","text":"

str

"},{"location":"graph/#dagpipe.graph.Graph.add_edge","title":"add_edge","text":"
add_edge(src: Node, dst: Node) -> None\n

Add a directed edge from src to dst.

Parameters:

Name Type Description Default src Node

Source node.

required dst Node

Destination node.

required

Raises:

Type Description TypeError

If src or dst is not a Node.

ValueError

If the edge would create a cycle or if src and dst are common.

Notes "},{"location":"graph/#dagpipe.graph.Graph.add_root","title":"add_root","text":"
add_root(node: Node) -> None\n

Add a root node with no parents.

Parameters:

Name Type Description Default node Node

Node to add as a root.

required

Raises:

Type Description TypeError

If node is not a Node instance.

"},{"location":"graph/#dagpipe.graph.Graph.children","title":"children","text":"
children(node: Node) -> tuple[Node, ...]\n

Return child nodes of a node.

Parameters:

Name Type Description Default node Node

Node to query.

required

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Outgoing neighbors.

"},{"location":"graph/#dagpipe.graph.Graph.nodes","title":"nodes","text":"
nodes() -> tuple[Node, ...]\n

Return all nodes in the graph.

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: All registered nodes.

"},{"location":"graph/#dagpipe.graph.Graph.parents","title":"parents","text":"
parents(node: Node) -> tuple[Node, ...]\n

Return parent nodes of a node.

Parameters:

Name Type Description Default node Node

Node to query.

required

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Incoming neighbors.

"},{"location":"graph/#dagpipe.graph.Graph.roots","title":"roots","text":"
roots() -> tuple[Node, ...]\n

Return root nodes (nodes with no incoming edges).

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Entry point nodes.

"},{"location":"node/","title":"Node","text":""},{"location":"node/#dagpipe.node","title":"dagpipe.node","text":""},{"location":"node/#dagpipe.node--summary","title":"Summary","text":"

Defines the Node abstraction used by dagpipe.

A node represents a single unit of pipeline execution logic. It consumes one State and produces zero, one, or many new State objects.

Nodes are connected using a Graph and executed by an Engine.

"},{"location":"node/#dagpipe.node--design-principles","title":"Design principles","text":""},{"location":"node/#dagpipe.node-classes","title":"Classes","text":""},{"location":"node/#dagpipe.node.AsyncNode","title":"AsyncNode","text":"

Bases: Node

Base class for nodes whose execution is asynchronous.

Subclasses implement resolve_async (an async generator yielding derived State objects). The engine dispatches to resolve_async when running an async traversal (see Engine.run_async).

Sync-only engines (and the base Node.run) treat an AsyncNode as a no-op consumer: calling run on an AsyncNode returns no states, signalling that an async engine is required.

"},{"location":"node/#dagpipe.node.AsyncNode-functions","title":"Functions","text":""},{"location":"node/#dagpipe.node.AsyncNode.__hash__","title":"__hash__","text":"
__hash__() -> int\n

Return stable hash based on node ID.

"},{"location":"node/#dagpipe.node.AsyncNode.__hash__--returns","title":"Returns","text":"

int

"},{"location":"node/#dagpipe.node.AsyncNode.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"node/#dagpipe.node.AsyncNode.__repr__--returns","title":"Returns","text":"

str

"},{"location":"node/#dagpipe.node.AsyncNode.__str__","title":"__str__","text":"
__str__() -> str\n

Return display representation.

"},{"location":"node/#dagpipe.node.AsyncNode.__str__--returns","title":"Returns","text":"

str

"},{"location":"node/#dagpipe.node.AsyncNode.clean_id_and_name","title":"clean_id_and_name classmethod","text":"
clean_id_and_name() -> None\n

Normalize and validate node ID and display name.

Raises:

Type Description TypeError

If ID is not a string.

ValueError

If ID format is invalid.

Notes

Guarantees:

- Generates ID from module and class name if missing.\n- Validates ID format.\n- Generates human-readable name if missing.\n
"},{"location":"node/#dagpipe.node.AsyncNode.fork","title":"fork","text":"
fork(\n    state: State,\n    *,\n    payload_update: Any = None,\n    confidence_delta: float = 0.0,\n    metadata_update: Any = None\n) -> State\n

Create a child State attributed to this node.

Parameters:

Name Type Description Default state State

Parent execution state.

required payload_update Any

Dot-path payload updates.

None confidence_delta float

Confidence adjustment.

0.0 metadata_update Any

Metadata updates.

None

Returns:

Name Type Description State State

New child execution state.

Notes

Responsibilities:

- Convenience wrapper around `State.fork()` that automatically\n  records this node's ID in state history.\n
"},{"location":"node/#dagpipe.node.AsyncNode.is_async","title":"is_async","text":"
is_async() -> bool\n

Return whether this node executes asynchronously.

"},{"location":"node/#dagpipe.node.AsyncNode.node_id_to_name","title":"node_id_to_name staticmethod","text":"
node_id_to_name(node_id: str) -> str\n

Convert a dotted snake_case node ID into a human-readable name.

Parameters:

Name Type Description Default node_id str

Unique node identifier (e.g., 'entity.resolve.numeric_merchant').

required

Returns:

Name Type Description str str

Human-readable display name (e.g., 'Entity \u203a Resolve \u203a Numeric Merchant').

"},{"location":"node/#dagpipe.node.AsyncNode.resolve_async","title":"resolve_async async","text":"
resolve_async(state: State) -> Iterable[State]\n

Execute node logic asynchronously.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description Iterable[State]

Iterable[State]: Derived execution state(s).

Notes

Subclasses implement this. Must not mutate the input state. Should use fork() to create child states.

"},{"location":"node/#dagpipe.node.AsyncNode.run","title":"run","text":"
run(state: State) -> tuple[State, ...]\n

Execute this node on a State.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Derived execution states.

Raises:

Type Description TypeError

If resolve() yields a non-State object.

"},{"location":"node/#dagpipe.node.AsyncNode.run_async","title":"run_async async","text":"
run_async(state: State) -> tuple[State, ...]\n

Execute this node asynchronously on a state, validating outputs.

"},{"location":"node/#dagpipe.node.Node","title":"Node","text":"

Bases: ABC

Base class for all dagpipe execution nodes.

Attributes:

Name Type Description id str

Unique identifier of the node (snake_case dotted format).

name str

Human-readable display name.

Notes

Responsibilities:

- Represents a deterministic unit of execution in the pipeline graph.\n- Consumes one `State` and produces zero, one, or many derived states.\n- Defines execution logic and enables branching, filtering, and transformation.\n

Guarantees:

- Nodes must never mutate the input `State`.\n- Instances are singletons per subclass and reused across executions.\n
"},{"location":"node/#dagpipe.node.Node-functions","title":"Functions","text":""},{"location":"node/#dagpipe.node.Node.__hash__","title":"__hash__","text":"
__hash__() -> int\n

Return stable hash based on node ID.

"},{"location":"node/#dagpipe.node.Node.__hash__--returns","title":"Returns","text":"

int

"},{"location":"node/#dagpipe.node.Node.__new__","title":"__new__","text":"
__new__(*args: Any, **kwargs: Any) -> Node\n

Create or reuse a Node instance.

Stateless subclasses (no parameterized __init__) share one singleton instance per class \u2014 matching the original dagpipe behaviour underpinning set_registry-style configuration. Subclasses that declare an __init__ requiring instance-state arguments get a fresh instance per construction so pipeline builders can inject per-run dependencies.

"},{"location":"node/#dagpipe.node.Node.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"node/#dagpipe.node.Node.__repr__--returns","title":"Returns","text":"

str

"},{"location":"node/#dagpipe.node.Node.__str__","title":"__str__","text":"
__str__() -> str\n

Return display representation.

"},{"location":"node/#dagpipe.node.Node.__str__--returns","title":"Returns","text":"

str

"},{"location":"node/#dagpipe.node.Node.clean_id_and_name","title":"clean_id_and_name classmethod","text":"
clean_id_and_name() -> None\n

Normalize and validate node ID and display name.

Raises:

Type Description TypeError

If ID is not a string.

ValueError

If ID format is invalid.

Notes

Guarantees:

- Generates ID from module and class name if missing.\n- Validates ID format.\n- Generates human-readable name if missing.\n
"},{"location":"node/#dagpipe.node.Node.fork","title":"fork","text":"
fork(\n    state: State,\n    *,\n    payload_update: Any = None,\n    confidence_delta: float = 0.0,\n    metadata_update: Any = None\n) -> State\n

Create a child State attributed to this node.

Parameters:

Name Type Description Default state State

Parent execution state.

required payload_update Any

Dot-path payload updates.

None confidence_delta float

Confidence adjustment.

0.0 metadata_update Any

Metadata updates.

None

Returns:

Name Type Description State State

New child execution state.

Notes

Responsibilities:

- Convenience wrapper around `State.fork()` that automatically\n  records this node's ID in state history.\n
"},{"location":"node/#dagpipe.node.Node.is_async","title":"is_async","text":"
is_async() -> bool\n

Return whether this node executes asynchronously.

"},{"location":"node/#dagpipe.node.Node.node_id_to_name","title":"node_id_to_name staticmethod","text":"
node_id_to_name(node_id: str) -> str\n

Convert a dotted snake_case node ID into a human-readable name.

Parameters:

Name Type Description Default node_id str

Unique node identifier (e.g., 'entity.resolve.numeric_merchant').

required

Returns:

Name Type Description str str

Human-readable display name (e.g., 'Entity \u203a Resolve \u203a Numeric Merchant').

"},{"location":"node/#dagpipe.node.Node.resolve","title":"resolve abstractmethod","text":"
resolve(state: State) -> Iterable[State]\n

Execute node logic.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description Iterable[State]

Iterable[State]: Derived execution state(s).

Notes

Responsibilities:

- Subclasses implement specific resolution behavior.\n- Must not mutate input state.\n- Should use `fork()` to create child states.\n- May yield zero states to terminate a branch.\n
"},{"location":"node/#dagpipe.node.Node.run","title":"run","text":"
run(state: State) -> tuple[State, ...]\n

Execute this node on a State.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Derived execution states.

Raises:

Type Description TypeError

If resolve() yields a non-State object.

"},{"location":"state/","title":"State","text":""},{"location":"state/#dagpipe.state","title":"dagpipe.state","text":""},{"location":"state/#dagpipe.state--summary","title":"Summary","text":"

Defines the core State object used by dagpipe.

The State represents a single point in pipeline execution. It contains arbitrary data and metadata and is designed to be immutable. Instead of modifying an existing state, nodes create new child states via fork().

"},{"location":"state/#dagpipe.state--design-principles","title":"Design principles","text":""},{"location":"state/#dagpipe.state-classes","title":"Classes","text":""},{"location":"state/#dagpipe.state.Payload","title":"Payload dataclass","text":"
Payload(_data: Mapping[str, Any])\n

Immutable hierarchical container with dot-path access.

Attributes:

Name Type Description _data Mapping[str, Any]

Immutable hierarchical data structure.

Notes

Responsibilities:

- Stores execution data used by `State`.\n- Supports efficient atomic updates without modifying existing instances.\n- `Payload` instances are fully thread-safe due to immutability.\n
"},{"location":"state/#dagpipe.state.Payload-functions","title":"Functions","text":""},{"location":"state/#dagpipe.state.Payload.as_dict","title":"as_dict","text":"
as_dict() -> Mapping[str, Any]\n

Return underlying mapping.

Returns:

Type Description Mapping[str, Any]

Mapping[str, Any]: Read-only view of the underlying data.

"},{"location":"state/#dagpipe.state.Payload.get","title":"get","text":"
get(path: str, default: Any = None) -> Any\n

Retrieve value using dot-path.

Parameters:

Name Type Description Default path str

Dot-separated path to the value.

required default Any

Default value if path doesn't exist.

None

Returns:

Name Type Description Any Any

The retrieved value or default.

"},{"location":"state/#dagpipe.state.Payload.has","title":"has","text":"
has(path: str) -> bool\n

Return True if path exists.

Parameters:

Name Type Description Default path str

Dot-separated path to check.

required

Returns:

Name Type Description bool bool

Existence of the path.

"},{"location":"state/#dagpipe.state.Payload.iter_paths","title":"iter_paths classmethod","text":"
iter_paths(\n    data: Mapping[str, Any], prefix: str = \"\"\n) -> Iterable[str]\n

Recursively yield dot-paths for all leaf nodes.

Parameters:

Name Type Description Default data Mapping[str, Any]

The mapping to iterate over.

required prefix str

Current path prefix.

''

Yields:

Name Type Description str Iterable[str]

Dot-path for each leaf node.

"},{"location":"state/#dagpipe.state.Payload.keys","title":"keys","text":"
keys() -> Iterable[str]\n

Return top-level keys.

Returns:

Type Description Iterable[str]

Iterable[str]: Iterator over top-level keys.

"},{"location":"state/#dagpipe.state.Payload.update","title":"update","text":"
update(updates: Mapping[str, Any]) -> Payload\n

Create a new Payload with dot-path updates applied.

Parameters:

Name Type Description Default updates Mapping[str, Any]

Dot-path to value mapping.

required

Returns:

Name Type Description Payload Payload

New immutable payload instance with updates.

Notes

Guarantees:

- Preserves existing data by copying only modified branches.\n- Returns a new immutable `Payload`.\n
"},{"location":"state/#dagpipe.state.Schema","title":"Schema dataclass","text":"
Schema(tree: Mapping[str, SchemaNode])\n

Immutable hierarchical schema defining allowed payload structure.

Attributes:

Name Type Description tree Mapping[str, SchemaNode]

Hierarchical schema definition.

Notes

Responsibilities:

- Validates `State` payloads and updates.\n- Reusable across all `State` instances.\n- Fully thread-safe due to immutability.\n
"},{"location":"state/#dagpipe.state.Schema-functions","title":"Functions","text":""},{"location":"state/#dagpipe.state.Schema.validate_payload","title":"validate_payload","text":"
validate_payload(payload: Payload) -> None\n

Validate complete payload structure.

Parameters:

Name Type Description Default payload Payload

Payload to validate.

required

Raises:

Type Description SchemaError

If payload violates schema.

"},{"location":"state/#dagpipe.state.Schema.validate_update","title":"validate_update","text":"
validate_update(updates: Mapping[str, Any]) -> None\n

Validate payload update paths.

Parameters:

Name Type Description Default updates Mapping[str, Any]

Dot-path updates to validate.

required

Raises:

Type Description SchemaError

If any path is invalid according to the schema.

"},{"location":"state/#dagpipe.state.SchemaError","title":"SchemaError","text":"

Bases: Exception

Raised when payload data violates the declared schema.

"},{"location":"state/#dagpipe.state.SchemaError--indicates-invalid-structure-invalid-path-or-invalid-type","title":"Indicates invalid structure, invalid path, or invalid type.","text":""},{"location":"state/#dagpipe.state.State","title":"State dataclass","text":"
State(\n    payload: Payload,\n    confidence: float = 1.0,\n    parent: State | None = None,\n    depth: int = 0,\n    history: tuple[str, ...] = tuple(),\n    metadata: dict[str, Any] = dict(),\n)\n

Immutable execution state propagated through dagpipe pipeline.

Attributes:

Name Type Description payload Payload

Execution data container.

schema ClassVar[Schema]

Payload validation schema.

confidence float

Execution confidence score.

parent Optional[State]

Parent state reference.

depth int

Execution depth.

history Tuple[str, ...]

Ordered node execution lineage.

metadata Dict[str, Any]

Execution metadata.

Notes

Responsibilities:

- Represents a complete execution snapshot at a specific point in\n  pipeline traversal.\n- Fundamental unit of execution in `dagpipe`.\n- Fully thread-safe due to immutability.\n
"},{"location":"state/#dagpipe.state.State-functions","title":"Functions","text":""},{"location":"state/#dagpipe.state.State.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Concise debug representation.

Avoids printing full data for large states.

"},{"location":"state/#dagpipe.state.State.fork","title":"fork","text":"
fork(\n    *,\n    payload_update: Mapping[str, Any] | None = None,\n    confidence_delta: float = 0.0,\n    node_id: str | None = None,\n    metadata_update: Mapping[str, Any] | None = None\n) -> State\n

Create a new child State derived from this state.

Parameters:

Name Type Description Default payload_update Mapping[str, Any] | None

Dot-path updates applied to the payload.

None confidence_delta float

Adjustment applied to current confidence.

0.0 node_id str | None

Identifier of the node creating this state.

None metadata_update Mapping[str, Any] | None

Updates merged into state metadata.

None

Returns:

Name Type Description State State

A new immutable State instance.

Notes

Guarantees:

- This is the only supported mechanism for modifying execution data.\n- Validates payload updates, preserves lineage, increments depth,\n  and appends to history.\n
"},{"location":"state/#dagpipe.state.State.get","title":"get","text":"
get(key: str, default: Any = None) -> Any\n

Retrieve payload value.

Parameters:

Name Type Description Default key str

Dot-path key.

required default Any

Fallback value.

None

Returns:

Name Type Description Any Any

Stored value or default.

"},{"location":"state/#dagpipe.state.State.has","title":"has","text":"
has(key: str) -> bool\n

Check whether payload contains key.

Parameters:

Name Type Description Default key str

Dot-path key.

required

Returns:

Name Type Description bool bool

Existence of the key.

"},{"location":"state/#dagpipe.state.State.lineage","title":"lineage","text":"
lineage() -> tuple[State, ...]\n

Return lineage from root to this State.

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Ordered execution lineage (root first).

"},{"location":"yaml_loader/","title":"Yaml Loader","text":""},{"location":"yaml_loader/#dagpipe.yaml_loader","title":"dagpipe.yaml_loader","text":""},{"location":"yaml_loader/#dagpipe.yaml_loader--summary","title":"Summary","text":"

Loads dagpipe pipelines from YAML configuration.

Creates fully configured pipeline objects from declarative YAML definitions, including Schema, State subclasses, Node instances, Graph topology, and initial payloads.

"},{"location":"yaml_loader/#dagpipe.yaml_loader-classes","title":"Classes","text":""},{"location":"yaml_loader/#dagpipe.yaml_loader.Pipeline","title":"Pipeline dataclass","text":"
Pipeline(\n    engine: Engine,\n    state_cls: type[State],\n    initial_payload: Payload,\n)\n

Executable pipeline created from YAML configuration.

Attributes:

Name Type Description engine Engine

Execution engine responsible for running the pipeline.

state_cls Type[State]

Dynamically created State subclass with configured schema.

initial_payload Payload

Default payload used when execution begins.

Notes

Responsibilities:

- Encapsulates engine, state type, and initial payload.\n- Provides a simplified interface for executing configured pipelines.\n- Safe for concurrent execution if underlying nodes are thread-safe.\n
"},{"location":"yaml_loader/#dagpipe.yaml_loader.Pipeline-functions","title":"Functions","text":""},{"location":"yaml_loader/#dagpipe.yaml_loader.Pipeline.run","title":"run","text":"
run(\n    payload_override: Mapping[str, Any] | None = None,\n) -> list[State]\n

Execute the pipeline.

Parameters:

Name Type Description Default payload_override Mapping[str, Any] | None

Payload values overriding initial payload.

None

Returns:

Type Description list[State]

list[State]: Terminal execution states.

Notes

Responsibilities:

- Merges override payload with initial payload.\n- Creates root `State` and executes engine.\n
"},{"location":"yaml_loader/#dagpipe.yaml_loader-functions","title":"Functions","text":""},{"location":"yaml_loader/#dagpipe.yaml_loader.load_pipeline","title":"load_pipeline","text":"
load_pipeline(path: str) -> Pipeline\n

Load pipeline from YAML file.

Parameters:

Name Type Description Default path str

Path to YAML configuration file.

required

Returns:

Name Type Description Pipeline Pipeline

Executable pipeline instance.

Notes

Responsibilities:

- Loads YAML configuration and builds schema.\n- Creates `State` subclass and loads `Node` instances.\n- Builds `Graph` topology and initializes `Engine`.\n
"},{"location":"dagpipe/","title":"Dagpipe","text":""},{"location":"dagpipe/#dagpipe","title":"dagpipe","text":""},{"location":"dagpipe/#dagpipe--summary","title":"Summary","text":"

Directed acyclic graph execution framework for deterministic state propagation.

dagpipe executes pipelines composed of nodes connected in a directed acyclic graph (DAG). Each node receives an immutable State and optionally produces derived states for downstream nodes.

"},{"location":"dagpipe/#dagpipe--installation","title":"Installation","text":"

Install using pip:

pip install dagpipe\n
"},{"location":"dagpipe/#dagpipe--quick-start","title":"Quick Start","text":"
from dagpipe import State, Payload, Schema, Graph, Engine\nfrom dagpipe.node import Node\n\nclass HelloNode(Node):\n    id = \"hello\"\n    def resolve(self, state):\n        yield self.fork(state, payload_update={\"msg\": \"hello\"})\n\n# Build and run\ngraph = Graph()\ngraph.add_root(HelloNode())\nengine = Engine(graph)\n\nclass MyState(State):\n    schema = Schema({})\n\nresults = engine.run(MyState(payload=Payload({})))\n
"},{"location":"dagpipe/#dagpipe--public-api","title":"Public API","text":"

This package re-exports the core pipeline components. Consumers should import from this namespace for standard usage.

"},{"location":"dagpipe/#dagpipe--execution-core","title":"Execution Core","text":""},{"location":"dagpipe/#dagpipe--state-data","title":"State & Data","text":""},{"location":"dagpipe/#dagpipe--declarative-pipelines","title":"Declarative Pipelines","text":""},{"location":"dagpipe/#dagpipe-classes","title":"Classes","text":""},{"location":"dagpipe/#dagpipe.AsyncNode","title":"AsyncNode","text":"

Bases: Node

Base class for nodes whose execution is asynchronous.

Subclasses implement resolve_async (an async generator yielding derived State objects). The engine dispatches to resolve_async when running an async traversal (see Engine.run_async).

Sync-only engines (and the base Node.run) treat an AsyncNode as a no-op consumer: calling run on an AsyncNode returns no states, signalling that an async engine is required.

"},{"location":"dagpipe/#dagpipe.AsyncNode-functions","title":"Functions","text":""},{"location":"dagpipe/#dagpipe.AsyncNode.__hash__","title":"__hash__","text":"
__hash__() -> int\n

Return stable hash based on node ID.

"},{"location":"dagpipe/#dagpipe.AsyncNode.__hash__--returns","title":"Returns","text":"

int

"},{"location":"dagpipe/#dagpipe.AsyncNode.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"dagpipe/#dagpipe.AsyncNode.__repr__--returns","title":"Returns","text":"

str

"},{"location":"dagpipe/#dagpipe.AsyncNode.__str__","title":"__str__","text":"
__str__() -> str\n

Return display representation.

"},{"location":"dagpipe/#dagpipe.AsyncNode.__str__--returns","title":"Returns","text":"

str

"},{"location":"dagpipe/#dagpipe.AsyncNode.clean_id_and_name","title":"clean_id_and_name classmethod","text":"
clean_id_and_name() -> None\n

Normalize and validate node ID and display name.

Raises:

Type Description TypeError

If ID is not a string.

ValueError

If ID format is invalid.

Notes

Guarantees:

- Generates ID from module and class name if missing.\n- Validates ID format.\n- Generates human-readable name if missing.\n
"},{"location":"dagpipe/#dagpipe.AsyncNode.fork","title":"fork","text":"
fork(\n    state: State,\n    *,\n    payload_update: Any = None,\n    confidence_delta: float = 0.0,\n    metadata_update: Any = None\n) -> State\n

Create a child State attributed to this node.

Parameters:

Name Type Description Default state State

Parent execution state.

required payload_update Any

Dot-path payload updates.

None confidence_delta float

Confidence adjustment.

0.0 metadata_update Any

Metadata updates.

None

Returns:

Name Type Description State State

New child execution state.

Notes

Responsibilities:

- Convenience wrapper around `State.fork()` that automatically\n  records this node's ID in state history.\n
"},{"location":"dagpipe/#dagpipe.AsyncNode.is_async","title":"is_async","text":"
is_async() -> bool\n

Return whether this node executes asynchronously.

"},{"location":"dagpipe/#dagpipe.AsyncNode.node_id_to_name","title":"node_id_to_name staticmethod","text":"
node_id_to_name(node_id: str) -> str\n

Convert a dotted snake_case node ID into a human-readable name.

Parameters:

Name Type Description Default node_id str

Unique node identifier (e.g., 'entity.resolve.numeric_merchant').

required

Returns:

Name Type Description str str

Human-readable display name (e.g., 'Entity \u203a Resolve \u203a Numeric Merchant').

"},{"location":"dagpipe/#dagpipe.AsyncNode.resolve_async","title":"resolve_async async","text":"
resolve_async(state: State) -> Iterable[State]\n

Execute node logic asynchronously.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description Iterable[State]

Iterable[State]: Derived execution state(s).

Notes

Subclasses implement this. Must not mutate the input state. Should use fork() to create child states.

"},{"location":"dagpipe/#dagpipe.AsyncNode.run","title":"run","text":"
run(state: State) -> tuple[State, ...]\n

Execute this node on a State.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Derived execution states.

Raises:

Type Description TypeError

If resolve() yields a non-State object.

"},{"location":"dagpipe/#dagpipe.AsyncNode.run_async","title":"run_async async","text":"
run_async(state: State) -> tuple[State, ...]\n

Execute this node asynchronously on a state, validating outputs.

"},{"location":"dagpipe/#dagpipe.Engine","title":"Engine","text":"
Engine(\n    nodes_or_graph: Sequence[Node] | Graph,\n    *,\n    on_step: StepHook | None = None\n)\n

Execution engine responsible for running pipeline logic.

Notes

Responsibilities:

- Accepts either a linear sequence of `Node` objects or a `Graph`\n  defining execution topology.\n- Propagates immutable `State` objects through `Node` objects and\n  collects terminal states.\n- Supports synchronous (`run`) and asynchronous (`run_async`)\n  execution, dispatching per-node.\n- Supports step-wise / resumable execution and progress hooks.\n

Guarantees:

- Never mutates `State`, `Node`, or `Graph` instances.\n- `State` objects are never modified in place; each branch produces\n  independent instances.\n- Execution order is deterministic and follows graph or pipeline topology.\n- Thread-safe for concurrent execution.\n
"},{"location":"dagpipe/#dagpipe.Engine-attributes","title":"Attributes","text":""},{"location":"dagpipe/#dagpipe.Engine.nodes","title":"nodes property","text":"
nodes: tuple[Node, ...]\n

Return nodes managed by this engine.

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Ordered sequence in linear mode or all nodes in graph mode.

"},{"location":"dagpipe/#dagpipe.Engine-functions","title":"Functions","text":""},{"location":"dagpipe/#dagpipe.Engine.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return the canonical string representation of the object.

Returns:

Name Type Description str str

Representation that uniquely identifies the object and its configuration.

"},{"location":"dagpipe/#dagpipe.Engine.run","title":"run","text":"
run(root: State) -> list[State]\n

Execute the pipeline starting from a root State.

Parameters:

Name Type Description Default root State

Initial execution state.

required

Returns:

Type Description list[State]

list[State]: Terminal execution states produced by the pipeline.

Raises:

Type Description TypeError

If root is not a State instance.

RuntimeError

If the engine execution mode is invalid.

Notes

Responsibilities:

- Selects execution mode, propagates state through nodes, creates\n  new instances for branches, and collects terminal states.\n
"},{"location":"dagpipe/#dagpipe.Engine.run_async","title":"run_async async","text":"
run_async(root: State) -> list[State]\n

Execute the pipeline starting from root, dispatching sync vs async nodes.

Parameters:

Name Type Description Default root State

Initial execution state.

required

Returns:

Type Description list[State]

list[State]: Terminal execution states produced by the pipeline.

Notes

Each node is executed with Node.run when synchronous and AsyncNode.run_async when asynchronous. Linear and graph topologies are both supported.

"},{"location":"dagpipe/#dagpipe.Engine.run_steps","title":"run_steps","text":"
run_steps(\n    root: State,\n    *,\n    resume_from: int | None = None,\n    on_step: StepHook | None = None\n) -> Iterator[StepResult]\n

Execute the pipeline step-by-step, yielding one StepResult per step.

Parameters:

Name Type Description Default root State

Initial execution state.

required resume_from int | None

Skip steps at index < resume_from (for resume-after-partial). Steps are 0-indexed.

None on_step StepHook | None

Callback (step, status, message) invoked per step; falls back to the engine-level hook when unset.

None

Yields:

Name Type Description StepResult StepResult

One per executed node/step, carrying the produced states.

Notes

This is a synchronous, generator-based checkpoint interface compatible with the imperative resume-by-step behaviour of the legacy orchestrator. Use run_steps_async for async nodes.

"},{"location":"dagpipe/#dagpipe.Engine.run_steps_async","title":"run_steps_async async","text":"
run_steps_async(\n    root: State,\n    *,\n    resume_from: int | None = None,\n    on_step: AsyncStepHook | None = None\n) -> AsyncIterator[StepResult]\n

Async variant of run_steps supporting AsyncNode execution.

Parameters:

Name Type Description Default root State

Initial execution state.

required resume_from int | None

Skip steps at index < resume_from.

None on_step AsyncStepHook | None

Async callback (step, status, message) invoked per step.

None

Yields:

Name Type Description StepResult AsyncIterator[StepResult]

One per executed node/step.

"},{"location":"dagpipe/#dagpipe.Graph","title":"Graph","text":"
Graph()\n

Directed Acyclic Graph defining execution topology of Node objects.

Notes

Responsibilities:

- Stores node connectivity and validates that the topology remains acyclic.\n- Structure determines how `State` flows between nodes during execution.\n

Guarantees:

- Topology is acyclic. Node relationships remain consistent.\n- Thread-safe for concurrent reads after construction.\n

Create an empty Graph.

"},{"location":"dagpipe/#dagpipe.Graph--initializes-node-registry-and-edge-mappings","title":"Initializes node registry and edge mappings.","text":""},{"location":"dagpipe/#dagpipe.Graph-functions","title":"Functions","text":""},{"location":"dagpipe/#dagpipe.Graph.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"dagpipe/#dagpipe.Graph.__repr__--returns","title":"Returns","text":"

str

"},{"location":"dagpipe/#dagpipe.Graph.add_edge","title":"add_edge","text":"
add_edge(src: Node, dst: Node) -> None\n

Add a directed edge from src to dst.

Parameters:

Name Type Description Default src Node

Source node.

required dst Node

Destination node.

required

Raises:

Type Description TypeError

If src or dst is not a Node.

ValueError

If the edge would create a cycle or if src and dst are common.

Notes "},{"location":"dagpipe/#dagpipe.Graph.add_root","title":"add_root","text":"
add_root(node: Node) -> None\n

Add a root node with no parents.

Parameters:

Name Type Description Default node Node

Node to add as a root.

required

Raises:

Type Description TypeError

If node is not a Node instance.

"},{"location":"dagpipe/#dagpipe.Graph.children","title":"children","text":"
children(node: Node) -> tuple[Node, ...]\n

Return child nodes of a node.

Parameters:

Name Type Description Default node Node

Node to query.

required

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Outgoing neighbors.

"},{"location":"dagpipe/#dagpipe.Graph.nodes","title":"nodes","text":"
nodes() -> tuple[Node, ...]\n

Return all nodes in the graph.

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: All registered nodes.

"},{"location":"dagpipe/#dagpipe.Graph.parents","title":"parents","text":"
parents(node: Node) -> tuple[Node, ...]\n

Return parent nodes of a node.

Parameters:

Name Type Description Default node Node

Node to query.

required

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Incoming neighbors.

"},{"location":"dagpipe/#dagpipe.Graph.roots","title":"roots","text":"
roots() -> tuple[Node, ...]\n

Return root nodes (nodes with no incoming edges).

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Entry point nodes.

"},{"location":"dagpipe/#dagpipe.Node","title":"Node","text":"

Bases: ABC

Base class for all dagpipe execution nodes.

Attributes:

Name Type Description id str

Unique identifier of the node (snake_case dotted format).

name str

Human-readable display name.

Notes

Responsibilities:

- Represents a deterministic unit of execution in the pipeline graph.\n- Consumes one `State` and produces zero, one, or many derived states.\n- Defines execution logic and enables branching, filtering, and transformation.\n

Guarantees:

- Nodes must never mutate the input `State`.\n- Instances are singletons per subclass and reused across executions.\n
"},{"location":"dagpipe/#dagpipe.Node-functions","title":"Functions","text":""},{"location":"dagpipe/#dagpipe.Node.__hash__","title":"__hash__","text":"
__hash__() -> int\n

Return stable hash based on node ID.

"},{"location":"dagpipe/#dagpipe.Node.__hash__--returns","title":"Returns","text":"

int

"},{"location":"dagpipe/#dagpipe.Node.__new__","title":"__new__","text":"
__new__(*args: Any, **kwargs: Any) -> Node\n

Create or reuse a Node instance.

Stateless subclasses (no parameterized __init__) share one singleton instance per class \u2014 matching the original dagpipe behaviour underpinning set_registry-style configuration. Subclasses that declare an __init__ requiring instance-state arguments get a fresh instance per construction so pipeline builders can inject per-run dependencies.

"},{"location":"dagpipe/#dagpipe.Node.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"dagpipe/#dagpipe.Node.__repr__--returns","title":"Returns","text":"

str

"},{"location":"dagpipe/#dagpipe.Node.__str__","title":"__str__","text":"
__str__() -> str\n

Return display representation.

"},{"location":"dagpipe/#dagpipe.Node.__str__--returns","title":"Returns","text":"

str

"},{"location":"dagpipe/#dagpipe.Node.clean_id_and_name","title":"clean_id_and_name classmethod","text":"
clean_id_and_name() -> None\n

Normalize and validate node ID and display name.

Raises:

Type Description TypeError

If ID is not a string.

ValueError

If ID format is invalid.

Notes

Guarantees:

- Generates ID from module and class name if missing.\n- Validates ID format.\n- Generates human-readable name if missing.\n
"},{"location":"dagpipe/#dagpipe.Node.fork","title":"fork","text":"
fork(\n    state: State,\n    *,\n    payload_update: Any = None,\n    confidence_delta: float = 0.0,\n    metadata_update: Any = None\n) -> State\n

Create a child State attributed to this node.

Parameters:

Name Type Description Default state State

Parent execution state.

required payload_update Any

Dot-path payload updates.

None confidence_delta float

Confidence adjustment.

0.0 metadata_update Any

Metadata updates.

None

Returns:

Name Type Description State State

New child execution state.

Notes

Responsibilities:

- Convenience wrapper around `State.fork()` that automatically\n  records this node's ID in state history.\n
"},{"location":"dagpipe/#dagpipe.Node.is_async","title":"is_async","text":"
is_async() -> bool\n

Return whether this node executes asynchronously.

"},{"location":"dagpipe/#dagpipe.Node.node_id_to_name","title":"node_id_to_name staticmethod","text":"
node_id_to_name(node_id: str) -> str\n

Convert a dotted snake_case node ID into a human-readable name.

Parameters:

Name Type Description Default node_id str

Unique node identifier (e.g., 'entity.resolve.numeric_merchant').

required

Returns:

Name Type Description str str

Human-readable display name (e.g., 'Entity \u203a Resolve \u203a Numeric Merchant').

"},{"location":"dagpipe/#dagpipe.Node.resolve","title":"resolve abstractmethod","text":"
resolve(state: State) -> Iterable[State]\n

Execute node logic.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description Iterable[State]

Iterable[State]: Derived execution state(s).

Notes

Responsibilities:

- Subclasses implement specific resolution behavior.\n- Must not mutate input state.\n- Should use `fork()` to create child states.\n- May yield zero states to terminate a branch.\n
"},{"location":"dagpipe/#dagpipe.Node.run","title":"run","text":"
run(state: State) -> tuple[State, ...]\n

Execute this node on a State.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Derived execution states.

Raises:

Type Description TypeError

If resolve() yields a non-State object.

"},{"location":"dagpipe/#dagpipe.Payload","title":"Payload dataclass","text":"
Payload(_data: Mapping[str, Any])\n

Immutable hierarchical container with dot-path access.

Attributes:

Name Type Description _data Mapping[str, Any]

Immutable hierarchical data structure.

Notes

Responsibilities:

- Stores execution data used by `State`.\n- Supports efficient atomic updates without modifying existing instances.\n- `Payload` instances are fully thread-safe due to immutability.\n
"},{"location":"dagpipe/#dagpipe.Payload-functions","title":"Functions","text":""},{"location":"dagpipe/#dagpipe.Payload.as_dict","title":"as_dict","text":"
as_dict() -> Mapping[str, Any]\n

Return underlying mapping.

Returns:

Type Description Mapping[str, Any]

Mapping[str, Any]: Read-only view of the underlying data.

"},{"location":"dagpipe/#dagpipe.Payload.get","title":"get","text":"
get(path: str, default: Any = None) -> Any\n

Retrieve value using dot-path.

Parameters:

Name Type Description Default path str

Dot-separated path to the value.

required default Any

Default value if path doesn't exist.

None

Returns:

Name Type Description Any Any

The retrieved value or default.

"},{"location":"dagpipe/#dagpipe.Payload.has","title":"has","text":"
has(path: str) -> bool\n

Return True if path exists.

Parameters:

Name Type Description Default path str

Dot-separated path to check.

required

Returns:

Name Type Description bool bool

Existence of the path.

"},{"location":"dagpipe/#dagpipe.Payload.iter_paths","title":"iter_paths classmethod","text":"
iter_paths(\n    data: Mapping[str, Any], prefix: str = \"\"\n) -> Iterable[str]\n

Recursively yield dot-paths for all leaf nodes.

Parameters:

Name Type Description Default data Mapping[str, Any]

The mapping to iterate over.

required prefix str

Current path prefix.

''

Yields:

Name Type Description str Iterable[str]

Dot-path for each leaf node.

"},{"location":"dagpipe/#dagpipe.Payload.keys","title":"keys","text":"
keys() -> Iterable[str]\n

Return top-level keys.

Returns:

Type Description Iterable[str]

Iterable[str]: Iterator over top-level keys.

"},{"location":"dagpipe/#dagpipe.Payload.update","title":"update","text":"
update(updates: Mapping[str, Any]) -> Payload\n

Create a new Payload with dot-path updates applied.

Parameters:

Name Type Description Default updates Mapping[str, Any]

Dot-path to value mapping.

required

Returns:

Name Type Description Payload Payload

New immutable payload instance with updates.

Notes

Guarantees:

- Preserves existing data by copying only modified branches.\n- Returns a new immutable `Payload`.\n
"},{"location":"dagpipe/#dagpipe.Pipeline","title":"Pipeline dataclass","text":"
Pipeline(\n    engine: Engine,\n    state_cls: type[State],\n    initial_payload: Payload,\n)\n

Executable pipeline created from YAML configuration.

Attributes:

Name Type Description engine Engine

Execution engine responsible for running the pipeline.

state_cls Type[State]

Dynamically created State subclass with configured schema.

initial_payload Payload

Default payload used when execution begins.

Notes

Responsibilities:

- Encapsulates engine, state type, and initial payload.\n- Provides a simplified interface for executing configured pipelines.\n- Safe for concurrent execution if underlying nodes are thread-safe.\n
"},{"location":"dagpipe/#dagpipe.Pipeline-functions","title":"Functions","text":""},{"location":"dagpipe/#dagpipe.Pipeline.run","title":"run","text":"
run(\n    payload_override: Mapping[str, Any] | None = None,\n) -> list[State]\n

Execute the pipeline.

Parameters:

Name Type Description Default payload_override Mapping[str, Any] | None

Payload values overriding initial payload.

None

Returns:

Type Description list[State]

list[State]: Terminal execution states.

Notes

Responsibilities:

- Merges override payload with initial payload.\n- Creates root `State` and executes engine.\n
"},{"location":"dagpipe/#dagpipe.ProgressMessage","title":"ProgressMessage","text":"
ProgressMessage(\n    *,\n    lines: int | None = None,\n    blocks: int | None = None,\n    count: int | None = None,\n    unit: str | None = None,\n    raw_ocr_line: str | None = None,\n    error: str | None = None,\n    step: str = \"\",\n    status: str = \"\"\n)\n

Lightweight progress payload emitted by engine step hooks.

Mirrors the imperative ProgressMessage used by the legacy orchestrator so callers can surface counts/lines/errors without coupling the engine to pydantic.

"},{"location":"dagpipe/#dagpipe.Schema","title":"Schema dataclass","text":"
Schema(tree: Mapping[str, SchemaNode])\n

Immutable hierarchical schema defining allowed payload structure.

Attributes:

Name Type Description tree Mapping[str, SchemaNode]

Hierarchical schema definition.

Notes

Responsibilities:

- Validates `State` payloads and updates.\n- Reusable across all `State` instances.\n- Fully thread-safe due to immutability.\n
"},{"location":"dagpipe/#dagpipe.Schema-functions","title":"Functions","text":""},{"location":"dagpipe/#dagpipe.Schema.validate_payload","title":"validate_payload","text":"
validate_payload(payload: Payload) -> None\n

Validate complete payload structure.

Parameters:

Name Type Description Default payload Payload

Payload to validate.

required

Raises:

Type Description SchemaError

If payload violates schema.

"},{"location":"dagpipe/#dagpipe.Schema.validate_update","title":"validate_update","text":"
validate_update(updates: Mapping[str, Any]) -> None\n

Validate payload update paths.

Parameters:

Name Type Description Default updates Mapping[str, Any]

Dot-path updates to validate.

required

Raises:

Type Description SchemaError

If any path is invalid according to the schema.

"},{"location":"dagpipe/#dagpipe.SchemaError","title":"SchemaError","text":"

Bases: Exception

Raised when payload data violates the declared schema.

"},{"location":"dagpipe/#dagpipe.SchemaError--indicates-invalid-structure-invalid-path-or-invalid-type","title":"Indicates invalid structure, invalid path, or invalid type.","text":""},{"location":"dagpipe/#dagpipe.State","title":"State dataclass","text":"
State(\n    payload: Payload,\n    confidence: float = 1.0,\n    parent: State | None = None,\n    depth: int = 0,\n    history: tuple[str, ...] = tuple(),\n    metadata: dict[str, Any] = dict(),\n)\n

Immutable execution state propagated through dagpipe pipeline.

Attributes:

Name Type Description payload Payload

Execution data container.

schema ClassVar[Schema]

Payload validation schema.

confidence float

Execution confidence score.

parent Optional[State]

Parent state reference.

depth int

Execution depth.

history Tuple[str, ...]

Ordered node execution lineage.

metadata Dict[str, Any]

Execution metadata.

Notes

Responsibilities:

- Represents a complete execution snapshot at a specific point in\n  pipeline traversal.\n- Fundamental unit of execution in `dagpipe`.\n- Fully thread-safe due to immutability.\n
"},{"location":"dagpipe/#dagpipe.State-functions","title":"Functions","text":""},{"location":"dagpipe/#dagpipe.State.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Concise debug representation.

Avoids printing full data for large states.

"},{"location":"dagpipe/#dagpipe.State.fork","title":"fork","text":"
fork(\n    *,\n    payload_update: Mapping[str, Any] | None = None,\n    confidence_delta: float = 0.0,\n    node_id: str | None = None,\n    metadata_update: Mapping[str, Any] | None = None\n) -> State\n

Create a new child State derived from this state.

Parameters:

Name Type Description Default payload_update Mapping[str, Any] | None

Dot-path updates applied to the payload.

None confidence_delta float

Adjustment applied to current confidence.

0.0 node_id str | None

Identifier of the node creating this state.

None metadata_update Mapping[str, Any] | None

Updates merged into state metadata.

None

Returns:

Name Type Description State State

A new immutable State instance.

Notes

Guarantees:

- This is the only supported mechanism for modifying execution data.\n- Validates payload updates, preserves lineage, increments depth,\n  and appends to history.\n
"},{"location":"dagpipe/#dagpipe.State.get","title":"get","text":"
get(key: str, default: Any = None) -> Any\n

Retrieve payload value.

Parameters:

Name Type Description Default key str

Dot-path key.

required default Any

Fallback value.

None

Returns:

Name Type Description Any Any

Stored value or default.

"},{"location":"dagpipe/#dagpipe.State.has","title":"has","text":"
has(key: str) -> bool\n

Check whether payload contains key.

Parameters:

Name Type Description Default key str

Dot-path key.

required

Returns:

Name Type Description bool bool

Existence of the key.

"},{"location":"dagpipe/#dagpipe.State.lineage","title":"lineage","text":"
lineage() -> tuple[State, ...]\n

Return lineage from root to this State.

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Ordered execution lineage (root first).

"},{"location":"dagpipe/#dagpipe.StepResult","title":"StepResult","text":"
StepResult(\n    index: int,\n    node_id: str,\n    states: tuple[State, ...],\n    completed: bool,\n)\n

A single checkpointed step within an async/resumable engine run.

Attributes:

Name Type Description index int

Ordinal index of the step.

node_id str

Identifier of the node associated with this step.

states tuple[State, ...]

States produced by running this step.

completed bool

Whether this step succeeded (vs. paused/interrupted).

Initialise StepResult.

Parameters:

Name Type Description Default index int

Ordinal index of the step.

required node_id str

Identifier of the node associated with this step.

required states tuple[State, ...]

States produced by running this step.

required completed bool

Whether this step succeeded (vs. paused/interrupted).

required"},{"location":"dagpipe/#dagpipe.StepResult-functions","title":"Functions","text":""},{"location":"dagpipe/#dagpipe-functions","title":"Functions","text":""},{"location":"dagpipe/#dagpipe.load_pipeline","title":"load_pipeline","text":"
load_pipeline(path: str) -> Pipeline\n

Load pipeline from YAML file.

Parameters:

Name Type Description Default path str

Path to YAML configuration file.

required

Returns:

Name Type Description Pipeline Pipeline

Executable pipeline instance.

Notes

Responsibilities:

- Loads YAML configuration and builds schema.\n- Creates `State` subclass and loads `Node` instances.\n- Builds `Graph` topology and initializes `Engine`.\n
"},{"location":"dagpipe/engine/","title":"Engine","text":""},{"location":"dagpipe/engine/#dagpipe.engine","title":"dagpipe.engine","text":""},{"location":"dagpipe/engine/#dagpipe.engine--summary","title":"Summary","text":"

Execution engine responsible for running pipelines and graphs.

The Engine executes Node objects and propagates immutable State instances through either a linear sequence or a directed acyclic graph (Graph). It orchestrates execution order, branching, and state propagation.

"},{"location":"dagpipe/engine/#dagpipe.engine--guarantees","title":"Guarantees","text":""},{"location":"dagpipe/engine/#dagpipe.engine-classes","title":"Classes","text":""},{"location":"dagpipe/engine/#dagpipe.engine.Engine","title":"Engine","text":"
Engine(\n    nodes_or_graph: Sequence[Node] | Graph,\n    *,\n    on_step: StepHook | None = None\n)\n

Execution engine responsible for running pipeline logic.

Notes

Responsibilities:

- Accepts either a linear sequence of `Node` objects or a `Graph`\n  defining execution topology.\n- Propagates immutable `State` objects through `Node` objects and\n  collects terminal states.\n- Supports synchronous (`run`) and asynchronous (`run_async`)\n  execution, dispatching per-node.\n- Supports step-wise / resumable execution and progress hooks.\n

Guarantees:

- Never mutates `State`, `Node`, or `Graph` instances.\n- `State` objects are never modified in place; each branch produces\n  independent instances.\n- Execution order is deterministic and follows graph or pipeline topology.\n- Thread-safe for concurrent execution.\n
"},{"location":"dagpipe/engine/#dagpipe.engine.Engine-attributes","title":"Attributes","text":""},{"location":"dagpipe/engine/#dagpipe.engine.Engine.nodes","title":"nodes property","text":"
nodes: tuple[Node, ...]\n

Return nodes managed by this engine.

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Ordered sequence in linear mode or all nodes in graph mode.

"},{"location":"dagpipe/engine/#dagpipe.engine.Engine-functions","title":"Functions","text":""},{"location":"dagpipe/engine/#dagpipe.engine.Engine.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return the canonical string representation of the object.

Returns:

Name Type Description str str

Representation that uniquely identifies the object and its configuration.

"},{"location":"dagpipe/engine/#dagpipe.engine.Engine.run","title":"run","text":"
run(root: State) -> list[State]\n

Execute the pipeline starting from a root State.

Parameters:

Name Type Description Default root State

Initial execution state.

required

Returns:

Type Description list[State]

list[State]: Terminal execution states produced by the pipeline.

Raises:

Type Description TypeError

If root is not a State instance.

RuntimeError

If the engine execution mode is invalid.

Notes

Responsibilities:

- Selects execution mode, propagates state through nodes, creates\n  new instances for branches, and collects terminal states.\n
"},{"location":"dagpipe/engine/#dagpipe.engine.Engine.run_async","title":"run_async async","text":"
run_async(root: State) -> list[State]\n

Execute the pipeline starting from root, dispatching sync vs async nodes.

Parameters:

Name Type Description Default root State

Initial execution state.

required

Returns:

Type Description list[State]

list[State]: Terminal execution states produced by the pipeline.

Notes

Each node is executed with Node.run when synchronous and AsyncNode.run_async when asynchronous. Linear and graph topologies are both supported.

"},{"location":"dagpipe/engine/#dagpipe.engine.Engine.run_steps","title":"run_steps","text":"
run_steps(\n    root: State,\n    *,\n    resume_from: int | None = None,\n    on_step: StepHook | None = None\n) -> Iterator[StepResult]\n

Execute the pipeline step-by-step, yielding one StepResult per step.

Parameters:

Name Type Description Default root State

Initial execution state.

required resume_from int | None

Skip steps at index < resume_from (for resume-after-partial). Steps are 0-indexed.

None on_step StepHook | None

Callback (step, status, message) invoked per step; falls back to the engine-level hook when unset.

None

Yields:

Name Type Description StepResult StepResult

One per executed node/step, carrying the produced states.

Notes

This is a synchronous, generator-based checkpoint interface compatible with the imperative resume-by-step behaviour of the legacy orchestrator. Use run_steps_async for async nodes.

"},{"location":"dagpipe/engine/#dagpipe.engine.Engine.run_steps_async","title":"run_steps_async async","text":"
run_steps_async(\n    root: State,\n    *,\n    resume_from: int | None = None,\n    on_step: AsyncStepHook | None = None\n) -> AsyncIterator[StepResult]\n

Async variant of run_steps supporting AsyncNode execution.

Parameters:

Name Type Description Default root State

Initial execution state.

required resume_from int | None

Skip steps at index < resume_from.

None on_step AsyncStepHook | None

Async callback (step, status, message) invoked per step.

None

Yields:

Name Type Description StepResult AsyncIterator[StepResult]

One per executed node/step.

"},{"location":"dagpipe/engine/#dagpipe.engine.ProgressMessage","title":"ProgressMessage","text":"
ProgressMessage(\n    *,\n    lines: int | None = None,\n    blocks: int | None = None,\n    count: int | None = None,\n    unit: str | None = None,\n    raw_ocr_line: str | None = None,\n    error: str | None = None,\n    step: str = \"\",\n    status: str = \"\"\n)\n

Lightweight progress payload emitted by engine step hooks.

Mirrors the imperative ProgressMessage used by the legacy orchestrator so callers can surface counts/lines/errors without coupling the engine to pydantic.

"},{"location":"dagpipe/engine/#dagpipe.engine.StepResult","title":"StepResult","text":"
StepResult(\n    index: int,\n    node_id: str,\n    states: tuple[State, ...],\n    completed: bool,\n)\n

A single checkpointed step within an async/resumable engine run.

Attributes:

Name Type Description index int

Ordinal index of the step.

node_id str

Identifier of the node associated with this step.

states tuple[State, ...]

States produced by running this step.

completed bool

Whether this step succeeded (vs. paused/interrupted).

Initialise StepResult.

Parameters:

Name Type Description Default index int

Ordinal index of the step.

required node_id str

Identifier of the node associated with this step.

required states tuple[State, ...]

States produced by running this step.

required completed bool

Whether this step succeeded (vs. paused/interrupted).

required"},{"location":"dagpipe/engine/#dagpipe.engine.StepResult-functions","title":"Functions","text":""},{"location":"dagpipe/graph/","title":"Graph","text":""},{"location":"dagpipe/graph/#dagpipe.graph","title":"dagpipe.graph","text":""},{"location":"dagpipe/graph/#dagpipe.graph--summary","title":"Summary","text":"

Defines DAG structure connecting nodes.

A Graph describes execution topology only. It does not execute nodes or manage State. Execution is handled by an Engine.

"},{"location":"dagpipe/graph/#dagpipe.graph--responsibilities","title":"Responsibilities","text":""},{"location":"dagpipe/graph/#dagpipe.graph-classes","title":"Classes","text":""},{"location":"dagpipe/graph/#dagpipe.graph.Graph","title":"Graph","text":"
Graph()\n

Directed Acyclic Graph defining execution topology of Node objects.

Notes

Responsibilities:

- Stores node connectivity and validates that the topology remains acyclic.\n- Structure determines how `State` flows between nodes during execution.\n

Guarantees:

- Topology is acyclic. Node relationships remain consistent.\n- Thread-safe for concurrent reads after construction.\n

Create an empty Graph.

"},{"location":"dagpipe/graph/#dagpipe.graph.Graph--initializes-node-registry-and-edge-mappings","title":"Initializes node registry and edge mappings.","text":""},{"location":"dagpipe/graph/#dagpipe.graph.Graph-functions","title":"Functions","text":""},{"location":"dagpipe/graph/#dagpipe.graph.Graph.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"dagpipe/graph/#dagpipe.graph.Graph.__repr__--returns","title":"Returns","text":"

str

"},{"location":"dagpipe/graph/#dagpipe.graph.Graph.add_edge","title":"add_edge","text":"
add_edge(src: Node, dst: Node) -> None\n

Add a directed edge from src to dst.

Parameters:

Name Type Description Default src Node

Source node.

required dst Node

Destination node.

required

Raises:

Type Description TypeError

If src or dst is not a Node.

ValueError

If the edge would create a cycle or if src and dst are common.

Notes "},{"location":"dagpipe/graph/#dagpipe.graph.Graph.add_root","title":"add_root","text":"
add_root(node: Node) -> None\n

Add a root node with no parents.

Parameters:

Name Type Description Default node Node

Node to add as a root.

required

Raises:

Type Description TypeError

If node is not a Node instance.

"},{"location":"dagpipe/graph/#dagpipe.graph.Graph.children","title":"children","text":"
children(node: Node) -> tuple[Node, ...]\n

Return child nodes of a node.

Parameters:

Name Type Description Default node Node

Node to query.

required

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Outgoing neighbors.

"},{"location":"dagpipe/graph/#dagpipe.graph.Graph.nodes","title":"nodes","text":"
nodes() -> tuple[Node, ...]\n

Return all nodes in the graph.

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: All registered nodes.

"},{"location":"dagpipe/graph/#dagpipe.graph.Graph.parents","title":"parents","text":"
parents(node: Node) -> tuple[Node, ...]\n

Return parent nodes of a node.

Parameters:

Name Type Description Default node Node

Node to query.

required

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Incoming neighbors.

"},{"location":"dagpipe/graph/#dagpipe.graph.Graph.roots","title":"roots","text":"
roots() -> tuple[Node, ...]\n

Return root nodes (nodes with no incoming edges).

Returns:

Type Description tuple[Node, ...]

tuple[Node, ...]: Entry point nodes.

"},{"location":"dagpipe/node/","title":"Node","text":""},{"location":"dagpipe/node/#dagpipe.node","title":"dagpipe.node","text":""},{"location":"dagpipe/node/#dagpipe.node--summary","title":"Summary","text":"

Defines the Node abstraction used by dagpipe.

A node represents a single unit of pipeline execution logic. It consumes one State and produces zero, one, or many new State objects.

Nodes are connected using a Graph and executed by an Engine.

"},{"location":"dagpipe/node/#dagpipe.node--design-principles","title":"Design principles","text":""},{"location":"dagpipe/node/#dagpipe.node-classes","title":"Classes","text":""},{"location":"dagpipe/node/#dagpipe.node.AsyncNode","title":"AsyncNode","text":"

Bases: Node

Base class for nodes whose execution is asynchronous.

Subclasses implement resolve_async (an async generator yielding derived State objects). The engine dispatches to resolve_async when running an async traversal (see Engine.run_async).

Sync-only engines (and the base Node.run) treat an AsyncNode as a no-op consumer: calling run on an AsyncNode returns no states, signalling that an async engine is required.

"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode-functions","title":"Functions","text":""},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.__hash__","title":"__hash__","text":"
__hash__() -> int\n

Return stable hash based on node ID.

"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.__hash__--returns","title":"Returns","text":"

int

"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.__repr__--returns","title":"Returns","text":"

str

"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.__str__","title":"__str__","text":"
__str__() -> str\n

Return display representation.

"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.__str__--returns","title":"Returns","text":"

str

"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.clean_id_and_name","title":"clean_id_and_name classmethod","text":"
clean_id_and_name() -> None\n

Normalize and validate node ID and display name.

Raises:

Type Description TypeError

If ID is not a string.

ValueError

If ID format is invalid.

Notes

Guarantees:

- Generates ID from module and class name if missing.\n- Validates ID format.\n- Generates human-readable name if missing.\n
"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.fork","title":"fork","text":"
fork(\n    state: State,\n    *,\n    payload_update: Any = None,\n    confidence_delta: float = 0.0,\n    metadata_update: Any = None\n) -> State\n

Create a child State attributed to this node.

Parameters:

Name Type Description Default state State

Parent execution state.

required payload_update Any

Dot-path payload updates.

None confidence_delta float

Confidence adjustment.

0.0 metadata_update Any

Metadata updates.

None

Returns:

Name Type Description State State

New child execution state.

Notes

Responsibilities:

- Convenience wrapper around `State.fork()` that automatically\n  records this node's ID in state history.\n
"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.is_async","title":"is_async","text":"
is_async() -> bool\n

Return whether this node executes asynchronously.

"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.node_id_to_name","title":"node_id_to_name staticmethod","text":"
node_id_to_name(node_id: str) -> str\n

Convert a dotted snake_case node ID into a human-readable name.

Parameters:

Name Type Description Default node_id str

Unique node identifier (e.g., 'entity.resolve.numeric_merchant').

required

Returns:

Name Type Description str str

Human-readable display name (e.g., 'Entity \u203a Resolve \u203a Numeric Merchant').

"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.resolve_async","title":"resolve_async async","text":"
resolve_async(state: State) -> Iterable[State]\n

Execute node logic asynchronously.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description Iterable[State]

Iterable[State]: Derived execution state(s).

Notes

Subclasses implement this. Must not mutate the input state. Should use fork() to create child states.

"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.run","title":"run","text":"
run(state: State) -> tuple[State, ...]\n

Execute this node on a State.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Derived execution states.

Raises:

Type Description TypeError

If resolve() yields a non-State object.

"},{"location":"dagpipe/node/#dagpipe.node.AsyncNode.run_async","title":"run_async async","text":"
run_async(state: State) -> tuple[State, ...]\n

Execute this node asynchronously on a state, validating outputs.

"},{"location":"dagpipe/node/#dagpipe.node.Node","title":"Node","text":"

Bases: ABC

Base class for all dagpipe execution nodes.

Attributes:

Name Type Description id str

Unique identifier of the node (snake_case dotted format).

name str

Human-readable display name.

Notes

Responsibilities:

- Represents a deterministic unit of execution in the pipeline graph.\n- Consumes one `State` and produces zero, one, or many derived states.\n- Defines execution logic and enables branching, filtering, and transformation.\n

Guarantees:

- Nodes must never mutate the input `State`.\n- Instances are singletons per subclass and reused across executions.\n
"},{"location":"dagpipe/node/#dagpipe.node.Node-functions","title":"Functions","text":""},{"location":"dagpipe/node/#dagpipe.node.Node.__hash__","title":"__hash__","text":"
__hash__() -> int\n

Return stable hash based on node ID.

"},{"location":"dagpipe/node/#dagpipe.node.Node.__hash__--returns","title":"Returns","text":"

int

"},{"location":"dagpipe/node/#dagpipe.node.Node.__new__","title":"__new__","text":"
__new__(*args: Any, **kwargs: Any) -> Node\n

Create or reuse a Node instance.

Stateless subclasses (no parameterized __init__) share one singleton instance per class \u2014 matching the original dagpipe behaviour underpinning set_registry-style configuration. Subclasses that declare an __init__ requiring instance-state arguments get a fresh instance per construction so pipeline builders can inject per-run dependencies.

"},{"location":"dagpipe/node/#dagpipe.node.Node.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Return debug representation.

"},{"location":"dagpipe/node/#dagpipe.node.Node.__repr__--returns","title":"Returns","text":"

str

"},{"location":"dagpipe/node/#dagpipe.node.Node.__str__","title":"__str__","text":"
__str__() -> str\n

Return display representation.

"},{"location":"dagpipe/node/#dagpipe.node.Node.__str__--returns","title":"Returns","text":"

str

"},{"location":"dagpipe/node/#dagpipe.node.Node.clean_id_and_name","title":"clean_id_and_name classmethod","text":"
clean_id_and_name() -> None\n

Normalize and validate node ID and display name.

Raises:

Type Description TypeError

If ID is not a string.

ValueError

If ID format is invalid.

Notes

Guarantees:

- Generates ID from module and class name if missing.\n- Validates ID format.\n- Generates human-readable name if missing.\n
"},{"location":"dagpipe/node/#dagpipe.node.Node.fork","title":"fork","text":"
fork(\n    state: State,\n    *,\n    payload_update: Any = None,\n    confidence_delta: float = 0.0,\n    metadata_update: Any = None\n) -> State\n

Create a child State attributed to this node.

Parameters:

Name Type Description Default state State

Parent execution state.

required payload_update Any

Dot-path payload updates.

None confidence_delta float

Confidence adjustment.

0.0 metadata_update Any

Metadata updates.

None

Returns:

Name Type Description State State

New child execution state.

Notes

Responsibilities:

- Convenience wrapper around `State.fork()` that automatically\n  records this node's ID in state history.\n
"},{"location":"dagpipe/node/#dagpipe.node.Node.is_async","title":"is_async","text":"
is_async() -> bool\n

Return whether this node executes asynchronously.

"},{"location":"dagpipe/node/#dagpipe.node.Node.node_id_to_name","title":"node_id_to_name staticmethod","text":"
node_id_to_name(node_id: str) -> str\n

Convert a dotted snake_case node ID into a human-readable name.

Parameters:

Name Type Description Default node_id str

Unique node identifier (e.g., 'entity.resolve.numeric_merchant').

required

Returns:

Name Type Description str str

Human-readable display name (e.g., 'Entity \u203a Resolve \u203a Numeric Merchant').

"},{"location":"dagpipe/node/#dagpipe.node.Node.resolve","title":"resolve abstractmethod","text":"
resolve(state: State) -> Iterable[State]\n

Execute node logic.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description Iterable[State]

Iterable[State]: Derived execution state(s).

Notes

Responsibilities:

- Subclasses implement specific resolution behavior.\n- Must not mutate input state.\n- Should use `fork()` to create child states.\n- May yield zero states to terminate a branch.\n
"},{"location":"dagpipe/node/#dagpipe.node.Node.run","title":"run","text":"
run(state: State) -> tuple[State, ...]\n

Execute this node on a State.

Parameters:

Name Type Description Default state State

Input execution state.

required

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Derived execution states.

Raises:

Type Description TypeError

If resolve() yields a non-State object.

"},{"location":"dagpipe/state/","title":"State","text":""},{"location":"dagpipe/state/#dagpipe.state","title":"dagpipe.state","text":""},{"location":"dagpipe/state/#dagpipe.state--summary","title":"Summary","text":"

Defines the core State object used by dagpipe.

The State represents a single point in pipeline execution. It contains arbitrary data and metadata and is designed to be immutable. Instead of modifying an existing state, nodes create new child states via fork().

"},{"location":"dagpipe/state/#dagpipe.state--design-principles","title":"Design principles","text":""},{"location":"dagpipe/state/#dagpipe.state-classes","title":"Classes","text":""},{"location":"dagpipe/state/#dagpipe.state.Payload","title":"Payload dataclass","text":"
Payload(_data: Mapping[str, Any])\n

Immutable hierarchical container with dot-path access.

Attributes:

Name Type Description _data Mapping[str, Any]

Immutable hierarchical data structure.

Notes

Responsibilities:

- Stores execution data used by `State`.\n- Supports efficient atomic updates without modifying existing instances.\n- `Payload` instances are fully thread-safe due to immutability.\n
"},{"location":"dagpipe/state/#dagpipe.state.Payload-functions","title":"Functions","text":""},{"location":"dagpipe/state/#dagpipe.state.Payload.as_dict","title":"as_dict","text":"
as_dict() -> Mapping[str, Any]\n

Return underlying mapping.

Returns:

Type Description Mapping[str, Any]

Mapping[str, Any]: Read-only view of the underlying data.

"},{"location":"dagpipe/state/#dagpipe.state.Payload.get","title":"get","text":"
get(path: str, default: Any = None) -> Any\n

Retrieve value using dot-path.

Parameters:

Name Type Description Default path str

Dot-separated path to the value.

required default Any

Default value if path doesn't exist.

None

Returns:

Name Type Description Any Any

The retrieved value or default.

"},{"location":"dagpipe/state/#dagpipe.state.Payload.has","title":"has","text":"
has(path: str) -> bool\n

Return True if path exists.

Parameters:

Name Type Description Default path str

Dot-separated path to check.

required

Returns:

Name Type Description bool bool

Existence of the path.

"},{"location":"dagpipe/state/#dagpipe.state.Payload.iter_paths","title":"iter_paths classmethod","text":"
iter_paths(\n    data: Mapping[str, Any], prefix: str = \"\"\n) -> Iterable[str]\n

Recursively yield dot-paths for all leaf nodes.

Parameters:

Name Type Description Default data Mapping[str, Any]

The mapping to iterate over.

required prefix str

Current path prefix.

''

Yields:

Name Type Description str Iterable[str]

Dot-path for each leaf node.

"},{"location":"dagpipe/state/#dagpipe.state.Payload.keys","title":"keys","text":"
keys() -> Iterable[str]\n

Return top-level keys.

Returns:

Type Description Iterable[str]

Iterable[str]: Iterator over top-level keys.

"},{"location":"dagpipe/state/#dagpipe.state.Payload.update","title":"update","text":"
update(updates: Mapping[str, Any]) -> Payload\n

Create a new Payload with dot-path updates applied.

Parameters:

Name Type Description Default updates Mapping[str, Any]

Dot-path to value mapping.

required

Returns:

Name Type Description Payload Payload

New immutable payload instance with updates.

Notes

Guarantees:

- Preserves existing data by copying only modified branches.\n- Returns a new immutable `Payload`.\n
"},{"location":"dagpipe/state/#dagpipe.state.Schema","title":"Schema dataclass","text":"
Schema(tree: Mapping[str, SchemaNode])\n

Immutable hierarchical schema defining allowed payload structure.

Attributes:

Name Type Description tree Mapping[str, SchemaNode]

Hierarchical schema definition.

Notes

Responsibilities:

- Validates `State` payloads and updates.\n- Reusable across all `State` instances.\n- Fully thread-safe due to immutability.\n
"},{"location":"dagpipe/state/#dagpipe.state.Schema-functions","title":"Functions","text":""},{"location":"dagpipe/state/#dagpipe.state.Schema.validate_payload","title":"validate_payload","text":"
validate_payload(payload: Payload) -> None\n

Validate complete payload structure.

Parameters:

Name Type Description Default payload Payload

Payload to validate.

required

Raises:

Type Description SchemaError

If payload violates schema.

"},{"location":"dagpipe/state/#dagpipe.state.Schema.validate_update","title":"validate_update","text":"
validate_update(updates: Mapping[str, Any]) -> None\n

Validate payload update paths.

Parameters:

Name Type Description Default updates Mapping[str, Any]

Dot-path updates to validate.

required

Raises:

Type Description SchemaError

If any path is invalid according to the schema.

"},{"location":"dagpipe/state/#dagpipe.state.SchemaError","title":"SchemaError","text":"

Bases: Exception

Raised when payload data violates the declared schema.

"},{"location":"dagpipe/state/#dagpipe.state.SchemaError--indicates-invalid-structure-invalid-path-or-invalid-type","title":"Indicates invalid structure, invalid path, or invalid type.","text":""},{"location":"dagpipe/state/#dagpipe.state.State","title":"State dataclass","text":"
State(\n    payload: Payload,\n    confidence: float = 1.0,\n    parent: State | None = None,\n    depth: int = 0,\n    history: tuple[str, ...] = tuple(),\n    metadata: dict[str, Any] = dict(),\n)\n

Immutable execution state propagated through dagpipe pipeline.

Attributes:

Name Type Description payload Payload

Execution data container.

schema ClassVar[Schema]

Payload validation schema.

confidence float

Execution confidence score.

parent Optional[State]

Parent state reference.

depth int

Execution depth.

history Tuple[str, ...]

Ordered node execution lineage.

metadata Dict[str, Any]

Execution metadata.

Notes

Responsibilities:

- Represents a complete execution snapshot at a specific point in\n  pipeline traversal.\n- Fundamental unit of execution in `dagpipe`.\n- Fully thread-safe due to immutability.\n
"},{"location":"dagpipe/state/#dagpipe.state.State-functions","title":"Functions","text":""},{"location":"dagpipe/state/#dagpipe.state.State.__repr__","title":"__repr__","text":"
__repr__() -> str\n

Concise debug representation.

Avoids printing full data for large states.

"},{"location":"dagpipe/state/#dagpipe.state.State.fork","title":"fork","text":"
fork(\n    *,\n    payload_update: Mapping[str, Any] | None = None,\n    confidence_delta: float = 0.0,\n    node_id: str | None = None,\n    metadata_update: Mapping[str, Any] | None = None\n) -> State\n

Create a new child State derived from this state.

Parameters:

Name Type Description Default payload_update Mapping[str, Any] | None

Dot-path updates applied to the payload.

None confidence_delta float

Adjustment applied to current confidence.

0.0 node_id str | None

Identifier of the node creating this state.

None metadata_update Mapping[str, Any] | None

Updates merged into state metadata.

None

Returns:

Name Type Description State State

A new immutable State instance.

Notes

Guarantees:

- This is the only supported mechanism for modifying execution data.\n- Validates payload updates, preserves lineage, increments depth,\n  and appends to history.\n
"},{"location":"dagpipe/state/#dagpipe.state.State.get","title":"get","text":"
get(key: str, default: Any = None) -> Any\n

Retrieve payload value.

Parameters:

Name Type Description Default key str

Dot-path key.

required default Any

Fallback value.

None

Returns:

Name Type Description Any Any

Stored value or default.

"},{"location":"dagpipe/state/#dagpipe.state.State.has","title":"has","text":"
has(key: str) -> bool\n

Check whether payload contains key.

Parameters:

Name Type Description Default key str

Dot-path key.

required

Returns:

Name Type Description bool bool

Existence of the key.

"},{"location":"dagpipe/state/#dagpipe.state.State.lineage","title":"lineage","text":"
lineage() -> tuple[State, ...]\n

Return lineage from root to this State.

Returns:

Type Description tuple[State, ...]

tuple[State, ...]: Ordered execution lineage (root first).

"},{"location":"dagpipe/yaml_loader/","title":"Yaml Loader","text":""},{"location":"dagpipe/yaml_loader/#dagpipe.yaml_loader","title":"dagpipe.yaml_loader","text":""},{"location":"dagpipe/yaml_loader/#dagpipe.yaml_loader--summary","title":"Summary","text":"

Loads dagpipe pipelines from YAML configuration.

Creates fully configured pipeline objects from declarative YAML definitions, including Schema, State subclasses, Node instances, Graph topology, and initial payloads.

"},{"location":"dagpipe/yaml_loader/#dagpipe.yaml_loader-classes","title":"Classes","text":""},{"location":"dagpipe/yaml_loader/#dagpipe.yaml_loader.Pipeline","title":"Pipeline dataclass","text":"
Pipeline(\n    engine: Engine,\n    state_cls: type[State],\n    initial_payload: Payload,\n)\n

Executable pipeline created from YAML configuration.

Attributes:

Name Type Description engine Engine

Execution engine responsible for running the pipeline.

state_cls Type[State]

Dynamically created State subclass with configured schema.

initial_payload Payload

Default payload used when execution begins.

Notes

Responsibilities:

- Encapsulates engine, state type, and initial payload.\n- Provides a simplified interface for executing configured pipelines.\n- Safe for concurrent execution if underlying nodes are thread-safe.\n
"},{"location":"dagpipe/yaml_loader/#dagpipe.yaml_loader.Pipeline-functions","title":"Functions","text":""},{"location":"dagpipe/yaml_loader/#dagpipe.yaml_loader.Pipeline.run","title":"run","text":"
run(\n    payload_override: Mapping[str, Any] | None = None,\n) -> list[State]\n

Execute the pipeline.

Parameters:

Name Type Description Default payload_override Mapping[str, Any] | None

Payload values overriding initial payload.

None

Returns:

Type Description list[State]

list[State]: Terminal execution states.

Notes

Responsibilities:

- Merges override payload with initial payload.\n- Creates root `State` and executes engine.\n
"},{"location":"dagpipe/yaml_loader/#dagpipe.yaml_loader-functions","title":"Functions","text":""},{"location":"dagpipe/yaml_loader/#dagpipe.yaml_loader.load_pipeline","title":"load_pipeline","text":"
load_pipeline(path: str) -> Pipeline\n

Load pipeline from YAML file.

Parameters:

Name Type Description Default path str

Path to YAML configuration file.

required

Returns:

Name Type Description Pipeline Pipeline

Executable pipeline instance.

Notes

Responsibilities:

- Loads YAML configuration and builds schema.\n- Creates `State` subclass and loads `Node` instances.\n- Builds `Graph` topology and initializes `Engine`.\n
"}]} \ No newline at end of file diff --git a/dagpipe/lib/sitemap.xml.gz b/dagpipe/lib/sitemap.xml.gz index e496a0d39e2d89445c665e87a8c93f68e7f55814..01465253604ef76b3b2b517f1166919160e58317 100644 GIT binary patch delta 13 Ucmb=gXP58h;ArStF_FCj03F=~f&c&j delta 13 Ucmb=gXP58h;AnWhbRv5N03V_R+yDRo diff --git a/hexa/lib/design/index.html b/hexa/lib/design/index.html deleted file mode 100644 index a2db63c..0000000 --- a/hexa/lib/design/index.html +++ /dev/null @@ -1,1353 +0,0 @@ - - - - - - - - - - - - - - - - - - - hexa — Design (extraction pipeline, Axis/Icici) - hexa - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - Skip to content - - -
-
- -
- - - - - - -
- - -
- -
- - - - - - -
-
- - - -
-
-
- - - - - -
-
-
- - - -
-
-
- - - -
-
-
- - - -
-
- - - - - -

hexa — Design (extraction pipeline, Axis/Icici)

-
-

Note: The extraction pipeline discussed below is a sample/example illustrating the hexa composition pattern.

-
-

This document is the concrete counterpart to philosophy.md: where -the files are, what the tree looks like for both banks, and how a future container -instantiates it. Self-sufficient for a fresh agent.

-

File map (samples/extraction_pipeline/)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
FileLayerRole
extraction_pipeline_abc.pyABC contractsAll port ABCs + ExtractionPipeline root. Declares attributes, the ExtractionSource dataclass, port slots, @abstractmethods.
extraction_pipeline_impl.pyGeneric Impl*Concrete default implementations of every port; ImplExtractionPipeline root; clean_num helper.
extraction_pipeline_ambiguity.pyShared logicPorted ResolutionContext, AmbiguityHandler, AmountBalanceNotFound, validate_signs — used by ImplAmountBalancePort.
extraction_pipeline_banks.pyComposition rootAxisExtractionPipeline(ImplExtractionPipeline) + IciciExtractionPipeline (each re-pins txn_dicts).
banks/axis/pdf.pyBank specializationAxisNumberPort, AxisDescPort, AxisAmountBalancePort, AxisTransactionParserPort, AxisTxnDictsPort.
banks/icici/pdf.pyBank specializationIcici* mirror of the above + Icici-only adjust_balance/missing_number_candidates.
extraction_pipeline.yamlContract specDeclarative spec of the port tree; interchangeable with _abc.py via the optional utilities.
-

The port tree (ABC contract)

-
 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
-10
-11
-12
-13
-14
-15
-16
ExtractionPipeline
-├── source / pipeline / trust_fallback          [attributes; source is a nested struct]
-├── ingest      → IngestPort                    (source → content)
-├── raw_lines   → RawLinesPort                  (content → raw_lines)
-├── txn_blocks  → TxnBlocksPort                 (raw_lines → txn_blocks)
-│   ├── bucket_finder → BucketFinderPort
-│   └── splitter      → SplitterPort
-├── txn_dicts   → TxnDictsPort                  (txn_blocks → txn_dicts)
-│   └── parser  → TransactionParserPort
-│       ├── number         → NumberPort
-│       ├── date           → DatePort
-│       ├── desc           → DescPort
-│       └── amount_balance → AmountBalancePort
-│           ├── number    → NumberPort
-│           └── ambiguity → AmbiguityPort
-└── raw_expense → RawExpensePort                (txn_dicts → list[RawExpense], terminal)
-
-

Generic default (Impl*)

-

ImplExtractionPipeline pins every stage slot to a concrete Impl* port:

-
 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
-10
-11
-12
-13
-14
-15
ImplExtractionPipeline
-├── ingest:      ImplIngestPort
-├── raw_lines:   ImplRawLinesPort
-├── txn_blocks:  ImplTxnBlocksPort
-│   ├── bucket_finder: ImplBucketFinderPort
-│   └── splitter:      ImplSplitterPort
-├── txn_dicts:   ImplTxnDictsPort
-│   └── parser:  ImplTransactionParserPort
-│       ├── number:        ImplNumberPort
-│       ├── date:          ImplDatePort
-│       ├── desc:          ImplDescPort
-│       └── amount_balance: ImplAmountBalancePort
-│           ├── number:    ImplNumberPort
-│           └── ambiguity: ImplAmbiguityPort
-└── raw_expense: ImplRawExpensePort
-
-

Bank variation (only what differs)

-

The root class re-pins only txn_dicts. Everything else inherits Impl*.

-

Axis (banks/axis/pdf.py)

-
1
-2
class AxisExtractionPipeline(ImplExtractionPipeline):
-    txn_dicts: AxisTxnDictsPort
-
-
1
-2
-3
-4
-5
-6
AxisTxnDictsPort(ImplTxnDictsPort)
-  └─ parser: AxisTransactionParserPort(ImplTransactionParserPort)
-      ├─ number:  AxisNumberPort          # NUMBER_RE, all clean numbers
-      ├─ desc:    AxisDescPort            # STARTTERS = [upi/, imps/, neft-, neft/, ach/, ach-, 2a/]
-      └─ amount_balance: AxisAmountBalancePort(ImplAmountBalancePort)
-          └─ number: AxisNumberPort       # re-pin back to axis number port
-
-

Notes: -- date is not redeclared on AxisTransactionParserPort — it inherits - ImplDatePort (correct already, adds no behavior). -- amount_balance.ambiguity is not redeclared — inherits ImplAmbiguityPort.

-

Icici (banks/icici/pdf.py)

-
1
-2
class IciciExtractionPipeline(ImplExtractionPipeline):
-    txn_dicts: IciciTxnDictsPort
-
-
1
-2
-3
-4
-5
-6
-7
-8
IciciTxnDictsPort(ImplTxnDictsPort)
-  └─ parser: IciciTransactionParserPort(ImplTransactionParserPort)
-      ├─ number:  IciciNumberPort         # NUMBER_RE, returns decimals[-2:] (last 2)
-      ├─ desc:    IciciDescPort           # STARTTERS + fix_ocr (UPl→UPI, ..., ICIC]→ICICI)
-      └─ amount_balance: IciciAmountBalancePort(ImplAmountBalancePort)
-          ├─ number: IciciNumberPort      # re-pin to icici number port
-          ├─ adjust_balance:              # neg-markers [(-), (-}, {-}, {-) , ()] → -abs(balance)
-          └─ missing_number_candidates:   # Icici-only one-number hook
-
-

Redundancy rule (applied in both banks)

-

Only redeclare a slot you are actually changing from Impl*. If a slot's -behavior is already correct from the parent Impl*, omit it. This keeps the -diff between a bank and Impl* minimal and unambiguous.

-
-

date: ImplDatePort was previously redeclared redundantly and has been removed — -it added mental load on the extender for zero behavior change. (Fidelity audit: -it is still concrete ImplDatePort in ImplTransactionParserPort, so behavior is -unchanged.)

-
-

Container (build(cls) in hexa/container.py)

-

The annotations are metadata; a small reflection builder turns a root class into -a wired instance:

-
1
build(AxisExtractionPipeline)  →  wired AxisExtractionPipeline
-
-

Algorithm (build(cls)):

-
    -
  1. Walk cls.__annotations__, merged across the MRO so inherited slot annotations - are visible (a bank subclass only adds/overrides a slot; the rest come from - Impl*).
    2. -
  2. For each port slot whose annotated type is a concrete Impl*/bank class - (not an ABC, not a builtin), instantiate it.
    3. -
  3. Recurse into that child — repeat until leaves (classes that declare no further - port slots).
    4. -
  4. setattr(parent, slot, child) to wire each slot onto the parent instance.
    5. -
  5. Apply config defaults from the class-level annotated attributes.
    6. -
-

Selection is fully static: the annotated type is the chosen implementation. -The container holds no per-bank branch logic — Axis vs Icici is decided purely by -which root class you call build on.

-

Control flow at runtime (target)

-

A leaf-stage port exposes one entry method; stages chain data through the shared -run(source) signature on the root:

-
1
-2
-3
-4
-5
-6
run(source)
-  ingest.materialize(source)            → content
-  raw_lines.extract(content)            → raw_lines
-  txn_blocks.extract(raw_lines)         → {opening_balance: [blocks]}
-  txn_dicts.extract(txn_blocks)         → [txn dicts]
-  raw_expense.build(txn_dicts)          → [RawExpense]   (terminal)
-
-

The stage ports map naturally onto dagpipe Nodes (ingest, raw_lines, -txn_blocks, txn_dicts, raw_expense), each node's resolve delegating to its -wired port subtree, connected as a dagpipe Graph and run via Engine. That merge -is a separate, future step.

-

Utilities (optional, hexa-agnostic)

-

The hexa utilities are implemented in hexa/ (the package root) and they are agnostic of the extraction pipeline sample. It ships a set of optional code-gen / validation utilities so users can choose their own workflow — they are never required. A user is free to:

-
    -
  • hand-write an ABC file and never touch YAML, or
    • -
  • start from YAML and generate the ABC file, or
    • -
  • start from an ABC file and emit YAML for it, or
    • -
  • use the ABC/YAML only as a spec (neither generated from the other).
    • -
-

The utilities only ever translate between two interchangeable representations of -the same contract — the .yaml spec and the _abc.py module. They never -generate or manage the Impl*/bank files (those carry behavior and slot choice, -which is a human decision).

-

Shared model

-

All utilities serialize through one neutral model, a PortNode tree. It is NOT -tied to the extraction sample — any port tree fits.

-
1
-2
-3
-4
-5
-6
-7
-8
-9
@dataclass
-class PortNode:
-    name: str                 # slot name, e.g. "parser", "amount_balance"
-    kind: str                 # "leaf" | "port" | "parent"
-    port_cls: str             # ABC class name, e.g. "TransactionParserPort"
-    attributes: dict[str, ConfigValue]  # annotated attributes: name -> type expr,
-                              # or a nested struct dict for grouped values
-    children: list["PortNode"]
-    methods: dict[str, MethodSpec]   # abstract methods
-
-

1. parse_yaml(path) -> PortNode

-

Read a YAML spec into the PortNode tree (YAML → model).

-
    -
  • Parses completely (structural validation only, no type-checking).
    • -
  • Supports the sample grammar (see extraction_pipeline.yaml): a root class whose - inline attributes (scalars and nested structs), port slots, and reserved - methods: recurse into child ports.
    • -
  • Raises ValueError with file/line on structural errors.
    • -
-

2. generate_abc(path_yaml, out_path=None) -> str

-

Generate _abc.py source from a YAML spec (YAML → ABC).

-
    -
  • parse_yaml(...) → write one class <PortCls>(ABC) per PortNode with - attribute annotations (nested structs become synthesized @dataclasses), port-slot - annotations, and @abstractmethod stubs, matching the style of the hand-written - extraction_pipeline_abc.py.
    • -
  • Deterministic (YAML-order) output so it is idempotent / diff-friendly.
    • -
  • out_path optional; always returns the generated source.
    • -
-

3. parse_abc(module_or_path) -> PortNode

-

Reflect over an ABC module into the PortNode tree (ABC → model).

-
    -
  • Reads __annotations__ / @abstractmethod from generic ABC classes (no - sample assumptions) to reconstruct the same PortNode tree.
    • -
-

4. generate_yaml(module_or_path, out_path=None) -> str

-

Generate a .yaml spec from an ABC module (ABC → YAML).

-
    -
  • parse_abc(...) → write the reverse of generate_abc.
    • -
  • Lets a user author the ABC file first and emit YAML for it.
    • -
-

5. check_matches(path_a, path_b) -> bool

-

Verify two representations agree — YAML vs ABC, either direction.

-
    -
  • Builds both PortNode models (parse_yaml + parse_abc) and compares: classes - exist and are ABCs, attributes match, port slots match, @abstractmethod - signatures match (missing keyword-only params treated conservatively).
    • -
  • Returns True only if the whole tree matches; otherwise False (or a diff).
    • -
-

Workflows (user's choice)

-
 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
-10
-11
-12
-13
# YAML-first
-edit spec.yaml
-  → generate_abc(spec.yaml, abc.py)          # regenerate ABCs
-  → hand-write/update Impl* + banks
-  → check_matches(spec.yaml, abc.py) == True # guard
-
-# ABC-first
-edit abc.py
-  → generate_yaml(abc.py, spec.yaml)         # emit YAML spec
-  → check_matches(spec.yaml, abc.py) == True
-
-# hand-written both (no utilities at all)
-edit abc.py + spec.yaml independently        # optional check_matches as a guard
-
-

Impl*/bank files are always hand-extended on top and are never generated.

- - - - - - - - - - - - - -
-
- - - - - -
- - - -
- - - -
-
-
-
- - - - - - - - - - - - \ No newline at end of file diff --git a/hexa/lib/index.html b/hexa/lib/index.html index 2be8263..96be93c 100644 --- a/hexa/lib/index.html +++ b/hexa/lib/index.html @@ -1361,10 +1361,10 @@ tree from annotations:

number: RegexNumberPort # <-- the dependency decision def parse(self, text: str) -> dict: - return {"value": self.number.extract(text)} + return {"value": self.number.extract(text)} pipeline = build(StandardParserPort) -print(pipeline.parse("Value: 42")) +print(pipeline.parse("Value: 42"))

CLI usage

@@ -1528,7 +1528,7 @@ modules via the shared model.

8
PortNode(
     name: str,
     port_cls: str,
-    kind: str = "leaf",
+    kind: str = "leaf",
     attributes: dict[str, ConfigValue] = dict(),
     children: list[PortNode] = list(),
     methods: dict[str, MethodSpec] = dict(),
@@ -1649,7 +1649,7 @@ nested dict of sub-attributes (a struct) for grouped values.

 
 
 
-
1
diff(other: PortNode, path: str = '') -> list[str]
+
1
diff(other: PortNode, path: str = '') -> list[str]
 

 
     

diff --git a/hexa/lib/model/index.html b/hexa/lib/model/index.html
index 85a9215..5dadc7b 100644
--- a/hexa/lib/model/index.html
+++ b/hexa/lib/model/index.html
@@ -1234,7 +1234,7 @@
 8
PortNode(
     name: str,
     port_cls: str,
-    kind: str = "leaf",
+    kind: str = "leaf",
     attributes: dict[str, ConfigValue] = dict(),
     children: list[PortNode] = list(),
     methods: dict[str, MethodSpec] = dict(),
@@ -1355,7 +1355,7 @@ nested dict of sub-attributes (a struct) for grouped values.

 
 
 
-
1
diff(other: PortNode, path: str = '') -> list[str]
+
1
diff(other: PortNode, path: str = '') -> list[str]
 

 
     

diff --git a/hexa/lib/parse_yaml/index.html b/hexa/lib/parse_yaml/index.html
index 9e5ea48..9a2e998 100644
--- a/hexa/lib/parse_yaml/index.html
+++ b/hexa/lib/parse_yaml/index.html
@@ -1042,7 +1042,7 @@ scalar-typed values) is a nested struct of attributes.

 6
```python
 from hexa import parse_yaml
 
-node = parse_yaml("samples/minimal/sample.yaml")
+node = parse_yaml("samples/minimal/sample.yaml")
 print(node)
 ```
 

diff --git a/hexa/lib/search/search_index.json b/hexa/lib/search/search_index.json
index ca0d76b..ead1112 100644
--- a/hexa/lib/search/search_index.json
+++ b/hexa/lib/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"hexa","text":""},{"location":"#hexa","title":"hexa","text":""},{"location":"#hexa--summary","title":"Summary","text":"
Type-declared dependency injection for Python.
 
Hexa is a pattern and utility library for structuring complex hierarchical pipelines. It rests on one core idea: a concrete class's annotated slot type is the dependency decision. Variation is expressed entirely as class annotations \u2014 never as imperative wiring, never as __init__ parameters, never as a composition root that assembles objects by hand.
 
All utilities serialize through a single shared model (PortNode):
 
     
  • parse_yaml / generate_yaml \u2014 between YAML specs and the model
    •  
  • parse_abc / generate_abc \u2014 between ABC Python modules and the model
    •  
  • check_matches \u2014 verify two representations describe the same tree
    •  
  • build \u2014 recursively instantiate a wired pipeline from annotations
    •  
"},{"location":"#hexa--installation","title":"Installation","text":"
Install using pip:
 
pip install hexa\n
"},{"location":"#hexa--quick-start","title":"Quick start","text":"
Define contracts, pin concrete types on the slots, and let build wire the tree from annotations:
 
from hexa import build\n\nclass NumberPort(ABC):\n    @abstractmethod\n    def extract(self, line: str) -> float: ...\n\nclass ParserPort(ABC):\n    number: NumberPort\n\n    @abstractmethod\n    def parse(self, text: str) -> dict: ...\n\nclass RegexNumberPort(NumberPort):\n    def extract(self, line: str) -> float:\n        ...\n\nclass StandardParserPort(ParserPort):\n    number: RegexNumberPort  # <-- the dependency decision\n\n    def parse(self, text: str) -> dict:\n        return {\"value\": self.number.extract(text)}\n\npipeline = build(StandardParserPort)\nprint(pipeline.parse(\"Value: 42\"))\n
"},{"location":"#hexa--cli-usage","title":"CLI usage","text":"
Convert between YAML and ABC representations:
 
hexa parse-yaml spec.yaml\nhexa generate-abc spec.yaml --out spec_abc.py\nhexa parse-abc spec_abc.py\nhexa generate-yaml spec_abc.py --out spec.yaml\nhexa check spec.yaml spec_abc.py\n
"},{"location":"#hexa--core-concepts","title":"Core concepts","text":""},{"location":"#hexa--shared-model","title":"Shared model","text":"
PortNode \u2014 the neutral tree every parser and generator serializes through.
"},{"location":"#hexa--converters","title":"Converters","text":"
Parsers and generators that translate between YAML specs and ABC Python modules via the shared model.
"},{"location":"#hexa--verification","title":"Verification","text":"
check_matches \u2014 machine-check that two representations agree.
"},{"location":"#hexa--container","title":"Container","text":"
build \u2014 recursively instantiate a full pipeline from class annotations.
"},{"location":"#hexa--notes","title":"Notes","text":"
     
  • All utilities share a single model; any port tree fits.
    •  
  • Generators emit source text only; hexa performs no runtime code generation.
    •  
"},{"location":"#hexa-classes","title":"Classes","text":""},{"location":"#hexa.MethodSpec","title":"MethodSpec  dataclass","text":"
MethodSpec(\n    name: str,\n    args: list[str] = list(),\n    kwargs: dict[str, str] = dict(),\n)\n
 
Specification of a single abstract method.
 
Attributes:
 Name Type Description name str 
Name of the method.
 args list[str] 
Positional parameter names.
 kwargs dict[str, str] 
Keyword-only parameter names mapped to their type expressions.
"},{"location":"#hexa.PortNode","title":"PortNode  dataclass","text":"
PortNode(\n    name: str,\n    port_cls: str,\n    kind: str = \"leaf\",\n    attributes: dict[str, ConfigValue] = dict(),\n    children: list[PortNode] = list(),\n    methods: dict[str, MethodSpec] = dict(),\n)\n
 
A node in the hexa port tree.
 
Each node is a leaf (no children), a port (has nested port slots), or the parent root of the tree. Every node carries a concrete typed shape derived from an ABC class name.
 
Attributes:
 Name Type Description name str 
Slot name, e.g. \"parser\", \"amount_balance\".
 port_cls str 
ABC class name that gives this port its shape, e.g. \"TransactionParserPort\".  Every port has a concrete typed shape.
 kind str 
One of \"leaf\", \"port\", or \"parent\". - \"leaf\": no children (e.g. NumberPort) - \"port\": has children (e.g. TransactionParserPort) - \"parent\": the root of the tree (e.g. ExtractionPipeline)
 attributes dict[str, ConfigValue] 
Annotated attributes: name -> type_expr for scalars, or a nested dict of sub-attributes (a struct) for grouped values.
 children list[PortNode] 
Nested port slots.
 methods dict[str, MethodSpec] 
Abstract methods keyed by method name.
"},{"location":"#hexa.PortNode-functions","title":"Functions","text":""},{"location":"#hexa.PortNode.diff","title":"diff","text":"
diff(other: PortNode, path: str = '') -> list[str]\n
 
Return a list of human-readable difference strings.
 
An empty list means the trees are equal.
 
Parameters:
 Name Type Description Default other PortNode 
Tree to compare against.
 required path str 
Path prefix used when recursing into nested nodes; defaults to the root path.
 '' 
Returns:
 Type Description list[str] 
list[str]: Human-readable difference strings, one per discrepancy.
"},{"location":"#hexa.PortNode.find","title":"find","text":"
find(name: str) -> PortNode | None\n
 
Find a direct child by slot name.
 
Parameters:
 Name Type Description Default name str 
Slot name of the child to look up.
 required 
Returns:
 Type Description PortNode | None 
PortNode | None: The matching child node, or None if no child has that name.
"},{"location":"#hexa.PortNode.walk","title":"walk","text":"
walk() -> list[PortNode]\n
 
Return all nodes in depth-first order (self first).
 
Returns:
 Type Description list[PortNode] 
list[PortNode]: All nodes of the tree, self first, depth-first.
"},{"location":"#hexa-functions","title":"Functions","text":""},{"location":"#hexa.build","title":"build","text":"
build(\n    cls: type[T],\n    *,\n    instances: dict[str, Any] | None = None,\n    config: dict[str, Any] | None = None\n) -> T\n
 
Recursively instantiate the port tree from annotations.
 
Parameters:
 Name Type Description Default cls type[T] 
The root class to build (e.g. AxisExtractionPipeline).
 required instances dict[str, Any] | None 
{slot_name: object} injected verbatim into every node whose annotations contain that name (runtime values \u2014 repos, handlers, clients \u2014 that must not be re-constructed). Applied to the whole tree by name.
 None config dict[str, Any] | None 
{field_name: value} propagated by name to every node's annotated config fields (non-port attributes), replacing the default None.
 None 
Returns:
 Name Type Description T T 
A fully wired instance of cls and all its nested ports.
"},{"location":"check_matches/","title":"Check Matches","text":""},{"location":"check_matches/#hexa.check_matches","title":"hexa.check_matches","text":""},{"location":"check_matches/#hexa.check_matches--summary","title":"Summary","text":"
Verify that two hexa representations agree.
 
check_matches parses two representations \u2014 one YAML spec and/or ABC Python module each \u2014 and reports whether they describe the same port tree.
"},{"location":"check_matches/#hexa.check_matches--notes","title":"Notes","text":"
     
  • Uses PortNode.diff to produce human-readable difference strings.
    •  
"},{"location":"check_matches/#hexa.check_matches-classes","title":"Classes","text":""},{"location":"check_matches/#hexa.check_matches-functions","title":"Functions","text":""},{"location":"check_matches/#hexa.check_matches.check_matches","title":"check_matches","text":"
check_matches(\n    path_a: str | Path, path_b: str | Path\n) -> bool | list[str]\n
 
Verify two representations agree.
 
Parameters:
 Name Type Description Default path_a str | Path 
Path to a .yaml or .py file.
 required path_b str | Path 
Path to a .yaml or .py file.
 required 
Returns:
 Type Description bool | list[str] 
bool | list[str]: True if they match exactly, otherwise a list of difference strings.
"},{"location":"cli/","title":"Cli","text":""},{"location":"cli/#hexa.cli","title":"hexa.cli","text":""},{"location":"cli/#hexa.cli--summary","title":"Summary","text":"
Command-line interface for hexa utilities.
"},{"location":"cli/#hexa.cli--usage","title":"Usage","text":"
hexa parse-yaml PATH\nhexa generate-abc PATH [--out FILE]\nhexa parse-abc PATH\nhexa generate-yaml PATH [--out FILE]\nhexa check PATH_A PATH_B\n
"},{"location":"cli/#hexa.cli-functions","title":"Functions","text":""},{"location":"container/","title":"Container","text":""},{"location":"container/#hexa.container","title":"hexa.container","text":""},{"location":"container/#hexa.container--summary","title":"Summary","text":"
Container to build a wired hexa pipeline from annotations.
 
build recursively instantiates the port tree implied by a root class's annotations, injecting provided runtime instances and config values by name.
"},{"location":"container/#hexa.container--notes","title":"Notes","text":"
     
  • Runtime instances and config values are applied by name across the whole tree.
    •  
"},{"location":"container/#hexa.container-functions","title":"Functions","text":""},{"location":"container/#hexa.container.build","title":"build","text":"
build(\n    cls: type[T],\n    *,\n    instances: dict[str, Any] | None = None,\n    config: dict[str, Any] | None = None\n) -> T\n
 
Recursively instantiate the port tree from annotations.
 
Parameters:
 Name Type Description Default cls type[T] 
The root class to build (e.g. AxisExtractionPipeline).
 required instances dict[str, Any] | None 
{slot_name: object} injected verbatim into every node whose annotations contain that name (runtime values \u2014 repos, handlers, clients \u2014 that must not be re-constructed). Applied to the whole tree by name.
 None config dict[str, Any] | None 
{field_name: value} propagated by name to every node's annotated config fields (non-port attributes), replacing the default None.
 None 
Returns:
 Name Type Description T T 
A fully wired instance of cls and all its nested ports.
"},{"location":"design/","title":"hexa \u2014 Design (extraction pipeline, Axis/Icici)","text":"
Note: The extraction pipeline discussed below is a sample/example illustrating the hexa composition pattern.
 
This document is the concrete counterpart to philosophy.md: where the files are, what the tree looks like for both banks, and how a future container instantiates it. Self-sufficient for a fresh agent.
"},{"location":"design/#file-map-samplesextraction_pipeline","title":"File map (samples/extraction_pipeline/)","text":"File Layer Role extraction_pipeline_abc.py ABC contracts All port ABCs + ExtractionPipeline root. Declares attributes, the ExtractionSource dataclass, port slots, @abstractmethods. extraction_pipeline_impl.py Generic Impl* Concrete default implementations of every port; ImplExtractionPipeline root; clean_num helper. extraction_pipeline_ambiguity.py Shared logic Ported ResolutionContext, AmbiguityHandler, AmountBalanceNotFound, validate_signs \u2014 used by ImplAmountBalancePort. extraction_pipeline_banks.py Composition root AxisExtractionPipeline(ImplExtractionPipeline) + IciciExtractionPipeline (each re-pins txn_dicts). banks/axis/pdf.py Bank specialization AxisNumberPort, AxisDescPort, AxisAmountBalancePort, AxisTransactionParserPort, AxisTxnDictsPort. banks/icici/pdf.py Bank specialization Icici* mirror of the above + Icici-only adjust_balance/missing_number_candidates. extraction_pipeline.yaml Contract spec Declarative spec of the port tree; interchangeable with _abc.py via the optional utilities."},{"location":"design/#the-port-tree-abc-contract","title":"The port tree (ABC contract)","text":"
ExtractionPipeline\n\u251c\u2500\u2500 source / pipeline / trust_fallback          [attributes; source is a nested struct]\n\u251c\u2500\u2500 ingest      \u2192 IngestPort                    (source \u2192 content)\n\u251c\u2500\u2500 raw_lines   \u2192 RawLinesPort                  (content \u2192 raw_lines)\n\u251c\u2500\u2500 txn_blocks  \u2192 TxnBlocksPort                 (raw_lines \u2192 txn_blocks)\n\u2502   \u251c\u2500\u2500 bucket_finder \u2192 BucketFinderPort\n\u2502   \u2514\u2500\u2500 splitter      \u2192 SplitterPort\n\u251c\u2500\u2500 txn_dicts   \u2192 TxnDictsPort                  (txn_blocks \u2192 txn_dicts)\n\u2502   \u2514\u2500\u2500 parser  \u2192 TransactionParserPort\n\u2502       \u251c\u2500\u2500 number         \u2192 NumberPort\n\u2502       \u251c\u2500\u2500 date           \u2192 DatePort\n\u2502       \u251c\u2500\u2500 desc           \u2192 DescPort\n\u2502       \u2514\u2500\u2500 amount_balance \u2192 AmountBalancePort\n\u2502           \u251c\u2500\u2500 number    \u2192 NumberPort\n\u2502           \u2514\u2500\u2500 ambiguity \u2192 AmbiguityPort\n\u2514\u2500\u2500 raw_expense \u2192 RawExpensePort                (txn_dicts \u2192 list[RawExpense], terminal)\n
"},{"location":"design/#generic-default-impl","title":"Generic default (Impl*)","text":"
ImplExtractionPipeline pins every stage slot to a concrete Impl* port:
 
ImplExtractionPipeline\n\u251c\u2500\u2500 ingest:      ImplIngestPort\n\u251c\u2500\u2500 raw_lines:   ImplRawLinesPort\n\u251c\u2500\u2500 txn_blocks:  ImplTxnBlocksPort\n\u2502   \u251c\u2500\u2500 bucket_finder: ImplBucketFinderPort\n\u2502   \u2514\u2500\u2500 splitter:      ImplSplitterPort\n\u251c\u2500\u2500 txn_dicts:   ImplTxnDictsPort\n\u2502   \u2514\u2500\u2500 parser:  ImplTransactionParserPort\n\u2502       \u251c\u2500\u2500 number:        ImplNumberPort\n\u2502       \u251c\u2500\u2500 date:          ImplDatePort\n\u2502       \u251c\u2500\u2500 desc:          ImplDescPort\n\u2502       \u2514\u2500\u2500 amount_balance: ImplAmountBalancePort\n\u2502           \u251c\u2500\u2500 number:    ImplNumberPort\n\u2502           \u2514\u2500\u2500 ambiguity: ImplAmbiguityPort\n\u2514\u2500\u2500 raw_expense: ImplRawExpensePort\n
"},{"location":"design/#bank-variation-only-what-differs","title":"Bank variation (only what differs)","text":"
The root class re-pins only txn_dicts. Everything else inherits Impl*.
"},{"location":"design/#axis-banksaxispdfpy","title":"Axis (banks/axis/pdf.py)","text":"
class AxisExtractionPipeline(ImplExtractionPipeline):\n    txn_dicts: AxisTxnDictsPort\n
 
AxisTxnDictsPort(ImplTxnDictsPort)\n  \u2514\u2500 parser: AxisTransactionParserPort(ImplTransactionParserPort)\n      \u251c\u2500 number:  AxisNumberPort          # NUMBER_RE, all clean numbers\n      \u251c\u2500 desc:    AxisDescPort            # STARTTERS = [upi/, imps/, neft-, neft/, ach/, ach-, 2a/]\n      \u2514\u2500 amount_balance: AxisAmountBalancePort(ImplAmountBalancePort)\n          \u2514\u2500 number: AxisNumberPort       # re-pin back to axis number port\n
 
Notes: - date is not redeclared on AxisTransactionParserPort \u2014 it inherits   ImplDatePort (correct already, adds no behavior). - amount_balance.ambiguity is not redeclared \u2014 inherits ImplAmbiguityPort.
"},{"location":"design/#icici-banksicicipdfpy","title":"Icici (banks/icici/pdf.py)","text":"
class IciciExtractionPipeline(ImplExtractionPipeline):\n    txn_dicts: IciciTxnDictsPort\n
 
IciciTxnDictsPort(ImplTxnDictsPort)\n  \u2514\u2500 parser: IciciTransactionParserPort(ImplTransactionParserPort)\n      \u251c\u2500 number:  IciciNumberPort         # NUMBER_RE, returns decimals[-2:] (last 2)\n      \u251c\u2500 desc:    IciciDescPort           # STARTTERS + fix_ocr (UPl\u2192UPI, ..., ICIC]\u2192ICICI)\n      \u2514\u2500 amount_balance: IciciAmountBalancePort(ImplAmountBalancePort)\n          \u251c\u2500 number: IciciNumberPort      # re-pin to icici number port\n          \u251c\u2500 adjust_balance:              # neg-markers [(-), (-}, {-}, {-) , ()] \u2192 -abs(balance)\n          \u2514\u2500 missing_number_candidates:   # Icici-only one-number hook\n
"},{"location":"design/#redundancy-rule-applied-in-both-banks","title":"Redundancy rule (applied in both banks)","text":"
Only redeclare a slot you are actually changing from Impl*. If a slot's behavior is already correct from the parent Impl*, omit it. This keeps the diff between a bank and Impl* minimal and unambiguous.
 
date: ImplDatePort was previously redeclared redundantly and has been removed \u2014 it added mental load on the extender for zero behavior change. (Fidelity audit: it is still concrete ImplDatePort in ImplTransactionParserPort, so behavior is unchanged.)
"},{"location":"design/#container-buildcls-in-hexacontainerpy","title":"Container (build(cls) in hexa/container.py)","text":"
The annotations are metadata; a small reflection builder turns a root class into a wired instance:
 
build(AxisExtractionPipeline)  \u2192  wired AxisExtractionPipeline\n
 
Algorithm (build(cls)):
 
     
  1. Walk cls.__annotations__, merged across the MRO so inherited slot annotations    are visible (a bank subclass only adds/overrides a slot; the rest come from    Impl*).
    2.  
  2. For each port slot whose annotated type is a concrete Impl*/bank class    (not an ABC, not a builtin), instantiate it.
    3.  
  3. Recurse into that child \u2014 repeat until leaves (classes that declare no further    port slots).
    4.  
  4. setattr(parent, slot, child) to wire each slot onto the parent instance.
    5.  
  5. Apply config defaults from the class-level annotated attributes.
    6.  
 
Selection is fully static: the annotated type is the chosen implementation. The container holds no per-bank branch logic \u2014 Axis vs Icici is decided purely by which root class you call build on.
"},{"location":"design/#control-flow-at-runtime-target","title":"Control flow at runtime (target)","text":"
A leaf-stage port exposes one entry method; stages chain data through the shared run(source) signature on the root:
 
run(source)\n  ingest.materialize(source)            \u2192 content\n  raw_lines.extract(content)            \u2192 raw_lines\n  txn_blocks.extract(raw_lines)         \u2192 {opening_balance: [blocks]}\n  txn_dicts.extract(txn_blocks)         \u2192 [txn dicts]\n  raw_expense.build(txn_dicts)          \u2192 [RawExpense]   (terminal)\n
 
The stage ports map naturally onto dagpipe Nodes (ingest, raw_lines, txn_blocks, txn_dicts, raw_expense), each node's resolve delegating to its wired port subtree, connected as a dagpipe Graph and run via Engine. That merge is a separate, future step.
"},{"location":"design/#utilities-optional-hexa-agnostic","title":"Utilities (optional, hexa-agnostic)","text":"
The hexa utilities are implemented in hexa/ (the package root) and they are agnostic of the extraction pipeline sample. It ships a set of optional code-gen / validation utilities so users can choose their own workflow \u2014 they are never required. A user is free to:
 
     
  • hand-write an ABC file and never touch YAML, or
    •  
  • start from YAML and generate the ABC file, or
    •  
  • start from an ABC file and emit YAML for it, or
    •  
  • use the ABC/YAML only as a spec (neither generated from the other).
    •  
 
The utilities only ever translate between two interchangeable representations of the same contract \u2014 the .yaml spec and the _abc.py module. They never generate or manage the Impl*/bank files (those carry behavior and slot choice, which is a human decision).
"},{"location":"design/#shared-model","title":"Shared model","text":"
All utilities serialize through one neutral model, a PortNode tree. It is NOT tied to the extraction sample \u2014 any port tree fits.
 
@dataclass\nclass PortNode:\n    name: str                 # slot name, e.g. \"parser\", \"amount_balance\"\n    kind: str                 # \"leaf\" | \"port\" | \"parent\"\n    port_cls: str             # ABC class name, e.g. \"TransactionParserPort\"\n    attributes: dict[str, ConfigValue]  # annotated attributes: name -> type expr,\n                              # or a nested struct dict for grouped values\n    children: list[\"PortNode\"]\n    methods: dict[str, MethodSpec]   # abstract methods\n
"},{"location":"design/#1-parse_yamlpath-portnode","title":"1. parse_yaml(path) -> PortNode","text":"
Read a YAML spec into the PortNode tree (YAML \u2192 model).
 
     
  • Parses completely (structural validation only, no type-checking).
    •  
  • Supports the sample grammar (see extraction_pipeline.yaml): a root class whose   inline attributes (scalars and nested structs), port slots, and reserved   methods: recurse into child ports.
    •  
  • Raises ValueError with file/line on structural errors.
    •  
"},{"location":"design/#2-generate_abcpath_yaml-out_pathnone-str","title":"2. generate_abc(path_yaml, out_path=None) -> str","text":"
Generate _abc.py source from a YAML spec (YAML \u2192 ABC).
 
     
  • parse_yaml(...) \u2192 write one class <PortCls>(ABC) per PortNode with   attribute annotations (nested structs become synthesized @dataclasses), port-slot   annotations, and @abstractmethod stubs, matching the style of the hand-written   extraction_pipeline_abc.py.
    •  
  • Deterministic (YAML-order) output so it is idempotent / diff-friendly.
    •  
  • out_path optional; always returns the generated source.
    •  
"},{"location":"design/#3-parse_abcmodule_or_path-portnode","title":"3. parse_abc(module_or_path) -> PortNode","text":"
Reflect over an ABC module into the PortNode tree (ABC \u2192 model).
 
     
  • Reads __annotations__ / @abstractmethod from generic ABC classes (no   sample assumptions) to reconstruct the same PortNode tree.
    •  
"},{"location":"design/#4-generate_yamlmodule_or_path-out_pathnone-str","title":"4. generate_yaml(module_or_path, out_path=None) -> str","text":"
Generate a .yaml spec from an ABC module (ABC \u2192 YAML).
 
     
  • parse_abc(...) \u2192 write the reverse of generate_abc.
    •  
  • Lets a user author the ABC file first and emit YAML for it.
    •  
"},{"location":"design/#5-check_matchespath_a-path_b-bool","title":"5. check_matches(path_a, path_b) -> bool","text":"
Verify two representations agree \u2014 YAML vs ABC, either direction.
 
     
  • Builds both PortNode models (parse_yaml + parse_abc) and compares: classes   exist and are ABCs, attributes match, port slots match, @abstractmethod   signatures match (missing keyword-only params treated conservatively).
    •  
  • Returns True only if the whole tree matches; otherwise False (or a diff).
    •  
"},{"location":"design/#workflows-users-choice","title":"Workflows (user's choice)","text":"
# YAML-first\nedit spec.yaml\n  \u2192 generate_abc(spec.yaml, abc.py)          # regenerate ABCs\n  \u2192 hand-write/update Impl* + banks\n  \u2192 check_matches(spec.yaml, abc.py) == True # guard\n\n# ABC-first\nedit abc.py\n  \u2192 generate_yaml(abc.py, spec.yaml)         # emit YAML spec\n  \u2192 check_matches(spec.yaml, abc.py) == True\n\n# hand-written both (no utilities at all)\nedit abc.py + spec.yaml independently        # optional check_matches as a guard\n
 
Impl*/bank files are always hand-extended on top and are never generated.
"},{"location":"generate_abc/","title":"Generate Abc","text":""},{"location":"generate_abc/#hexa.generate_abc","title":"hexa.generate_abc","text":""},{"location":"generate_abc/#hexa.generate_abc--summary","title":"Summary","text":"
Generate _abc.py source from a hexa PortNode tree.
"},{"location":"generate_abc/#hexa.generate_abc-classes","title":"Classes","text":""},{"location":"generate_abc/#hexa.generate_abc-functions","title":"Functions","text":""},{"location":"generate_abc/#hexa.generate_abc.generate_abc","title":"generate_abc","text":"
generate_abc(\n    node: PortNode, out_path: str | Path | None = None\n) -> str\n
 
Generate ABC Python source from a :class:PortNode tree.
 
Parameters:
 Name Type Description Default node PortNode 
The root of the parsed YAML (or generated) tree.
 required out_path str | Path | None 
Optional file path to write the generated source to.
 None 
Returns:
 Name Type Description str str 
The generated Python source code.
"},{"location":"generate_yaml/","title":"Generate Yaml","text":""},{"location":"generate_yaml/#hexa.generate_yaml","title":"hexa.generate_yaml","text":""},{"location":"generate_yaml/#hexa.generate_yaml--summary","title":"Summary","text":"
Generate YAML spec (nested mapping form) from a hexa PortNode tree.
"},{"location":"generate_yaml/#hexa.generate_yaml-classes","title":"Classes","text":""},{"location":"generate_yaml/#hexa.generate_yaml-functions","title":"Functions","text":""},{"location":"generate_yaml/#hexa.generate_yaml.generate_yaml","title":"generate_yaml","text":"
generate_yaml(\n    node: PortNode, out_path: str | Path | None = None\n) -> str\n
 
Generate YAML source from a :class:PortNode tree.
 
Parameters:
 Name Type Description Default node PortNode 
The root of the tree.
 required out_path str | Path | None 
Optional file path to write the generated source to.
 None 
Returns:
 Name Type Description str str 
The generated YAML source.
"},{"location":"model/","title":"Model","text":""},{"location":"model/#hexa.model","title":"hexa.model","text":""},{"location":"model/#hexa.model--summary","title":"Summary","text":"
Shared neutral model for hexa port trees.
 
All utilities (parse_yaml, generate_abc, parse_abc, generate_yaml, check_matches) serialize through this single model.
"},{"location":"model/#hexa.model--notes","title":"Notes","text":"
     
  • The model is NOT tied to any sample \u2014 any port tree fits.
    •  
"},{"location":"model/#hexa.model-classes","title":"Classes","text":""},{"location":"model/#hexa.model.MethodSpec","title":"MethodSpec  dataclass","text":"
MethodSpec(\n    name: str,\n    args: list[str] = list(),\n    kwargs: dict[str, str] = dict(),\n)\n
 
Specification of a single abstract method.
 
Attributes:
 Name Type Description name str 
Name of the method.
 args list[str] 
Positional parameter names.
 kwargs dict[str, str] 
Keyword-only parameter names mapped to their type expressions.
"},{"location":"model/#hexa.model.PortNode","title":"PortNode  dataclass","text":"
PortNode(\n    name: str,\n    port_cls: str,\n    kind: str = \"leaf\",\n    attributes: dict[str, ConfigValue] = dict(),\n    children: list[PortNode] = list(),\n    methods: dict[str, MethodSpec] = dict(),\n)\n
 
A node in the hexa port tree.
 
Each node is a leaf (no children), a port (has nested port slots), or the parent root of the tree. Every node carries a concrete typed shape derived from an ABC class name.
 
Attributes:
 Name Type Description name str 
Slot name, e.g. \"parser\", \"amount_balance\".
 port_cls str 
ABC class name that gives this port its shape, e.g. \"TransactionParserPort\".  Every port has a concrete typed shape.
 kind str 
One of \"leaf\", \"port\", or \"parent\". - \"leaf\": no children (e.g. NumberPort) - \"port\": has children (e.g. TransactionParserPort) - \"parent\": the root of the tree (e.g. ExtractionPipeline)
 attributes dict[str, ConfigValue] 
Annotated attributes: name -> type_expr for scalars, or a nested dict of sub-attributes (a struct) for grouped values.
 children list[PortNode] 
Nested port slots.
 methods dict[str, MethodSpec] 
Abstract methods keyed by method name.
"},{"location":"model/#hexa.model.PortNode-functions","title":"Functions","text":""},{"location":"model/#hexa.model.PortNode.diff","title":"diff","text":"
diff(other: PortNode, path: str = '') -> list[str]\n
 
Return a list of human-readable difference strings.
 
An empty list means the trees are equal.
 
Parameters:
 Name Type Description Default other PortNode 
Tree to compare against.
 required path str 
Path prefix used when recursing into nested nodes; defaults to the root path.
 '' 
Returns:
 Type Description list[str] 
list[str]: Human-readable difference strings, one per discrepancy.
"},{"location":"model/#hexa.model.PortNode.find","title":"find","text":"
find(name: str) -> PortNode | None\n
 
Find a direct child by slot name.
 
Parameters:
 Name Type Description Default name str 
Slot name of the child to look up.
 required 
Returns:
 Type Description PortNode | None 
PortNode | None: The matching child node, or None if no child has that name.
"},{"location":"model/#hexa.model.PortNode.walk","title":"walk","text":"
walk() -> list[PortNode]\n
 
Return all nodes in depth-first order (self first).
 
Returns:
 Type Description list[PortNode] 
list[PortNode]: All nodes of the tree, self first, depth-first.
"},{"location":"parse_abc/","title":"Parse Abc","text":""},{"location":"parse_abc/#hexa.parse_abc","title":"hexa.parse_abc","text":""},{"location":"parse_abc/#hexa.parse_abc--summary","title":"Summary","text":"
Parse an ABC Python module into a hexa PortNode tree.
 
ABC classes become port nodes.  A nested-struct attribute is expressed as a @dataclass annotation on an ABC port (e.g. source: ExtractionSource where ExtractionSource is a @dataclass); the dataclass's fields are expanded into a nested attributes entry.
"},{"location":"parse_abc/#hexa.parse_abc--notes","title":"Notes","text":"
     
  • Accepts a file path, a module import string, or an already-loaded module.
    •  
"},{"location":"parse_abc/#hexa.parse_abc-classes","title":"Classes","text":""},{"location":"parse_abc/#hexa.parse_abc-functions","title":"Functions","text":""},{"location":"parse_abc/#hexa.parse_abc.parse_abc","title":"parse_abc","text":"
parse_abc(module_or_path: str | Path | Any) -> PortNode\n
 
Parse an ABC Python module into a :class:PortNode tree.
 
ABC classes become port nodes.  A nested-struct attribute is expressed as a @dataclass annotation on an ABC port (e.g. source: ExtractionSource where ExtractionSource is a @dataclass); the dataclass's fields are expanded into a nested attributes entry.
 
Parameters:
 Name Type Description Default module_or_path str | Path | Any 
A file path to a .py file, a module import string, or a loaded module.
 required 
Returns:
 Name Type Description PortNode PortNode 
The root node of the tree.
"},{"location":"parse_yaml/","title":"Parse Yaml","text":""},{"location":"parse_yaml/#hexa.parse_yaml","title":"hexa.parse_yaml","text":""},{"location":"parse_yaml/#hexa.parse_yaml--summary","title":"Summary","text":"
Parse a hexa YAML spec into a :class:PortNode tree.
 
The YAML grammar uses a nested mapping format built from three member kinds:
 
     
  • attribute: a name: type scalar (e.g. source_type: str)
    •  
  • nested struct: a name: mapping of sub-attributes (e.g. source:)
    •  
  • port: a name: mapping whose single key is a class whose value is a   body, or a class key with a ... body.  A port carries a class shape.
    •  
 Disambiguation of a mapping value 
A mapping with exactly one key whose value is a non-scalar body (a mapping, ..., or a list) is a port.  Any other mapping (multiple keys, or all scalar-typed values) is a nested struct of attributes.
"},{"location":"parse_yaml/#hexa.parse_yaml--examples","title":"Examples","text":"
Parse a spec into a port tree:
 
```python\nfrom hexa import parse_yaml\n\nnode = parse_yaml(\"samples/minimal/sample.yaml\")\nprint(node)\n```\n
 
The YAML grammar:
 
```yaml\nRootClassName:\n  attr: type                 # scalar attribute\n  source:                    # nested struct (all scalar sub-attributes)\n    source_type: str\n    bank: str\n  methods:                   # reserved block of functions\n    run:\n      args: [a, b]\n  ingest:                    # port (single non-scalar class key)\n    IngestPort:\n      source_type: str\n      methods: {Method: ...}\n```\n
"},{"location":"parse_yaml/#hexa.parse_yaml--notes","title":"Notes","text":"
     
  • See samples/minimal/sample.yaml and   samples/extraction_pipeline/extraction_pipeline.yaml for worked examples.
    •  
"},{"location":"parse_yaml/#hexa.parse_yaml-classes","title":"Classes","text":""},{"location":"parse_yaml/#hexa.parse_yaml-functions","title":"Functions","text":""},{"location":"parse_yaml/#hexa.parse_yaml.parse_yaml","title":"parse_yaml","text":"
parse_yaml(path: str | Path) -> PortNode\n
 
Parse a hexa YAML spec file into a :class:PortNode tree.
 
Parameters:
 Name Type Description Default path str | Path 
Path to the .yaml file.
 required 
Returns:
 Name Type Description PortNode PortNode 
The root node of the parsed tree.
 
Raises:
 Type Description ValueError 
On structural errors in the YAML.
"},{"location":"philosophy/","title":"hexa \u2014 Philosophy","text":"
This document explains the why behind the type-declared dependency pattern used for the extraction pipeline sample \u2014 specifically the Axis/Icici bank variation. It is, together with design.md, self-sufficient: a fresh agent should be able to pick up the codebase from here and reason about (and extend) the pattern without needing the original session context.
"},{"location":"philosophy/#one-idea","title":"One idea","text":"
A concrete class's annotated slot type is the dependency decision.
 
Bank variation is expressed entirely as class annotations \u2014 never as imperative wiring, never as __init__ parameters, never as a composition-root that assembles objects by hand. The tree of ports a pipeline needs is declared once, and a bank differs from the generic pipeline only by which concrete types are pinned onto which slots.
"},{"location":"philosophy/#the-three-layers","title":"The three layers","text":"
The pattern is built from three fixed layers. Understanding which layer something belongs to is the whole mental model.
"},{"location":"philosophy/#layer-1-abc-contracts-samplesextraction_pipelineextraction_pipeline_abcpy","title":"Layer 1 \u2014 ABC contracts (samples/extraction_pipeline/extraction_pipeline_abc.py)","text":"
Authoritative declaration of need (\"WHAT a stage requires\").
 
     
  • Each port is an ABC.
    •  
  • It declares attributes as class-level annotated defaults   (e.g. TransactionParserPort.min_numbers: int = 3), optionally grouped as a   nested struct via a @dataclass (e.g. source: ExtractionSource).
    •  
  • It declares port slots as annotated attributes whose type is another ABC   (e.g. TransactionParserPort.number: NumberPort, amount_balance: AmountBalancePort).
    •  
  • It declares @abstractmethod bodies \u2014 the method signatures each concrete   implementation must provide.
    •  
 
The ABC never runs, never holds an instance, and never says which implementation to use. It only says what the shape is.
"},{"location":"philosophy/#layer-2-generic-concurrent-implementations-samplesextraction_pipelineextraction_pipeline_implpy","title":"Layer 2 \u2014 generic concurrent implementations (samples/extraction_pipeline/extraction_pipeline_impl.py)","text":"
The common default behavior (\"Imple \u2014 the shared adaptor\").
 
     
  • class ImplTransactionParserPort(TransactionParserPort) subclasses the ABC,   implements every @abstractmethod, and re-pins its port slots to concrete   Impl* types: number: ImplNumberPort, desc: ImplDescPort,   amount_balance: ImplAmountBalancePort.
    •  
  • ImplAmountBalancePort in turn pins number: ImplNumberPort and   ambiguity: ImplAmbiguityPort.
    •  
  • The root, ImplExtractionPipeline, pins its five stage slots to Impl* ports:   ingest, raw_lines, txn_blocks, txn_dicts, raw_expense.
    •  
 
Impl* is the default that shared, bank-agnostic behavior lives in. Any slot a bank does not override falls back to Impl* at runtime.
"},{"location":"philosophy/#layer-3-bank-specializations-samplesextraction_pipelinebanksaxisicicipdfpy-samplesextraction_pipelineextraction_pipeline_bankspy","title":"Layer 3 \u2014 bank specializations (samples/extraction_pipeline/banks/{axis,icici}/pdf.py, samples/extraction_pipeline/extraction_pipeline_banks.py)","text":"
The variation (\"what differs per bank\").
 
     
  • Bank classes subclass the concrete Impl* classes, never the ABC \u2014 they   extend, they never reinvent the contract.
    •  
  • Each Axis*/Icici* class overrides only the members that differ from the   generic Impl*; everything shared is inherited.
    •  
  • A bank root is a thin annotated subclass of ImplExtractionPipeline that   re-pins only the slots that differ:
    •  
 
class AxisExtractionPipeline(ImplExtractionPipeline):\n    txn_dicts: AxisTxnDictsPort\n
"},{"location":"philosophy/#the-re-pinning-cascade","title":"The re-pinning cascade","text":"
Bank-specific wiring is hierarchical and cascading. Each level narrows exactly one slot, and its narrowed type drags the next level's narrowing along with it:
 
AxisExtractionPipeline\n  \u2514\u2500 txn_dicts: AxisTxnDictsPort              (only override on the root)\n      \u2514\u2500 parser: AxisTransactionParserPort\n          \u251c\u2500 number:  AxisNumberPort          (get_numbers \u2192 all clean nums)\n          \u251c\u2500 desc:    AxisDescPort            (STARTTERS)\n          \u2514\u2500 amount_balance: AxisAmountBalancePort\n              \u2514\u2500 number: AxisNumberPort       (re-pin: same bank's number port)\n
 
Siblings that are already correct in Impl* are inherited, not redeclared. E.g. the parser's date slot comes from ImplTransactionParserPort; the amount-balance port's ambiguity comes from ImplAmountBalancePort. Re-declaring them adds mental load on the extender with zero behavior change.
 
Rule \u2014 only redeclare the slot you are actually changing. If a slot's behavior is already correct from Impl*, leave it out.
"},{"location":"philosophy/#the-two-fidelity-guarantees-this-pattern-preserves","title":"The two fidelity guarantees this pattern preserves","text":"
     
  1. Shared behavior stays shared. Slots like ingest, raw_lines, txn_blocks,    raw_expense, parser date, and ambiguity are bank-agnostic. Leaving them at    the Impl* default is deliberate, not an omission.
    2.  
  2. Banks never reinvent. Axis*/Icici* only extend Impl*. The ABC contract    in samples/extraction_pipeline/extraction_pipeline_abc.py is defined once and shared by every bank.
    3.  
"},{"location":"philosophy/#what-this-pattern-is-not","title":"What this pattern is NOT","text":"
     
  • No imperative assembly. No build_axis_pipeline() wiring objects by hand.
    •  
  • No constructor injection. Impl* classes have no __init__; wiring is done by   assigning concrete port instances onto annotated slots.
    •  
  • YAML and ABCs are interchangeable, and both optional. The .yaml spec and   the _abc.py module are two representations of the same contract. hexa is   agnostic of the extraction pipeline sample and never imposes a workflow: a user can hand-write the   ABC file, start from YAML (ABC generated), start from an ABC file (YAML   emitted), or use both as a spec. The optional utilities in   design.md \u00a7 Utilities translate between them and can verify they   agree \u2014 but the bank Impl*/specialization files are always hand-written   (their slots are a manual choice) and never generated.
    •  
"},{"location":"philosophy/#runtime-instantiation","title":"Runtime instantiation","text":"
The annotations are the decision; hexa.build(cls) (see design.md \u00a7 Container) turns a root class into a wired instance by reading __annotations__ + MRO, recursively instantiating each pinned concrete type, and setattring the child onto the parent slot. This document is scoped to the notation \u2014 the rest of the tree can be expressed purely as class annotations and filled at runtime by the container.
"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"hexa","text":""},{"location":"#hexa","title":"hexa","text":""},{"location":"#hexa--summary","title":"Summary","text":"
Type-declared dependency injection for Python.
 
Hexa is a pattern and utility library for structuring complex hierarchical pipelines. It rests on one core idea: a concrete class's annotated slot type is the dependency decision. Variation is expressed entirely as class annotations \u2014 never as imperative wiring, never as __init__ parameters, never as a composition root that assembles objects by hand.
 
All utilities serialize through a single shared model (PortNode):
 
     
  • parse_yaml / generate_yaml \u2014 between YAML specs and the model
    •  
  • parse_abc / generate_abc \u2014 between ABC Python modules and the model
    •  
  • check_matches \u2014 verify two representations describe the same tree
    •  
  • build \u2014 recursively instantiate a wired pipeline from annotations
    •  
"},{"location":"#hexa--installation","title":"Installation","text":"
Install using pip:
 
pip install hexa\n
"},{"location":"#hexa--quick-start","title":"Quick start","text":"
Define contracts, pin concrete types on the slots, and let build wire the tree from annotations:
 
from hexa import build\n\nclass NumberPort(ABC):\n    @abstractmethod\n    def extract(self, line: str) -> float: ...\n\nclass ParserPort(ABC):\n    number: NumberPort\n\n    @abstractmethod\n    def parse(self, text: str) -> dict: ...\n\nclass RegexNumberPort(NumberPort):\n    def extract(self, line: str) -> float:\n        ...\n\nclass StandardParserPort(ParserPort):\n    number: RegexNumberPort  # <-- the dependency decision\n\n    def parse(self, text: str) -> dict:\n        return {\"value\": self.number.extract(text)}\n\npipeline = build(StandardParserPort)\nprint(pipeline.parse(\"Value: 42\"))\n
"},{"location":"#hexa--cli-usage","title":"CLI usage","text":"
Convert between YAML and ABC representations:
 
hexa parse-yaml spec.yaml\nhexa generate-abc spec.yaml --out spec_abc.py\nhexa parse-abc spec_abc.py\nhexa generate-yaml spec_abc.py --out spec.yaml\nhexa check spec.yaml spec_abc.py\n
"},{"location":"#hexa--core-concepts","title":"Core concepts","text":""},{"location":"#hexa--shared-model","title":"Shared model","text":"
PortNode \u2014 the neutral tree every parser and generator serializes through.
"},{"location":"#hexa--converters","title":"Converters","text":"
Parsers and generators that translate between YAML specs and ABC Python modules via the shared model.
"},{"location":"#hexa--verification","title":"Verification","text":"
check_matches \u2014 machine-check that two representations agree.
"},{"location":"#hexa--container","title":"Container","text":"
build \u2014 recursively instantiate a full pipeline from class annotations.
"},{"location":"#hexa--notes","title":"Notes","text":"
     
  • All utilities share a single model; any port tree fits.
    •  
  • Generators emit source text only; hexa performs no runtime code generation.
    •  
"},{"location":"#hexa-classes","title":"Classes","text":""},{"location":"#hexa.MethodSpec","title":"MethodSpec  dataclass","text":"
MethodSpec(\n    name: str,\n    args: list[str] = list(),\n    kwargs: dict[str, str] = dict(),\n)\n
 
Specification of a single abstract method.
 
Attributes:
 Name Type Description name str 
Name of the method.
 args list[str] 
Positional parameter names.
 kwargs dict[str, str] 
Keyword-only parameter names mapped to their type expressions.
"},{"location":"#hexa.PortNode","title":"PortNode  dataclass","text":"
PortNode(\n    name: str,\n    port_cls: str,\n    kind: str = \"leaf\",\n    attributes: dict[str, ConfigValue] = dict(),\n    children: list[PortNode] = list(),\n    methods: dict[str, MethodSpec] = dict(),\n)\n
 
A node in the hexa port tree.
 
Each node is a leaf (no children), a port (has nested port slots), or the parent root of the tree. Every node carries a concrete typed shape derived from an ABC class name.
 
Attributes:
 Name Type Description name str 
Slot name, e.g. \"parser\", \"amount_balance\".
 port_cls str 
ABC class name that gives this port its shape, e.g. \"TransactionParserPort\".  Every port has a concrete typed shape.
 kind str 
One of \"leaf\", \"port\", or \"parent\". - \"leaf\": no children (e.g. NumberPort) - \"port\": has children (e.g. TransactionParserPort) - \"parent\": the root of the tree (e.g. ExtractionPipeline)
 attributes dict[str, ConfigValue] 
Annotated attributes: name -> type_expr for scalars, or a nested dict of sub-attributes (a struct) for grouped values.
 children list[PortNode] 
Nested port slots.
 methods dict[str, MethodSpec] 
Abstract methods keyed by method name.
"},{"location":"#hexa.PortNode-functions","title":"Functions","text":""},{"location":"#hexa.PortNode.diff","title":"diff","text":"
diff(other: PortNode, path: str = '') -> list[str]\n
 
Return a list of human-readable difference strings.
 
An empty list means the trees are equal.
 
Parameters:
 Name Type Description Default other PortNode 
Tree to compare against.
 required path str 
Path prefix used when recursing into nested nodes; defaults to the root path.
 '' 
Returns:
 Type Description list[str] 
list[str]: Human-readable difference strings, one per discrepancy.
"},{"location":"#hexa.PortNode.find","title":"find","text":"
find(name: str) -> PortNode | None\n
 
Find a direct child by slot name.
 
Parameters:
 Name Type Description Default name str 
Slot name of the child to look up.
 required 
Returns:
 Type Description PortNode | None 
PortNode | None: The matching child node, or None if no child has that name.
"},{"location":"#hexa.PortNode.walk","title":"walk","text":"
walk() -> list[PortNode]\n
 
Return all nodes in depth-first order (self first).
 
Returns:
 Type Description list[PortNode] 
list[PortNode]: All nodes of the tree, self first, depth-first.
"},{"location":"#hexa-functions","title":"Functions","text":""},{"location":"#hexa.build","title":"build","text":"
build(\n    cls: type[T],\n    *,\n    instances: dict[str, Any] | None = None,\n    config: dict[str, Any] | None = None\n) -> T\n
 
Recursively instantiate the port tree from annotations.
 
Parameters:
 Name Type Description Default cls type[T] 
The root class to build (e.g. AxisExtractionPipeline).
 required instances dict[str, Any] | None 
{slot_name: object} injected verbatim into every node whose annotations contain that name (runtime values \u2014 repos, handlers, clients \u2014 that must not be re-constructed). Applied to the whole tree by name.
 None config dict[str, Any] | None 
{field_name: value} propagated by name to every node's annotated config fields (non-port attributes), replacing the default None.
 None 
Returns:
 Name Type Description T T 
A fully wired instance of cls and all its nested ports.
"},{"location":"check_matches/","title":"Check Matches","text":""},{"location":"check_matches/#hexa.check_matches","title":"hexa.check_matches","text":""},{"location":"check_matches/#hexa.check_matches--summary","title":"Summary","text":"
Verify that two hexa representations agree.
 
check_matches parses two representations \u2014 one YAML spec and/or ABC Python module each \u2014 and reports whether they describe the same port tree.
"},{"location":"check_matches/#hexa.check_matches--notes","title":"Notes","text":"
     
  • Uses PortNode.diff to produce human-readable difference strings.
    •  
"},{"location":"check_matches/#hexa.check_matches-classes","title":"Classes","text":""},{"location":"check_matches/#hexa.check_matches-functions","title":"Functions","text":""},{"location":"check_matches/#hexa.check_matches.check_matches","title":"check_matches","text":"
check_matches(\n    path_a: str | Path, path_b: str | Path\n) -> bool | list[str]\n
 
Verify two representations agree.
 
Parameters:
 Name Type Description Default path_a str | Path 
Path to a .yaml or .py file.
 required path_b str | Path 
Path to a .yaml or .py file.
 required 
Returns:
 Type Description bool | list[str] 
bool | list[str]: True if they match exactly, otherwise a list of difference strings.
"},{"location":"cli/","title":"Cli","text":""},{"location":"cli/#hexa.cli","title":"hexa.cli","text":""},{"location":"cli/#hexa.cli--summary","title":"Summary","text":"
Command-line interface for hexa utilities.
"},{"location":"cli/#hexa.cli--usage","title":"Usage","text":"
hexa parse-yaml PATH\nhexa generate-abc PATH [--out FILE]\nhexa parse-abc PATH\nhexa generate-yaml PATH [--out FILE]\nhexa check PATH_A PATH_B\n
"},{"location":"cli/#hexa.cli-functions","title":"Functions","text":""},{"location":"container/","title":"Container","text":""},{"location":"container/#hexa.container","title":"hexa.container","text":""},{"location":"container/#hexa.container--summary","title":"Summary","text":"
Container to build a wired hexa pipeline from annotations.
 
build recursively instantiates the port tree implied by a root class's annotations, injecting provided runtime instances and config values by name.
"},{"location":"container/#hexa.container--notes","title":"Notes","text":"
     
  • Runtime instances and config values are applied by name across the whole tree.
    •  
"},{"location":"container/#hexa.container-functions","title":"Functions","text":""},{"location":"container/#hexa.container.build","title":"build","text":"
build(\n    cls: type[T],\n    *,\n    instances: dict[str, Any] | None = None,\n    config: dict[str, Any] | None = None\n) -> T\n
 
Recursively instantiate the port tree from annotations.
 
Parameters:
 Name Type Description Default cls type[T] 
The root class to build (e.g. AxisExtractionPipeline).
 required instances dict[str, Any] | None 
{slot_name: object} injected verbatim into every node whose annotations contain that name (runtime values \u2014 repos, handlers, clients \u2014 that must not be re-constructed). Applied to the whole tree by name.
 None config dict[str, Any] | None 
{field_name: value} propagated by name to every node's annotated config fields (non-port attributes), replacing the default None.
 None 
Returns:
 Name Type Description T T 
A fully wired instance of cls and all its nested ports.
"},{"location":"generate_abc/","title":"Generate Abc","text":""},{"location":"generate_abc/#hexa.generate_abc","title":"hexa.generate_abc","text":""},{"location":"generate_abc/#hexa.generate_abc--summary","title":"Summary","text":"
Generate _abc.py source from a hexa PortNode tree.
"},{"location":"generate_abc/#hexa.generate_abc-classes","title":"Classes","text":""},{"location":"generate_abc/#hexa.generate_abc-functions","title":"Functions","text":""},{"location":"generate_abc/#hexa.generate_abc.generate_abc","title":"generate_abc","text":"
generate_abc(\n    node: PortNode, out_path: str | Path | None = None\n) -> str\n
 
Generate ABC Python source from a :class:PortNode tree.
 
Parameters:
 Name Type Description Default node PortNode 
The root of the parsed YAML (or generated) tree.
 required out_path str | Path | None 
Optional file path to write the generated source to.
 None 
Returns:
 Name Type Description str str 
The generated Python source code.
"},{"location":"generate_yaml/","title":"Generate Yaml","text":""},{"location":"generate_yaml/#hexa.generate_yaml","title":"hexa.generate_yaml","text":""},{"location":"generate_yaml/#hexa.generate_yaml--summary","title":"Summary","text":"
Generate YAML spec (nested mapping form) from a hexa PortNode tree.
"},{"location":"generate_yaml/#hexa.generate_yaml-classes","title":"Classes","text":""},{"location":"generate_yaml/#hexa.generate_yaml-functions","title":"Functions","text":""},{"location":"generate_yaml/#hexa.generate_yaml.generate_yaml","title":"generate_yaml","text":"
generate_yaml(\n    node: PortNode, out_path: str | Path | None = None\n) -> str\n
 
Generate YAML source from a :class:PortNode tree.
 
Parameters:
 Name Type Description Default node PortNode 
The root of the tree.
 required out_path str | Path | None 
Optional file path to write the generated source to.
 None 
Returns:
 Name Type Description str str 
The generated YAML source.
"},{"location":"model/","title":"Model","text":""},{"location":"model/#hexa.model","title":"hexa.model","text":""},{"location":"model/#hexa.model--summary","title":"Summary","text":"
Shared neutral model for hexa port trees.
 
All utilities (parse_yaml, generate_abc, parse_abc, generate_yaml, check_matches) serialize through this single model.
"},{"location":"model/#hexa.model--notes","title":"Notes","text":"
     
  • The model is NOT tied to any sample \u2014 any port tree fits.
    •  
"},{"location":"model/#hexa.model-classes","title":"Classes","text":""},{"location":"model/#hexa.model.MethodSpec","title":"MethodSpec  dataclass","text":"
MethodSpec(\n    name: str,\n    args: list[str] = list(),\n    kwargs: dict[str, str] = dict(),\n)\n
 
Specification of a single abstract method.
 
Attributes:
 Name Type Description name str 
Name of the method.
 args list[str] 
Positional parameter names.
 kwargs dict[str, str] 
Keyword-only parameter names mapped to their type expressions.
"},{"location":"model/#hexa.model.PortNode","title":"PortNode  dataclass","text":"
PortNode(\n    name: str,\n    port_cls: str,\n    kind: str = \"leaf\",\n    attributes: dict[str, ConfigValue] = dict(),\n    children: list[PortNode] = list(),\n    methods: dict[str, MethodSpec] = dict(),\n)\n
 
A node in the hexa port tree.
 
Each node is a leaf (no children), a port (has nested port slots), or the parent root of the tree. Every node carries a concrete typed shape derived from an ABC class name.
 
Attributes:
 Name Type Description name str 
Slot name, e.g. \"parser\", \"amount_balance\".
 port_cls str 
ABC class name that gives this port its shape, e.g. \"TransactionParserPort\".  Every port has a concrete typed shape.
 kind str 
One of \"leaf\", \"port\", or \"parent\". - \"leaf\": no children (e.g. NumberPort) - \"port\": has children (e.g. TransactionParserPort) - \"parent\": the root of the tree (e.g. ExtractionPipeline)
 attributes dict[str, ConfigValue] 
Annotated attributes: name -> type_expr for scalars, or a nested dict of sub-attributes (a struct) for grouped values.
 children list[PortNode] 
Nested port slots.
 methods dict[str, MethodSpec] 
Abstract methods keyed by method name.
"},{"location":"model/#hexa.model.PortNode-functions","title":"Functions","text":""},{"location":"model/#hexa.model.PortNode.diff","title":"diff","text":"
diff(other: PortNode, path: str = '') -> list[str]\n
 
Return a list of human-readable difference strings.
 
An empty list means the trees are equal.
 
Parameters:
 Name Type Description Default other PortNode 
Tree to compare against.
 required path str 
Path prefix used when recursing into nested nodes; defaults to the root path.
 '' 
Returns:
 Type Description list[str] 
list[str]: Human-readable difference strings, one per discrepancy.
"},{"location":"model/#hexa.model.PortNode.find","title":"find","text":"
find(name: str) -> PortNode | None\n
 
Find a direct child by slot name.
 
Parameters:
 Name Type Description Default name str 
Slot name of the child to look up.
 required 
Returns:
 Type Description PortNode | None 
PortNode | None: The matching child node, or None if no child has that name.
"},{"location":"model/#hexa.model.PortNode.walk","title":"walk","text":"
walk() -> list[PortNode]\n
 
Return all nodes in depth-first order (self first).
 
Returns:
 Type Description list[PortNode] 
list[PortNode]: All nodes of the tree, self first, depth-first.
"},{"location":"parse_abc/","title":"Parse Abc","text":""},{"location":"parse_abc/#hexa.parse_abc","title":"hexa.parse_abc","text":""},{"location":"parse_abc/#hexa.parse_abc--summary","title":"Summary","text":"
Parse an ABC Python module into a hexa PortNode tree.
 
ABC classes become port nodes.  A nested-struct attribute is expressed as a @dataclass annotation on an ABC port (e.g. source: ExtractionSource where ExtractionSource is a @dataclass); the dataclass's fields are expanded into a nested attributes entry.
"},{"location":"parse_abc/#hexa.parse_abc--notes","title":"Notes","text":"
     
  • Accepts a file path, a module import string, or an already-loaded module.
    •  
"},{"location":"parse_abc/#hexa.parse_abc-classes","title":"Classes","text":""},{"location":"parse_abc/#hexa.parse_abc-functions","title":"Functions","text":""},{"location":"parse_abc/#hexa.parse_abc.parse_abc","title":"parse_abc","text":"
parse_abc(module_or_path: str | Path | Any) -> PortNode\n
 
Parse an ABC Python module into a :class:PortNode tree.
 
ABC classes become port nodes.  A nested-struct attribute is expressed as a @dataclass annotation on an ABC port (e.g. source: ExtractionSource where ExtractionSource is a @dataclass); the dataclass's fields are expanded into a nested attributes entry.
 
Parameters:
 Name Type Description Default module_or_path str | Path | Any 
A file path to a .py file, a module import string, or a loaded module.
 required 
Returns:
 Name Type Description PortNode PortNode 
The root node of the tree.
"},{"location":"parse_yaml/","title":"Parse Yaml","text":""},{"location":"parse_yaml/#hexa.parse_yaml","title":"hexa.parse_yaml","text":""},{"location":"parse_yaml/#hexa.parse_yaml--summary","title":"Summary","text":"
Parse a hexa YAML spec into a :class:PortNode tree.
 
The YAML grammar uses a nested mapping format built from three member kinds:
 
     
  • attribute: a name: type scalar (e.g. source_type: str)
    •  
  • nested struct: a name: mapping of sub-attributes (e.g. source:)
    •  
  • port: a name: mapping whose single key is a class whose value is a   body, or a class key with a ... body.  A port carries a class shape.
    •  
 Disambiguation of a mapping value 
A mapping with exactly one key whose value is a non-scalar body (a mapping, ..., or a list) is a port.  Any other mapping (multiple keys, or all scalar-typed values) is a nested struct of attributes.
"},{"location":"parse_yaml/#hexa.parse_yaml--examples","title":"Examples","text":"
Parse a spec into a port tree:
 
```python\nfrom hexa import parse_yaml\n\nnode = parse_yaml(\"samples/minimal/sample.yaml\")\nprint(node)\n```\n
 
The YAML grammar:
 
```yaml\nRootClassName:\n  attr: type                 # scalar attribute\n  source:                    # nested struct (all scalar sub-attributes)\n    source_type: str\n    bank: str\n  methods:                   # reserved block of functions\n    run:\n      args: [a, b]\n  ingest:                    # port (single non-scalar class key)\n    IngestPort:\n      source_type: str\n      methods: {Method: ...}\n```\n
"},{"location":"parse_yaml/#hexa.parse_yaml--notes","title":"Notes","text":"
     
  • See samples/minimal/sample.yaml and   samples/extraction_pipeline/extraction_pipeline.yaml for worked examples.
    •  
"},{"location":"parse_yaml/#hexa.parse_yaml-classes","title":"Classes","text":""},{"location":"parse_yaml/#hexa.parse_yaml-functions","title":"Functions","text":""},{"location":"parse_yaml/#hexa.parse_yaml.parse_yaml","title":"parse_yaml","text":"
parse_yaml(path: str | Path) -> PortNode\n
 
Parse a hexa YAML spec file into a :class:PortNode tree.
 
Parameters:
 Name Type Description Default path str | Path 
Path to the .yaml file.
 required 
Returns:
 Name Type Description PortNode PortNode 
The root node of the parsed tree.
 
Raises:
 Type Description ValueError 
On structural errors in the YAML.
"}]}
\ No newline at end of file
diff --git a/hexa/wiki/01_overview/index.html b/hexa/wiki/01_overview/index.html
new file mode 100644
index 0000000..223c22f
--- /dev/null
+++ b/hexa/wiki/01_overview/index.html
@@ -0,0 +1,1163 @@
+
+
+
+  
+    
+      
+      
+      
+      
+      
+      
+        
+      
+      
+        
+      
+      
+      
+      
+    
+    
+      
+        Overview - Hexa Documentation
+      
+    
+    
+      
+      
+        
+        
+      
+      
+
+
+    
+    
+      
+    
+    
+      
+        
+        
+        
+        
+        
+      
+    
+    
+      
+    
+    
+    
+      
+
+    
+    
+    
+  
+  
+  
+    
+    
+    
+    
+    
+    
+  
+    
+    
+    
+    
+    
+    

+      
+    

+    
+    
+      
+
+

+  
+  
+

+    
+    

+      
+      
+        
+          
+            
+
+          
+        
+      
+      

+        

+          
+            
+              
+              

+                

+                  

+                    
+
+
+  
+
+
+  
+
+
+                  

+                

+              

+            
+            
+          
+          
+            

+              

+                
+                  
+
+
+
+
Library Overview

+
hexa lets you describe a hierarchical pipeline as a tree of typed contracts, then get a fully wired instance back from plain class annotations. This page builds the mental model: the three layers, the two interchangeable representations, and the container that reads annotations at runtime.

+

+
🏗️ Architecture

+
Text Only
┌───────────────────────────────────────────────────────────────┐
+│  Representation layer (optional, interchangeable)             │
+│                                                               │
+│   YAML spec (.yaml)  ◄── parse_yaml ──┐                       │
+│        │                             │  PortNode tree         │
+│        │  generate_abc               │  (the single model)    │
+│        ▼                             ▼                        │
+│   ABC module (_abc.py)  ◄─ generate_yaml ─┐                   │
+│   (contracts only: ABC + annotations)    │                    │
+└──────────────────────────────────────────┼────────────────────┘
+                                           │
+┌──────────────────────────────────────────┼────────────────────┐
+│  Runtime layer (annotations are decision)▼                    │
+│                                                               │
+│   root class ── build(cls, instances=, config=) ──► wired     │
+│   (e.g. ImplParentClass)                     instance          │
+│        │  __annotations__ merged across MRO                   │
+│        ▼                                                      │
+│   recursively: instantiate pinned concrete port, setattr on   │
+│   parent, inject runtime instances / config values by name    │
+└───────────────────────────────────────────────────────────────┘
+

+
Two layers, deliberately decoupled:

+
    
+
  1. Representation — the same contract can be written as a YAML spec or as a Python ABC module. hexa's converters translate between them through one neutral PortNode tree. This layer is optional: you may hand-write only the ABC file, only the YAML, or both.
    2. 
+
  2. Runtime — build(cls) reads the merged annotations of a root class. Every port slot annotated with a concrete type is the decision; the container instantiates the tree recursively and setattrs each child onto its parent.
    3. 
+

+

+
🔄 The Three Layers (the mental model)

+
Whatever you author, the pattern is built from three fixed layers:

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
LayerExample fileRole
1. ABC contractssamples/minimal/abc.pyDeclaration of need — attributes, port slots, @abstractmethod bodies. Never runs.
2. Generic Impl* / concrete defaultssamples/minimal/impl.pyShared behavior — subclasses the ABC, implements every @abstractmethod, re-pins slots to concrete types.
3. Specialization (optional)samples/extraction_pipeline/banks/{axis,icici}/pdf.pyVariation — overrides only the members that differ from Impl*.

+

+
The root class you hand to build is always a concrete type (layer 2 or 3). The ABC (layer 1) is only a contract — it is never instantiated.

+

+

+
🔄 Lifecycle Rules

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
StepCallWhy
1Author the contractAs YAML (.yaml) or ABC (_abc.py) — or both, and let check_matches prove they agree.
2Generate one from the other (optional)generate_abc(spec.yaml) when you want ABCs from YAML; generate_yaml(abc.py) for the reverse.
3Write concrete implementations (Impl*)Subclass the ABC, implement abstracts, pin each slot to a concrete port type.
4Wirebuild(ImplRoot, instances={"repo": repo}, config={"retries": 3}).
5Verify (optional)check_matches(spec.yaml, abc.py) returns True or a list of difference strings.

+
Steps 1–3 choose what the pipeline needs; step 4 is automatic. Variation between pipelines happens purely by which root class you build.

+

+
🧭 Which Entry Point Should You Use?

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Entry pointUse whenAdds
parse_yaml(path)You author specs in YAMLYAML → PortNode tree (structural validation only)
generate_abc(node)You want ABC source from a specYAML model → Python ABC module source
parse_abc(mod)You author contracts in PythonABC module → PortNode tree (reflection)
generate_yaml(node)You want a spec for your ABCsPython ABC module → YAML source
check_matches(a, b)You keep both representationsVerifies the two trees agree; prints diffs
build(cls, ...)You want a wired runtime instanceRecursively instantiates the port tree from annotations

+

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+                
+              

+            

+          
+          
+  
+
+
+        

+        
+          
+        
+      

+      
+        
+      
+    

+    

+      

+    

+    
+    
+    
+      
+      
+    
+    
+      
+      
+    
+  
+
\ No newline at end of file
diff --git a/hexa/wiki/02_components/index.html b/hexa/wiki/02_components/index.html
new file mode 100644
index 0000000..d19e3f2
--- /dev/null
+++ b/hexa/wiki/02_components/index.html
@@ -0,0 +1,1255 @@
+
+
+
+  
+    
+      
+      
+      
+      
+      
+      
+        
+      
+      
+        
+      
+      
+      
+      
+    
+    
+      
+        Core Components - Hexa Documentation
+      
+    
+    
+      
+      
+        
+        
+      
+      
+
+
+    
+    
+      
+    
+    
+      
+        
+        
+        
+        
+        
+      
+    
+    
+      
+    
+    
+    
+      
+
+    
+    
+    
+  
+  
+  
+    
+    
+    
+    
+    
+    
+  
+    
+    
+    
+    
+    
+    

+      
+    

+    
+    
+      
+
+

+  
+  
+

+    
+    

+      
+      
+        
+          
+            
+
+          
+        
+      
+      

+        

+          
+            
+              
+              

+                

+                  

+                    
+
+
+  
+
+
+  
+
+
+                  

+                

+              

+            
+            
+          
+          
+            

+              

+                
+                  
+
+
+
+
Core Components

+
A validated reference to the public API of hexa (package root exports from hexa/__init__.py). Signatures match the code exactly. For runnable recipes see the Use Cases. Exact parameter contracts live in the API reference under docs/lib/ — this page is usage-level.

+

+
1. The Shared Model — PortNode and MethodSpec

+
Every parser and generator serializes through one neutral tree. It is not tied to any sample — any port tree fits.

+
1.1 PortNode

+
A node in a hexa port tree. Three kinds:

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
KindMeaningExample
parentThe root of the treeExtractionPipeline
portHas nested port slots (children)TransactionParserPort
leafNo childrenNumberPort

+
Fields:

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
FieldTypeMeaning
namestrSlot name, e.g. "parser", "amount_balance"
port_clsstrABC class name that gives this port its shape
kindstr"leaf" | "port" | "parent"
attributesdict[str, ConfigValue]name → type_expr for scalars, or a nested dict of sub-attributes for grouped values
childrenlist[PortNode]Nested port slots
methodsdict[str, MethodSpec]Abstract methods keyed by method name

+
Tree helpers (used internally and handy for inspection):

+
    
+
  • node.find(name) -> PortNode | None — direct child by slot name.
    • 
+
  • node.walk() -> list[PortNode] — all nodes, depth-first, self first.
    • 
+
  • node.diff(other, path="") -> list[str] — human-readable difference strings; empty list means equal.
    • 
+

+
1.2 MethodSpec

+
Specification of a single abstract method: name, positional args: list[str], and keyword-only kwargs: dict[str, str] (name → type expression). Supports equality (==/!=).

+
1.3 ConfigValue

+
str | NestedConfig, where NestedConfig = dict[str, ConfigValue]. An attribute is either a scalar type expression (e.g. "str") or a nested structure of sub-attributes. Nesting is recursive.

+

+
2. Converters — YAML ⇄ ABC

+
All converters are pure (they operate on the model) and optional: you never have to touch them if you hand-write a single representation.

+
2.1 parse_yaml(path) -> PortNode

+
Parses a hexa YAML spec into a PortNode tree. The grammar uses three member kinds:

+
    
+
  • attribute — name: type scalar (e.g. source_type: str)
    • 
+
  • nested struct — name: mapping of sub-attributes (e.g. source:)
    • 
+
  • port — name: mapping whose single key is a class whose value is a body (e.g. ingest: IngestPort: ...)
    • 
+

+
Disambiguation rule: a mapping with exactly one key whose value is a non-scalar body is a port; any other mapping is a nested struct of attributes. Raises ValueError on structural errors (see Error Handling).

+
2.2 generate_abc(node, out_path=None) -> str

+
Generates _abc.py source from a PortNode tree. Emits class X(ABC) per node, synthesized @dataclasses for nested structs, port-slot annotations, and @abstractmethod stubs. Deterministic (walk order) so output is idempotent. When out_path is given the source is also written there; the source string is always returned.

+
2.3 parse_abc(module_or_path) -> PortNode

+
Reflects an ABC Python module into a PortNode tree. Accepts a .py file path, a module import string, or an already-loaded module. ABC classes become nodes; a @dataclass annotation on a port expands into a nested attributes entry. The root is auto-detected (the class not referenced as any other slot, preferring names containing Pipeline/Root/Parent).

+
2.4 generate_yaml(node, out_path=None) -> str

+
Generates a YAML spec (nested mapping form) from a PortNode tree — the reverse of generate_abc. A deferred/empty slot renders as slot: Class: ....

+

+
3. Verification — check_matches(a, b)

+
check_matches(path_a, path_b) -> bool | list[str] parses two representations (one .yaml/.yml and one .py, either direction) and compares the trees with PortNode.diff. Returns True when they describe the same tree, otherwise a list of human-readable difference strings. Inequality checks cover port_cls, kind, attributes (including nested structs), methods (args/kwargs), and children.

+

+
4. Container — build

+
build(cls, *, instances=None, config=None) -> T recursively instantiates the port tree implied by a root class's annotations.

+
Algorithm:

+
    
+
  1. Merge __annotations__ across the MRO (reverse order) so inherited slot annotations are visible.
    2. 
+
  2. Instantiate the parent class (cls(); if that raises TypeError, retry passing None for every __init__ parameter after self).
    3. 
+
  3. For each annotation:
    4. 
+
  4. A name in instances wins — the provided object is setattred verbatim, never reconstructed.
    5. 
+
  5. Builtin / typing / PEP 604-union types are config fields — applied from config by name.
    6. 
+
  6. Abstract classes are skipped (but take a config value when provided).
    7. 
+
  7. Any other concrete type is a port slot — build recurses and the child is setattred onto the parent.
    8. 
+
  8. Config defaults: any annotated attribute still unset is defaulted to its config value (or None) so methods can safely reference it.
    9. 
+

+

+
Runtime values (repos, handlers, clients) go in instances=; plain config values go in config=. Both propagate by name across the whole tree.

+

+

+
5. CLI

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
CommandBehavior
hexa parse-yaml PATHPrint the parsed PortNode tree.
hexa generate-abc PATH [--out FILE]Emit ABC source from a YAML spec.
hexa parse-abc PATHPrint the tree reflected from an ABC module.
hexa generate-yaml PATH [--out FILE]Emit a YAML spec from an ABC module.
hexa check PATH_A PATH_BPrint Matches./Differences found: and exit 0/1.

+
Errors print Error: <message> to stderr and exit 1.

+

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+                
+              

+            

+          
+          
+  
+
+
+        

+        
+          
+        
+      

+      
+        
+      
+    

+    

+      

+    

+    
+    
+    
+      
+      
+    
+    
+      
+      
+    
+  
+
\ No newline at end of file
diff --git a/hexa/wiki/03_use_cases/01_yaml_first/index.html b/hexa/wiki/03_use_cases/01_yaml_first/index.html
new file mode 100644
index 0000000..91915f7
--- /dev/null
+++ b/hexa/wiki/03_use_cases/01_yaml_first/index.html
@@ -0,0 +1,1110 @@
+
+
+
+  
+    
+      
+      
+      
+      
+      
+      
+        
+      
+      
+        
+      
+      
+      
+      
+    
+    
+      
+        YAML First - Hexa Documentation
+      
+    
+    
+      
+      
+        
+        
+      
+      
+
+
+    
+    
+      
+    
+    
+      
+        
+        
+        
+        
+        
+      
+    
+    
+      
+    
+    
+    
+      
+
+    
+    
+    
+  
+  
+  
+    
+    
+    
+    
+    
+    
+  
+    
+    
+    
+    
+    
+    

+      
+    

+    
+    
+      
+
+

+  
+  
+

+    
+    

+      
+      
+        
+          
+            
+
+          
+        
+      
+      

+        

+          
+            
+              
+              

+                

+                  

+                    
+
+
+  
+
+
+  
+
+
+                  

+                

+              

+            
+            
+          
+          
+            

+              

+                
+                  
+
+
+
+
Use Case 1: YAML-First Contract Authoring

+
Scenario: you design the contract as a YAML spec, generate ABC Python source from it, hand-write concrete implementations, and prove the spec and code agree.

+

+
📦 What's New?

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
ComponentDescription
parse_yamlReads the spec into the neutral PortNode tree.
generate_abcEmits _abc.py source (contracts only) from the tree.
check_matchesVerifies the spec and the ABC module describe the same tree.

+

+
🚀 Example

+
Start from the minimal sample spec (samples/minimal/sample.yaml):

+
YAML
ParentClass:
+  config_var1: str
+  config_var2: int
+  methods:
+    run:
+      args: [arg1, arg2]
+      kwargs: {kwarg1: int, kwarg2: str}
+  port1:
+    NonHexaChildClass1:
+      config_var11: str
+      config_var12: int
+      methods: {method1: {args: [arg1, arg2], kwargs: {kwarg1: int, kwarg2: str}}}
+  port2:
+    HexaChildClass2:
+      config_var21: str
+      config_var22: int
+      methods: {method1: {args: [arg1, arg2], kwargs: {kwarg1: int, kwarg2: str}}}
+      port1:
+        NonHexaPort1:
+          config_var11: str
+          config_var12: int
+          methods: {method1: {args: [arg1, arg2], kwargs: {kwarg1: int, kwarg2: str}}}
+

+
Generate the ABC module from the spec:

+
Bash
hexa generate-abc samples/minimal/sample.yaml --out spec_abc.py
+

+
The generated file declares the contracts only — attributes, port slots, and abstract methods:

+
Python
class ParentClass(ABC):
+    config_var1: str
+    config_var2: int
+
+    port1: NonHexaChildClass1
+    port2: HexaChildClass2
+    port3: HexaChildClass3
+
+    @abstractmethod
+    def run(self, arg1, arg2, *, kwarg1: int, kwarg2: str): ...
+

+
Now write the concrete implementation (hand-written — behavior and slot pinning are human decisions):

+
Python
class ImplParentClass(ParentClass):
+    port1: NonHexaChildClass1
+    port2: HexaChildClass2
+    port3: HexaChildClass3
+
+    def run(self, arg1, arg2, *, kwarg1: int = 0, kwarg2: str = ""):
+        return {
+            "port1": self.port1.method1(arg1, arg2, kwarg1=kwarg1, kwarg2=kwarg2),
+            "port2": self.port2.method1(arg1, arg2, kwarg1=kwarg1, kwarg2=kwarg2),
+            "port3": self.port3.method1(arg1, arg2, kwarg1=kwarg1, kwarg2=kwarg2),
+        }
+

+
Finally, guard the two representations against drift:

+
Bash
hexa check samples/minimal/sample.yaml samples/minimal/abc.py
+# Matches.   (exit 0)
+

+

+
💡 Tips

+
    
+
  • generate_abc emits contracts only — it never generates Impl*/specialization files. Slot pinning and method bodies are your call.
    • 
+
  • The YAML grammar distinguishes a port (a mapping whose single key has a non-scalar body) from a nested struct (any other mapping). Keep exactly one class key per port slot.
    • 
+
  • check_matches compares classes, attributes, slots, and @abstractmethod signatures; missing keyword-only params are treated conservatively so minor refactors don't false-negative.
    • 
+
  • All generators are deterministic (walk order), so regenerating a spec from a module and diffing is stable.
    • 
+

+

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+                
+              

+            

+          
+          
+  
+
+
+        

+        
+          
+        
+      

+      
+        
+      
+    

+    

+      

+    

+    
+    
+    
+      
+      
+    
+    
+      
+      
+    
+  
+
\ No newline at end of file
diff --git a/hexa/wiki/03_use_cases/02_abc_first/index.html b/hexa/wiki/03_use_cases/02_abc_first/index.html
new file mode 100644
index 0000000..4e331fa
--- /dev/null
+++ b/hexa/wiki/03_use_cases/02_abc_first/index.html
@@ -0,0 +1,1088 @@
+
+
+
+  
+    
+      
+      
+      
+      
+      
+      
+        
+      
+      
+        
+      
+      
+      
+      
+    
+    
+      
+        ABC First - Hexa Documentation
+      
+    
+    
+      
+      
+        
+        
+      
+      
+
+
+    
+    
+      
+    
+    
+      
+        
+        
+        
+        
+        
+      
+    
+    
+      
+    
+    
+    
+      
+
+    
+    
+    
+  
+  
+  
+    
+    
+    
+    
+    
+    
+  
+    
+    
+    
+    
+    
+    

+      
+    

+    
+    
+      
+
+

+  
+  
+

+    
+    

+      
+      
+        
+          
+            
+
+          
+        
+      
+      

+        

+          
+            
+              
+              

+                

+                  

+                    
+
+
+  
+
+
+  
+
+
+                  

+                

+              

+            
+            
+          
+          
+            

+              

+                
+                  
+
+
+
+
Use Case 2: ABC-First Contract Authoring

+
Scenario: you prefer to author contracts as Python ABC modules, then emit — and verify — a YAML spec for them.

+

+
📦 What's New?

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
ComponentDescription
parse_abcReflects an ABC module into the neutral PortNode tree.
generate_yamlEmits a YAML spec from the tree.
check_matchesVerifies the two representations agree (reverse direction).

+

+
🚀 Example

+
Author the contract in Python (see samples/minimal/abc.py). A nested-struct attribute is expressed as a @dataclass annotation on an ABC port; the dataclass fields expand into a nested attributes entry when parsed:

+
Python
from abc import ABC, abstractmethod
+
+
+class NonHexaPort1(ABC):
+    config_var11: str
+    config_var12: int
+
+    @abstractmethod
+    def method1(self, arg1, arg2, *, kwarg1: int = 0, kwarg2: str = "") -> str: ...
+
+
+class ParentClass(ABC):
+    config_var1: str
+    config_var2: int
+
+    port1: NonHexaChildClass1
+    port2: HexaChildClass2
+    port3: HexaChildClass3
+
+    @abstractmethod
+    def run(self, arg1, arg2, *, kwarg1: int = 0, kwarg2: str = "") -> dict: ...
+

+
Emit the YAML spec from the module:

+
Bash
hexa generate-yaml samples/minimal/abc.py --out spec.yaml
+

+
Inspect the parsed tree to make sure the root was detected correctly:

+
Bash
hexa parse-abc samples/minimal/abc.py
+# PortNode(name='ParentClass', port_cls='ParentClass', kind='parent', ...)
+

+
And verify the spec against the module:

+
Bash
hexa check spec.yaml samples/minimal/abc.py
+# Matches.   (exit 0)
+

+

+
💡 Tips

+
    
+
  • Root detection: parse_abc picks the class that is not referenced as any other slot; on ties it prefers names containing Pipeline, Root, or Parent.
    • 
+
  • parse_abc accepts a .py path, an import string (e.g. samples.minimal.abc), or an already-loaded module, so it composes well in scripts.
    • 
+
  • A @dataclass referenced from a port becomes a nested struct; a dataclass field that references an ABC becomes a nested port slot keyed under the struct.
    • 
+
  • YAML generated this way is exactly the grammar parse_yaml understands — the two directions round-trip (node1.diff(node3) == [] in the repo's roundtrip tests).
    • 
+

+

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+                
+              

+            

+          
+          
+  
+
+
+        

+        
+          
+        
+      

+      
+        
+      
+    

+    

+      

+    

+    
+    
+    
+      
+      
+    
+    
+      
+      
+    
+  
+
\ No newline at end of file
diff --git a/hexa/wiki/03_use_cases/03_wiring_runtime/index.html b/hexa/wiki/03_use_cases/03_wiring_runtime/index.html
new file mode 100644
index 0000000..aaf9535
--- /dev/null
+++ b/hexa/wiki/03_use_cases/03_wiring_runtime/index.html
@@ -0,0 +1,1113 @@
+
+
+
+  
+    
+      
+      
+      
+      
+      
+      
+        
+      
+      
+        
+      
+      
+      
+      
+    
+    
+      
+        Wiring & Runtime - Hexa Documentation
+      
+    
+    
+      
+      
+        
+        
+      
+      
+
+
+    
+    
+      
+    
+    
+      
+        
+        
+        
+        
+        
+      
+    
+    
+      
+    
+    
+    
+      
+
+    
+    
+    
+  
+  
+  
+    
+    
+    
+    
+    
+    
+  
+    
+    
+    
+    
+    
+    

+      
+    

+    
+    
+      
+
+

+  
+  
+

+    
+    

+      
+      
+        
+          
+            
+
+          
+        
+      
+      

+        

+          
+            
+              
+              

+                

+                  

+                    
+
+
+  
+
+
+  
+
+
+                  

+                

+              

+            
+            
+          
+          
+            

+              

+                
+                  
+
+
+
+
Use Case 3: Wiring & Runtime Injection

+
Scenario: you have concrete classes with annotated slots and you want a fully wired instance — including runtime values (repos, clients) and config values (retries, labels, thresholds) — without writing any assembly code.

+

+
📦 What's New?

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
ComponentDescription
buildRecursively instantiates the port tree from annotations.
instances=Inject objects verbatim (never reconstructed) by name.
config=Propagate plain config values by name across the whole tree.

+

+
🚀 Example

+
This mirrors tests/test_container.py exactly. Define leaf, mid-level, and root classes purely with annotations:

+
Python
from hexa import build
+
+
+class _Repo:
+    def __init__(self):
+        self.name = "injected-repo"
+
+
+class _Leaf:
+    field: int = 0
+
+
+class _Mid:
+    leaf: _Leaf
+    threshold: float = 0.0
+
+
+class _Root:
+    repo: _Repo
+    mid: _Mid
+    retries: int = 0
+    label: str = ""
+
+
+# Provide a runtime instance for a slot, and config values by name.
+repo = _Repo()
+root = build(
+    _Root,
+    instances={"repo": repo},
+    config={"retries": 3, "label": "fast", "threshold": 2.5},
+)
+
+assert root.repo is repo            # injected verbatim, not reconstructed
+assert root.retries == 3            # config applied
+assert root.label == "fast"
+assert root.mid.leaf is not None    # nested ports built recursively
+assert root.mid.threshold == 2.5    # config propagates by name into nested nodes
+

+
Without any overrides, the same annotations still produce a working tree — every slot is instantiated from its annotated type and config fields keep their class defaults:

+
Python
root = build(_Root)
+assert isinstance(root.repo, _Repo)
+assert isinstance(root.mid, _Mid)
+assert isinstance(root.mid.leaf, _Leaf)
+assert root.retries == 0 and root.label == ""
+

+
Optional / typing-generic config fields are handled the same way — a str | None annotated attribute takes its config value when provided and defaults to the class default otherwise:

+
Python
class _OptionalConfig:
+    trust_fallback: str | None = None
+    starters: list | None = None
+    port: _Leaf
+
+
+built = build(_OptionalConfig, config={"trust_fallback": "amount"})
+assert built.trust_fallback == "amount"
+assert built.starters is None        # unrelated optional field keeps its default
+

+

+
💡 Tips

+
    
+
  • instances= wins over reconstruction. If a name is in instances, its object is setattred verbatim at every node that annotates it. Use it for things that must not be re-created (repos, handlers, clients).
    • 
+
  • config= targets non-port annotated fields. Builtin, typing, and PEP 604-union (X | None) annotated fields are treated as config, applied by name and propagated to every node.
    • 
+
  • Abstract classes are never instantiated; they take a config value if one is provided (useful for defaults on CLI-style options).
    • 
+
  • Only concrete annotated types become wired ports. String forward references are skipped (the container assumes Impl files pin actual types).
    • 
+
  • Port instances are wired by setattr(parent, slot, child) — so methods can reference self.portX safely once the root is built.
    • 
+
  • If cls() raises TypeError, build retries by passing None for each declared __init__ parameter — convenient for classes with optional constructor args.
    • 
+

+

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+                
+              

+            

+          
+          
+  
+
+
+        

+        
+          
+        
+      

+      
+        
+      
+    

+    

+      

+    

+    
+    
+    
+      
+      
+    
+    
+      
+      
+    
+  
+
\ No newline at end of file
diff --git a/hexa/wiki/04_design/index.html b/hexa/wiki/04_design/index.html
new file mode 100644
index 0000000..c0e5c25
--- /dev/null
+++ b/hexa/wiki/04_design/index.html
@@ -0,0 +1,1373 @@
+
+
+
+  
+    
+      
+      
+      
+      
+      
+      
+        
+      
+      
+        
+      
+      
+      
+      
+    
+    
+      
+        Design - Hexa Documentation
+      
+    
+    
+      
+      
+        
+        
+      
+      
+
+
+    
+    
+      
+    
+    
+      
+        
+        
+        
+        
+        
+      
+    
+    
+      
+    
+    
+    
+      
+
+    
+    
+    
+  
+  
+  
+    
+    
+    
+    
+    
+    
+  
+    
+    
+    
+    
+    
+    

+      
+    

+    
+    
+      
+
+

+  
+  
+

+    
+    

+      
+      
+        
+          
+            
+
+          
+        
+      
+      

+        

+          
+            
+              
+              

+                

+                  

+                    
+
+
+  
+
+
+  
+
+
+                  

+                

+              

+            
+            
+          
+          
+            

+              

+                
+                  
+
+
+
+
Design — Extraction Pipeline (Axis / Icici)

+

+
Note: The extraction pipeline discussed below is a sample/example illustrating the hexa composition pattern.

+

+
This document is the concrete counterpart to Philosophy: where the files are, what the tree looks like for both banks, and how a future container instantiates it. Self-sufficient for a fresh agent.

+

+
File map (samples/extraction_pipeline/)

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
FileLayerRole
extraction_pipeline_abc.pyABC contractsAll port ABCs + ExtractionPipeline root. Declares attributes, the ExtractionSource dataclass, port slots, @abstractmethods.
extraction_pipeline_impl.pyGeneric Impl*Concrete default implementations of every port; ImplExtractionPipeline root; clean_num helper.
extraction_pipeline_ambiguity.pyShared logicPorted ResolutionContext, AmbiguityHandler, AmountBalanceNotFound, validate_signs — used by ImplAmountBalancePort.
extraction_pipeline_banks.pyComposition rootAxisExtractionPipeline(ImplExtractionPipeline) + IciciExtractionPipeline (each re-pins txn_dicts).
banks/axis/pdf.pyBank specializationAxisNumberPort, AxisDescPort, AxisAmountBalancePort, AxisTransactionParserPort, AxisTxnDictsPort.
banks/icici/pdf.pyBank specializationIcici* mirror of the above + Icici-only adjust_balance/missing_number_candidates.
extraction_pipeline.yamlContract specDeclarative spec of the port tree; interchangeable with _abc.py via the optional utilities.

+

+
The port tree (ABC contract)

+
Text Only
ExtractionPipeline
+├── source / pipeline / trust_fallback          [attributes; source is a nested struct]
+├── ingest      → IngestPort                    (source → content)
+├── raw_lines   → RawLinesPort                  (content → raw_lines)
+├── txn_blocks  → TxnBlocksPort                 (raw_lines → txn_blocks)
+│   ├── bucket_finder → BucketFinderPort
+│   └── splitter      → SplitterPort
+├── txn_dicts   → TxnDictsPort                  (txn_blocks → txn_dicts)
+│   └── parser  → TransactionParserPort
+│       ├── number         → NumberPort
+│       ├── date           → DatePort
+│       ├── desc           → DescPort
+│       └── amount_balance → AmountBalancePort
+│           ├── number    → NumberPort
+│           └── ambiguity → AmbiguityPort
+└── raw_expense → RawExpensePort                (txn_dicts → list[RawExpense], terminal)
+

+

+
Generic default (Impl*)

+
ImplExtractionPipeline pins every stage slot to a concrete Impl* port:

+
Text Only
ImplExtractionPipeline
+├── ingest:      ImplIngestPort
+├── raw_lines:   ImplRawLinesPort
+├── txn_blocks:  ImplTxnBlocksPort
+│   ├── bucket_finder: ImplBucketFinderPort
+│   └── splitter:      ImplSplitterPort
+├── txn_dicts:   ImplTxnDictsPort
+│   └── parser:  ImplTransactionParserPort
+│       ├── number:        ImplNumberPort
+│       ├── date:          ImplDatePort
+│       ├── desc:          ImplDescPort
+│       └── amount_balance: ImplAmountBalancePort
+│           ├── number:    ImplNumberPort
+│           └── ambiguity: ImplAmbiguityPort
+└── raw_expense: ImplRawExpensePort
+

+

+
Bank variation (only what differs)

+
The root class re-pins only txn_dicts. Everything else inherits Impl*.

+
Axis (banks/axis/pdf.py)

+
Python
class AxisExtractionPipeline(ImplExtractionPipeline):
+    txn_dicts: AxisTxnDictsPort
+

+
Python
AxisTxnDictsPort(ImplTxnDictsPort)
+  └─ parser: AxisTransactionParserPort(ImplTransactionParserPort)
+      ├─ number:  AxisNumberPort          # NUMBER_RE, all clean numbers
+      ├─ desc:    AxisDescPort            # STARTTERS = [upi/, imps/, neft-, neft/, ach/, ach-, 2a/]
+      └─ amount_balance: AxisAmountBalancePort(ImplAmountBalancePort)
+          └─ number: AxisNumberPort       # re-pin back to axis number port
+

+
Notes:

+
    
+
  • date is not redeclared on AxisTransactionParserPort — it inherits ImplDatePort (correct already, adds no behavior).
    • 
+
  • amount_balance.ambiguity is not redeclared — inherits ImplAmbiguityPort.
    • 
+

+
Icici (banks/icici/pdf.py)

+
Python
class IciciExtractionPipeline(ImplExtractionPipeline):
+    txn_dicts: IciciTxnDictsPort
+

+
Python
IciciTxnDictsPort(ImplTxnDictsPort)
+  └─ parser: IciciTransactionParserPort(ImplTransactionParserPort)
+      ├─ number:  IciciNumberPort         # NUMBER_RE, returns decimals[-2:] (last 2)
+      ├─ desc:    IciciDescPort           # STARTTERS + fix_ocr (UPl→UPI, ..., ICIC]→ICICI)
+      └─ amount_balance: IciciAmountBalancePort(ImplAmountBalancePort)
+          ├─ number: IciciNumberPort      # re-pin to icici number port
+          ├─ adjust_balance:              # neg-markers [(-), (-}, {-}, {-) , ()] → -abs(balance)
+          └─ missing_number_candidates:   # Icici-only one-number hook
+

+

+
Redundancy rule (applied in both banks)

+
Only redeclare a slot you are actually changing from Impl*. If a slot's behavior is already correct from the parent Impl*, omit it. This keeps the diff between a bank and Impl* minimal and unambiguous.

+

+
date: ImplDatePort was previously redeclared redundantly and has been removed — it added mental load on the extender for zero behavior change. (Fidelity audit: it is still concrete ImplDatePort in ImplTransactionParserPort, so behavior is unchanged.)

+

+

+
Container (build(cls) in hexa/container.py)

+
The annotations are metadata; a small reflection builder turns a root class into a wired instance:

+
Text Only
build(AxisExtractionPipeline)  →  wired AxisExtractionPipeline
+

+
Algorithm (build(cls)):

+
    
+
  1. Walk cls.__annotations__, merged across the MRO so inherited slot annotations are visible (a bank subclass only adds/overrides a slot; the rest come from Impl*).
    2. 
+
  2. For each port slot whose annotated type is a concrete Impl*/bank class (not an ABC, not a builtin), instantiate it.
    3. 
+
  3. Recurse into that child — repeat until leaves (classes that declare no further port slots).
    4. 
+
  4. setattr(parent, slot, child) to wire each slot onto the parent instance.
    5. 
+
  5. Apply config defaults from the class-level annotated attributes.
    6. 
+

+
Selection is fully static: the annotated type is the chosen implementation. The container holds no per-bank branch logic — Axis vs Icici is decided purely by which root class you call build on.

+
Control flow at runtime (target)

+
A leaf-stage port exposes one entry method; stages chain data through the shared run(source) signature on the root:

+
Text Only
run(source)
+  ingest.materialize(source)            → content
+  raw_lines.extract(content)            → raw_lines
+  txn_blocks.extract(raw_lines)         → {opening_balance: [blocks]}
+  txn_dicts.extract(txn_blocks)         → [txn dicts]
+  raw_expense.build(txn_dicts)          → [RawExpense]   (terminal)
+

+
The stage ports map naturally onto dagpipe Nodes (ingest, raw_lines, txn_blocks, txn_dicts, raw_expense), each node's resolve delegating to its wired port subtree, connected as a dagpipe Graph and run via Engine. That merge is a separate, future step.

+

+
Utilities (optional, hexa-agnostic)

+
The hexa utilities are implemented in hexa/ (the package root) and they are agnostic of the extraction pipeline sample. It ships a set of optional code-gen / validation utilities so users can choose their own workflow — they are never required. A user is free to:

+
    
+
  • hand-write an ABC file and never touch YAML, or
    • 
+
  • start from YAML and generate the ABC file, or
    • 
+
  • start from an ABC file and emit YAML for it, or
    • 
+
  • use the ABC/YAML only as a spec (neither generated from the other).
    • 
+

+
The utilities only ever translate between two interchangeable representations of the same contract — the .yaml spec and the _abc.py module. They never generate or manage the Impl*/bank files (those carry behavior and slot choice, which is a human decision).

+
Shared model

+
All utilities serialize through one neutral model, a PortNode tree. It is NOT tied to the extraction sample — any port tree fits.

+
Python
@dataclass
+class PortNode:
+    name: str                 # slot name, e.g. "parser", "amount_balance"
+    kind: str                 # "leaf" | "port" | "parent"
+    port_cls: str             # ABC class name, e.g. "TransactionParserPort"
+    attributes: dict[str, ConfigValue]  # annotated attributes: name -> type expr,
+                              # or a nested struct dict for grouped values
+    children: list["PortNode"]
+    methods: dict[str, MethodSpec]   # abstract methods
+

+
1. parse_yaml(path) -> PortNode

+
Read a YAML spec into the PortNode tree (YAML → model).

+
    
+
  • Parses completely (structural validation only, no type-checking).
    • 
+
  • Supports the sample grammar (see extraction_pipeline.yaml): a root class whose inline attributes (scalars and nested structs), port slots, and reserved methods: recurse into child ports.
    • 
+
  • Raises ValueError with file/line on structural errors.
    • 
+

+
2. generate_abc(path_yaml, out_path=None) -> str

+
Generate _abc.py source from a YAML spec (YAML → ABC).

+
    
+
  • parse_yaml(...) → write one class <PortCls>(ABC) per PortNode with attribute annotations (nested structs become synthesized @dataclasses), port-slot annotations, and @abstractmethod stubs, matching the style of the hand-written extraction_pipeline_abc.py.
    • 
+
  • Deterministic (YAML-order) output so it is idempotent / diff-friendly.
    • 
+
  • out_path optional; always returns the generated source.
    • 
+

+
3. parse_abc(module_or_path) -> PortNode

+
Reflect over an ABC module into the PortNode tree (ABC → model).

+
    
+
  • Reads __annotations__ / @abstractmethod from generic ABC classes (no sample assumptions) to reconstruct the same PortNode tree.
    • 
+

+
4. generate_yaml(module_or_path, out_path=None) -> str

+
Generate a .yaml spec from an ABC module (ABC → YAML).

+
    
+
  • parse_abc(...) → write the reverse of generate_abc.
    • 
+
  • Lets a user author the ABC file first and emit YAML for it.
    • 
+

+
5. check_matches(path_a, path_b) -> bool

+
Verify two representations agree — YAML vs ABC, either direction.

+
    
+
  • Builds both PortNode models (parse_yaml + parse_abc) and compares: classes exist and are ABCs, attributes match, port slots match, @abstractmethod signatures match (missing keyword-only params treated conservatively).
    • 
+
  • Returns True only if the whole tree matches; otherwise False (or a diff).
    • 
+

+
Workflows (user's choice)

+
Text Only
# YAML-first
+edit spec.yaml
+  → generate_abc(spec.yaml, abc.py)          # regenerate ABCs
+  → hand-write/update Impl* + banks
+  → check_matches(spec.yaml, abc.py) == True # guard
+
+# ABC-first
+edit abc.py
+  → generate_yaml(abc.py, spec.yaml)         # emit YAML spec
+  → check_matches(spec.yaml, abc.py) == True
+
+# hand-written both (no utilities at all)
+edit abc.py + spec.yaml independently        # optional check_matches as a guard
+

+
Impl*/bank files are always hand-extended on top and are never generated.

+

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+                
+              

+            

+          
+          
+  
+
+
+        

+        
+          
+        
+      

+      
+        
+      
+    

+    

+      

+    

+    
+    
+    
+      
+      
+    
+    
+      
+      
+    
+  
+
\ No newline at end of file
diff --git a/hexa/lib/philosophy/index.html b/hexa/wiki/05_philosophy/index.html
similarity index 54%
rename from hexa/lib/philosophy/index.html
rename to hexa/wiki/05_philosophy/index.html
index 5ba1403..e45d0ad 100644
--- a/hexa/lib/philosophy/index.html
+++ b/hexa/wiki/05_philosophy/index.html
@@ -9,6 +9,10 @@
       
       
       
+        
+      
+      
+        
       
       
       
@@ -16,7 +20,7 @@
     
     
       
-        hexa — Philosophy - hexa
+        Philosophy - Hexa Documentation
       
     
     
@@ -37,13 +41,11 @@
         
         
         
-        
-        
+        
+        
       
     
     
-      
-    
       
     
     
@@ -58,12 +60,10 @@
   
     
     
-      
     
     
     
-    
-    
+    
   
     
     
@@ -72,7 +72,7 @@
     

       
         
-        
+        
           Skip to content
         
       
@@ -84,11 +84,9 @@
     
       
 
-  
-
-

+

   

-                

-              
-            
-            
-              
-              

-                

-                  

-                    
 
 
                   

                 

               

             
+            
           
           
             

@@ -891,120 +956,80 @@
 
 
 
-
hexa — Philosophy

-
This document explains the why behind the type-declared dependency pattern used
-for the extraction pipeline sample — specifically the Axis/Icici bank variation. It is,
-together with design.md, self-sufficient: a fresh agent should be able
-to pick up the codebase from here and reason about (and extend) the pattern without
-needing the original session context.

-
One idea

+
Philosophy — Why Type-Declared Dependencies

+
This document explains the why behind the type-declared dependency pattern used for the extraction pipeline sample — specifically the Axis/Icici bank variation. It is, together with Design, self-sufficient: a fresh agent should be able to pick up the codebase from here and reason about (and extend) the pattern without needing the original session context.

+

+
One Idea

 

 
A concrete class's annotated slot type is the dependency decision.

 

-
Bank variation is expressed entirely as class annotations — never as imperative
-wiring, never as __init__ parameters, never as a composition-root that assembles
-objects by hand. The tree of ports a pipeline needs is declared once, and a bank
-differs from the generic pipeline only by which concrete types are pinned onto
-which slots.

-
The three layers

-
The pattern is built from three fixed layers. Understanding which layer something
-belongs to is the whole mental model.

-
Layer 1 — ABC contracts (samples/extraction_pipeline/extraction_pipeline_abc.py)

+
Bank variation is expressed entirely as class annotations — never as imperative wiring, never as __init__ parameters, never as a composition-root that assembles objects by hand. The tree of ports a pipeline needs is declared once, and a bank differs from the generic pipeline only by which concrete types are pinned onto which slots.

+

+
The Three Layers

+
The pattern is built from three fixed layers. Understanding which layer something belongs to is the whole mental model.

+
Layer 1 — ABC contracts (samples/extraction_pipeline/extraction_pipeline_abc.py)

 
Authoritative declaration of need ("WHAT a stage requires").

 
    
 
  • Each port is an ABC.
    • 
-
  • It declares attributes as class-level annotated defaults
-  (e.g. TransactionParserPort.min_numbers: int = 3), optionally grouped as a
-  nested struct via a @dataclass (e.g. source: ExtractionSource).
    • 
-
  • It declares port slots as annotated attributes whose type is another ABC
-  (e.g. TransactionParserPort.number: NumberPort, amount_balance: AmountBalancePort).
    • 
-
  • It declares @abstractmethod bodies — the method signatures each concrete
-  implementation must provide.
    • 
+
  • It declares attributes as class-level annotated defaults (e.g. TransactionParserPort.min_numbers: int = 3), optionally grouped as a nested struct via a @dataclass (e.g. source: ExtractionSource).
    • 
+
  • It declares port slots as annotated attributes whose type is another ABC (e.g. TransactionParserPort.number: NumberPort, amount_balance: AmountBalancePort).
    • 
+
  • It declares @abstractmethod bodies — the method signatures each concrete implementation must provide.
    • 
 

-
The ABC never runs, never holds an instance, and never says which implementation
-to use. It only says what the shape is.

-
Layer 2 — generic concurrent implementations (samples/extraction_pipeline/extraction_pipeline_impl.py)

-
The common default behavior ("Imple — the shared adaptor").

+
The ABC never runs, never holds an instance, and never says which implementation to use. It only says what the shape is.

+
Layer 2 — Generic concurrent implementations (samples/extraction_pipeline/extraction_pipeline_impl.py)

+
The common default behavior ("Impl* — the shared adaptor").

 
    
-
  • class ImplTransactionParserPort(TransactionParserPort) subclasses the ABC,
-  implements every @abstractmethod, and re-pins its port slots to concrete
-  Impl* types: number: ImplNumberPort, desc: ImplDescPort,
-  amount_balance: ImplAmountBalancePort.
    • 
-
  • ImplAmountBalancePort in turn pins number: ImplNumberPort and
-  ambiguity: ImplAmbiguityPort.
    • 
-
  • The root, ImplExtractionPipeline, pins its five stage slots to Impl* ports:
-  ingest, raw_lines, txn_blocks, txn_dicts, raw_expense.
    • 
+
  • class ImplTransactionParserPort(TransactionParserPort) subclasses the ABC, implements every @abstractmethod, and re-pins its port slots to concrete Impl* types: number: ImplNumberPort, desc: ImplDescPort, amount_balance: ImplAmountBalancePort.
    • 
+
  • ImplAmountBalancePort in turn pins number: ImplNumberPort and ambiguity: ImplAmbiguityPort.
    • 
+
  • The root, ImplExtractionPipeline, pins its five stage slots to Impl* ports: ingest, raw_lines, txn_blocks, txn_dicts, raw_expense.
    • 
 

-
Impl* is the default that shared, bank-agnostic behavior lives in. Any slot a
-bank does not override falls back to Impl* at runtime.

-
Layer 3 — bank specializations (samples/extraction_pipeline/banks/{axis,icici}/pdf.py, samples/extraction_pipeline/extraction_pipeline_banks.py)

+
Impl* is the default that shared, bank-agnostic behavior lives in. Any slot a bank does not override falls back to Impl* at runtime.

+
Layer 3 — Bank specializations (samples/extraction_pipeline/banks/{axis,icici}/pdf.py, samples/extraction_pipeline/extraction_pipeline_banks.py)

 
The variation ("what differs per bank").

 
    
-
  • Bank classes subclass the concrete Impl* classes, never the ABC — they
-  extend, they never reinvent the contract.
    • 
-
  • Each Axis*/Icici* class overrides only the members that differ from the
-  generic Impl*; everything shared is inherited.
    • 
-
  • A bank root is a thin annotated subclass of ImplExtractionPipeline that
-  re-pins only the slots that differ:
    • 
+
  • Bank classes subclass the concrete Impl* classes, never the ABC — they extend, they never reinvent the contract.
    • 
+
  • Each Axis*/Icici* class overrides only the members that differ from the generic Impl*; everything shared is inherited.
    • 
+
  • A bank root is a thin annotated subclass of ImplExtractionPipeline that re-pins only the slots that differ:
    • 
 

-
1
-2
class AxisExtractionPipeline(ImplExtractionPipeline):
-    txn_dicts: AxisTxnDictsPort
-

-
The re-pinning cascade

-
Bank-specific wiring is hierarchical and cascading. Each level narrows exactly
-one slot, and its narrowed type drags the next level's narrowing along with it:

-
1
-2
-3
-4
-5
-6
-7
AxisExtractionPipeline
-  └─ txn_dicts: AxisTxnDictsPort              (only override on the root)
-      └─ parser: AxisTransactionParserPort
-          ├─ number:  AxisNumberPort          (get_numbers → all clean nums)
-          ├─ desc:    AxisDescPort            (STARTTERS)
-          └─ amount_balance: AxisAmountBalancePort
-              └─ number: AxisNumberPort       (re-pin: same bank's number port)
-

-
Siblings that are already correct in Impl* are inherited, not redeclared.
-E.g. the parser's date slot comes from ImplTransactionParserPort; the
-amount-balance port's ambiguity comes from ImplAmountBalancePort. Re-declaring
-them adds mental load on the extender with zero behavior change.

+
Python
class AxisExtractionPipeline(ImplExtractionPipeline):
+    txn_dicts: AxisTxnDictsPort
+

+

+
The Re-Pinning Cascade

+
Bank-specific wiring is hierarchical and cascading. Each level narrows exactly one slot, and its narrowed type drags the next level's narrowing along with it:

+
Text Only
AxisExtractionPipeline
+  └─ txn_dicts: AxisTxnDictsPort              (only override on the root)
+      └─ parser: AxisTransactionParserPort
+          ├─ number:  AxisNumberPort          (get_numbers → all clean nums)
+          ├─ desc:    AxisDescPort            (STARTTERS)
+          └─ amount_balance: AxisAmountBalancePort
+              └─ number: AxisNumberPort       (re-pin: same bank's number port)
+

+
Siblings that are already correct in Impl* are inherited, not redeclared. E.g. the parser's date slot comes from ImplTransactionParserPort; the amount-balance port's ambiguity comes from ImplAmountBalancePort. Re-declaring them adds mental load on the extender with zero behavior change.

 

-
Rule — only redeclare the slot you are actually changing. If a slot's
-behavior is already correct from Impl*, leave it out.

+
Rule — only redeclare the slot you are actually changing. If a slot's behavior is already correct from Impl*, leave it out.

 

-
The two fidelity guarantees this pattern preserves

+

+
The Two Fidelity Guarantees This Pattern Preserves

 
    
-
  1. Shared behavior stays shared. Slots like ingest, raw_lines, txn_blocks,
-   raw_expense, parser date, and ambiguity are bank-agnostic. Leaving them at
-   the Impl* default is deliberate, not an omission.
    2. 
-
  2. Banks never reinvent. Axis*/Icici* only extend Impl*. The ABC contract
-   in samples/extraction_pipeline/extraction_pipeline_abc.py is defined once and shared by every bank.
    3. 
+
  3. Shared behavior stays shared. Slots like ingest, raw_lines, txn_blocks, raw_expense, parser date, and ambiguity are bank-agnostic. Leaving them at the Impl* default is deliberate, not an omission.
    4. 
+
  4. Banks never reinvent. Axis*/Icici* only extend Impl*. The ABC contract in samples/extraction_pipeline/extraction_pipeline_abc.py is defined once and shared by every bank.
    5. 
 

-
What this pattern is NOT

+

+
What This Pattern Is NOT

 
    
 
  • No imperative assembly. No build_axis_pipeline() wiring objects by hand.
    • 
-
  • No constructor injection. Impl* classes have no __init__; wiring is done by
-  assigning concrete port instances onto annotated slots.
    • 
-
  • YAML and ABCs are interchangeable, and both optional. The .yaml spec and
-  the _abc.py module are two representations of the same contract. hexa is
-  agnostic of the extraction pipeline sample and never imposes a workflow: a user can hand-write the
-  ABC file, start from YAML (ABC generated), start from an ABC file (YAML
-  emitted), or use both as a spec. The optional utilities in
-  design.md § Utilities translate between them and can verify they
-  agree — but the bank Impl*/specialization files are always hand-written
-  (their slots are a manual choice) and never generated.
    • 
+
  • No constructor injection. Impl* classes have no __init__; wiring is done by assigning concrete port instances onto annotated slots.
    • 
+
  • YAML and ABCs are interchangeable, and both optional. The .yaml spec and the _abc.py module are two representations of the same contract. hexa is agnostic of the extraction pipeline sample and never imposes a workflow: a user can hand-write the ABC file, start from YAML (ABC generated), start from an ABC file (YAML emitted), or use both as a spec. The optional utilities in Design § Utilities translate between them and can verify they agree — but the bank Impl*/specialization files are always hand-written (their slots are a manual choice) and never generated.
    • 
+

+

+
Runtime Instantiation

+
The annotations are the decision; hexa.build(cls) (see Design § Container) turns a root class into a wired instance by reading __annotations__ + MRO, recursively instantiating each pinned concrete type, and setattring the child onto the parent slot. This document is scoped to the notation — the rest of the tree can be expressed purely as class annotations and filled at runtime by the container.

+

+
+
-
Runtime instantiation

-
The annotations are the decision; hexa.build(cls) (see design.md §
-Container) turns a root class into a wired instance by reading __annotations__
-+ MRO, recursively instantiating each pinned concrete type, and setattring the
-child onto the parent slot. This document is scoped to the notation — the rest
-of the tree can be expressed purely as class annotations and filled at runtime by
-the container.

 
 
 
@@ -1037,6 +1062,46 @@ the container.