feat: add standards wiki, MCP registry, and app template
This commit is contained in:
84
standards_src/wiki/06_pipeline_scaffold.md
Normal file
84
standards_src/wiki/06_pipeline_scaffold.md
Normal file
@@ -0,0 +1,84 @@
|
||||
# Pipeline Scaffold
|
||||
|
||||
## Summary
|
||||
|
||||
Data-driven processing is expressed as a **composition graph**. The graph is
|
||||
built by a DI library; execution is driven by a DAG engine. Stages are pure and
|
||||
individually testable.
|
||||
|
||||
---
|
||||
|
||||
## When to use a pipeline
|
||||
|
||||
Use a pipeline when a resource is produced by a **multi-step transformation**
|
||||
of an input (a document, a file, a message) rather than by a simple CRUD write.
|
||||
|
||||
## Structure of a family
|
||||
|
||||
A **family** is one input shape (e.g. a file format or a message kind). Each
|
||||
family has the same skeleton:
|
||||
|
||||
```
|
||||
pipelines/<family>/
|
||||
abc.py # port contracts (stage interfaces + root interface)
|
||||
impl.py # concrete stages + root implementation
|
||||
pipeline.yaml # graph spec: nodes, edges, roots, schema
|
||||
resources/
|
||||
pipeline.yaml # runtime spec consumed by the loader
|
||||
patterns/*.yaml # rule fragments (pattern registry)
|
||||
banks/<variant>.py # optional variants that re-pin a few slots
|
||||
```
|
||||
|
||||
## Shared scaffolding
|
||||
|
||||
```
|
||||
pipelines/shared/
|
||||
ports.py # RegistryPort, FinalizePort, etc.
|
||||
loader.py # build(root_cls) -> wires the graph + engine
|
||||
registry.py # resolve_root(...) -> root class for a (kind, variant)
|
||||
```
|
||||
|
||||
## The loader
|
||||
|
||||
```python
|
||||
def build(root_cls, *, instances=None, config=None):
|
||||
# merge resources/patterns/*.yaml into a pattern registry
|
||||
# bind registry/config into each graph node by id
|
||||
# build the schema (state) class and the engine
|
||||
root = root_cls()
|
||||
root._engine = Engine(graph, state_cls, payload_cls)
|
||||
return root
|
||||
```
|
||||
|
||||
## The registry
|
||||
|
||||
A single lookup replaces hand-rolled `if/else` selection:
|
||||
|
||||
```python
|
||||
_FAMILIES = {
|
||||
"csv": {"acme": {"heuristic": AcmeCsvPipeline}},
|
||||
"json": {"globex": {"heuristic": GlobexJsonPipeline}},
|
||||
}
|
||||
|
||||
def resolve_root(kind, variant, mode="heuristic"):
|
||||
return _FAMILIES[kind][variant][mode]
|
||||
```
|
||||
|
||||
Adding a new variant means adding one class and one registry entry.
|
||||
|
||||
## Driving a pipeline
|
||||
|
||||
Stages are driven explicitly so intermediate state can be persisted:
|
||||
|
||||
```python
|
||||
pipeline = build(resolve_root(kind, variant))
|
||||
content = pipeline.ingest.materialize(raw_bytes)
|
||||
records = pipeline.reader.read(content)
|
||||
items = pipeline.normalize.build(records)
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
Because stages are pure, test each stage in isolation and test the assembled
|
||||
root against **fixture inputs** whose expected outputs are stored as golden
|
||||
files. A parity test replays fixtures headlessly.
|
||||
Reference in New Issue
Block a user