Vishesh 'ironeagle' Bangotra
9929f05334
- collect.py: copy repo MCP bundles to ROOT/<name>/mcp (instead of ROOT/mcp/<name>); add nginx location guard to deny /<name>/mcp/ - config.py: bundle roots now point to <repo>/mcp and standards/mcp for static bundles - Rename core.py -> mcp_server.py and update imports in main.py and registry.py - Dockerfile: per-repo MCP COPYs under /app/<name>/mcp, drop blog copy; add ports 8008/8009 in .drone.yml - Restore standards/mcp/libraries.json from git for registry server
8 lines
2.6 KiB
JSON
8 lines
2.6 KiB
JSON
{
|
|
"module": "pipeline_scaffold",
|
|
"content": {
|
|
"path": "pipeline_scaffold",
|
|
"docstring": "# Pipeline Scaffold\n\n## Summary\n\nData-driven processing is expressed as a **composition graph**. The graph is\nbuilt by a DI library; execution is driven by a DAG engine. Stages are pure and\nindividually testable.\n\n---\n\n## When to use a pipeline\n\nUse a pipeline when a resource is produced by a **multi-step transformation**\nof an input (a document, a file, a message) rather than by a simple CRUD write.\n\n## Structure of a family\n\nA **family** is one input shape (e.g. a file format or a message kind). Each\nfamily has the same skeleton:\n\n```\npipelines/<family>/\n abc.py # port contracts (stage interfaces + root interface)\n impl.py # concrete stages + root implementation\n pipeline.yaml # graph spec: nodes, edges, roots, schema\n resources/\n pipeline.yaml # runtime spec consumed by the loader\n patterns/*.yaml # rule fragments (pattern registry)\n banks/<variant>.py # optional variants that re-pin a few slots\n```\n\n## Shared scaffolding\n\n```\npipelines/shared/\n ports.py # RegistryPort, FinalizePort, etc.\n loader.py # build(root_cls) -> wires the graph + engine\n registry.py # resolve_root(...) -> root class for a (kind, variant)\n```\n\n## The loader\n\n```python\ndef build(root_cls, *, instances=None, config=None):\n # merge resources/patterns/*.yaml into a pattern registry\n # bind registry/config into each graph node by id\n # build the schema (state) class and the engine\n root = root_cls()\n root._engine = Engine(graph, state_cls, payload_cls)\n return root\n```\n\n## The registry\n\nA single lookup replaces hand-rolled `if/else` selection:\n\n```python\n_FAMILIES = {\n \"csv\": {\"acme\": {\"heuristic\": AcmeCsvPipeline}},\n \"json\": {\"globex\": {\"heuristic\": GlobexJsonPipeline}},\n}\n\ndef resolve_root(kind, variant, mode=\"heuristic\"):\n return _FAMILIES[kind][variant][mode]\n```\n\nAdding a new variant means adding one class and one registry entry.\n\n## Driving a pipeline\n\nStages are driven explicitly so intermediate state can be persisted:\n\n```python\npipeline = build(resolve_root(kind, variant))\ncontent = pipeline.ingest.materialize(raw_bytes)\nrecords = pipeline.reader.read(content)\nitems = pipeline.normalize.build(records)\n```\n\n## Testing\n\nBecause stages are pure, test each stage in isolation and test the assembled\nroot against **fixture inputs** whose expected outputs are stored as golden\nfiles. A parity test replays fixtures headlessly.\n",
|
|
"objects": {}
|
|
}
|
|
} |