@@ -506,6 +528,7 @@
The reference half of the Standards description: the enforceable rules,
patterns, and the internal library catalog that agents consult while coding.
+- Build Recipe — the ordered, agentic checklist for a new app.
- Layering Rules — the strict request → domain → repository layering and
the rules that enforce it.
- Composition Root — a single composition point wiring the whole app.
diff --git a/standards/lib/search/search_index.json b/standards/lib/search/search_index.json
index 38c20a2..f5ed8a8 100644
--- a/standards/lib/search/search_index.json
+++ b/standards/lib/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Standards Reference","text":"The reference half of the Standards description: the enforceable rules, patterns, and the internal library catalog that agents consult while coding.
- Layering Rules \u2014 the strict request \u2192 domain \u2192 repository layering and the rules that enforce it.
- Composition Root \u2014 a single composition point wiring the whole app.
- Spec-First Routing \u2014 the OpenAPI contract as the routing source of truth.
- CRUD Template \u2014 the standardized generic resource endpoint scaffold.
- Pipeline Scaffold \u2014 the deterministic pipeline skeleton for multi-stage jobs.
- Worker & Progress \u2014 queued work, progress reporting, and visibility.
- Human-in-the-Loop \u2014 guardrails and approvals inside automations.
- Infra Adapters \u2014 how infrastructure connects without leaking into domain.
- Genericity Matrix \u2014 what is library-backed, template-provided, or authored.
- Library Registry \u2014 the catalog of internal libraries behind all of it.
Read the Wiki for the philosophy and rationale; these pages are the reference that mirrors the standards MCP bundle.
"},{"location":"02_layering_rules/","title":"Layering Rules","text":""},{"location":"02_layering_rules/#summary","title":"Summary","text":"The dependency direction is fixed. Violating it is a design defect.
"},{"location":"02_layering_rules/#allowed-imports","title":"Allowed imports","text":"
interfaces/* -> domain/* (services, ports, models)\ninterfaces/* -> infra/adapters/* (only read-only value objects)\ndomain/* -> domain/* (same layer)\ndomain/* -> pipelines/* (call a pipeline via a port)\ndomain/*/ports/* -> infra/* (bind adapters + libraries at the seam)\ninfra/adapters/* -> domain/ports/* (implement the interface)\nmain.py/init.py -> everything (composition root only)\n
"},{"location":"02_layering_rules/#forbidden-imports","title":"Forbidden imports","text":"domain/* -> interfaces/* (never)\ndomain/* -> fastapi/starlette (never)\ninfra/adapters/* -> interfaces/* (never)\ninterfaces/* -> infra/db/* (state lives in resources, not imports)\n
"},{"location":"02_layering_rules/#why","title":"Why","text":" - The business logic is testable without a server or an event loop.
- Adapters can be swapped (in-memory fake, real database) without touching business logic.
- The HTTP edge stays thin and replaceable.
"},{"location":"02_layering_rules/#the-portadapter-contract","title":"The port/adapter contract","text":"The ports/ package is the seam where the domain meets infrastructure. Each domain/<resource>/ports/ module declares the resource's models/document shape and exposes factory functions that bind a concrete adapter \u2014 so a port module is allowed to import from infra/ and from the persistence libraries. Everything outside ports/ in domain/ stays framework- and driver-free.
domain/<resource>/ports/db.py -> document model + create_<resource>_repo()\ndomain/<resource>/ports/service.py -> service + get_<resource>_service()\ninfra/db/repos/<vendor>.py -> repository adapter factory\ninfra/adapters/service.py -> generic CRUD service base\n
The composition root calls the port factories; nothing else constructs an adapter.
"},{"location":"02_layering_rules/#enforcement","title":"Enforcement","text":" domain/ must never import the web framework (fastapi / starlette) or interfaces/. Only the ports/ seam, infra/, and the composition root touch drivers. - Reviewers (and CI) reject
import fastapi inside domain/ or infra/.
"},{"location":"03_composition_root/","title":"Composition Root","text":""},{"location":"03_composition_root/#summary","title":"Summary","text":"All wiring happens in exactly one module. It builds a flat resource registry and exposes it to the HTTP layer and to scripts.
"},{"location":"03_composition_root/#the-resource-registry","title":"The resource registry","text":"init.py returns a plain dict[str, Any]. Keys follow two conventions:
<resource>_repo \u2014 a repository adapter for a resource. <resource>_svc \u2014 a service for a resource.
Plus named infrastructure adaptors:
http_cache_store \u2014 response cache backend. progress_bus \u2014 progress/event bus. <vendor>_<kind> \u2014 fetchers, clients, stores.
The API stores the dict on app.state.db_resources during lifespan startup. Handlers read from it; nothing imports concrete adapters.
"},{"location":"03_composition_root/#shape","title":"Shape","text":"_resources: dict = {}\n_initialized = False\n\nasync def init(settings=None) -> dict:\n global _initialized\n if _initialized:\n return _resources\n\n await setup_database(uri=settings.mongo_uri, db_name=settings.mongo_db)\n\n # 1. leaf repositories\n user_repo = create_user_repo()\n\n # 2. dependent repositories (dependencies passed by keyword)\n wallet_repo = create_wallet_repo(user_repo=user_repo)\n\n # 3. services (build via the DI library, injecting the repo)\n _resources[\"user_repo\"] = user_repo\n _resources[\"wallet_repo\"] = wallet_repo\n _resources[\"user_svc\"] = get_user_service(user_repo)\n _resources[\"wallet_svc\"] = get_wallet_service(wallet_repo)\n\n # 4. infrastructure adaptors (optional)\n _resources[\"http_cache_store\"] = HttpCacheStore(...)\n _resources[\"progress_bus\"] = ProgressEventBus()\n\n _initialized = True\n return _resources\n\nasync def shutdown() -> None:\n global _initialized\n await close_stores(_resources)\n await shutdown_database()\n _resources.clear()\n _initialized = False\n
"},{"location":"03_composition_root/#typed-accessors","title":"Typed accessors","text":"Expose a get_svc(name) helper with one @overload per key so scripts get typing without importing concrete classes:
@overload\ndef get_svc(name: Literal[\"user_svc\"]) -> UserService: ...\n@overload\ndef get_svc(name: Literal[\"wallet_svc\"]) -> WalletService: ...\n\ndef get_svc(name: str):\n return _resources[name]\n
Also expose get_db() (raw database handle) and pure helpers such as compute_fingerprint(payload) (canonical hash used for deduplication).
"},{"location":"03_composition_root/#rules","title":"Rules","text":" init() is idempotent. - The composition root is the only place that imports concrete adapters.
- Handlers never call
init(); they read request.app.state.db_resources. - Scripts call
init() / shutdown() around their work.
"},{"location":"04_spec_first_routing/","title":"Spec-First Routing","text":""},{"location":"04_spec_first_routing/#summary","title":"Summary","text":"The OpenAPI document is the single source of truth for the HTTP contract. Handlers are bound to it by name; there is no decorator router.
"},{"location":"04_spec_first_routing/#the-contract","title":"The contract","text":" openapi.yaml (or .json) declares paths, schemas, parameters, responses, and security. - A custom
operationId names the handler function. - The framework (
openapi_first) validates requests/responses against the spec and dispatches to the handler whose name equals operationId.
"},{"location":"04_spec_first_routing/#app-bootstrap","title":"App bootstrap","text":"# main.py\nfrom openapi_first.app import OpenAPIFirstApp\n\napp = OpenAPIFirstApp(\n openapi_path=\"openapi.yaml\",\n routes_module=interfaces.openapi, # module whose namespace holds handlers\n title=\"<App> API\",\n version=\"0.1.0\",\n lifespan=lifespan,\n)\n
lifespan calls the composition root:
@asynccontextmanager\nasync def lifespan(app):\n app.state.db_resources = await init()\n yield\n await shutdown()\n
"},{"location":"04_spec_first_routing/#handler-registry","title":"Handler registry","text":"interfaces/openapi.py is a flat re-export of every handler, so the framework can resolve operationId across the whole app:
from interfaces.item.routes import *\nfrom interfaces.wallet.routes import *\nfrom interfaces.progress.sse import *\n
"},{"location":"04_spec_first_routing/#handler-shape","title":"Handler shape","text":"Handlers are thin and synchronous with respect to business logic:
async def create_item(request: Request, body: dict):\n svc = request.app.state.db_resources[\"item_svc\"]\n return await svc.create(Item.model_validate(body))\n
"},{"location":"04_spec_first_routing/#conventions","title":"Conventions","text":" operationId naming: list_<resource>, create_<resource>, get_<resource>, update_<resource>, delete_<resource>. - Read-only endpoints may use a cache decorator keyed on path + query + user.
- Validation errors ->
400; not found -> 404; delete -> 204. - Security is declared once in the spec and enforced by the framework; handlers contain no auth code.
"},{"location":"04_spec_first_routing/#codegen","title":"Codegen","text":"The openapi_first library can scaffold models and route stubs from the spec, and ships app templates (CRUD app, model app, service). Prefer codegen over hand-writing boilerplate.
"},{"location":"05_crud_template/","title":"CRUD Template","text":""},{"location":"05_crud_template/#summary","title":"Summary","text":"Every resource follows the same four-file slice. Repetition is intentional: it makes the codebase predictable and generatable.
"},{"location":"05_crud_template/#the-slice","title":"The slice","text":"domain/<resource>/\n models.py # pure Pydantic model (no DB fields)\n ports/\n db.py # document model + repository factory\n service.py # service class + factory\n converters.py # (optional) transforms\n
"},{"location":"05_crud_template/#modelspy","title":"models.py","text":"A framework-free Pydantic model describing the business shape.
class Item(BaseModel):\n name: str\n quantity: int = 0\n
"},{"location":"05_crud_template/#portsdbpy","title":"ports/db.py","text":"The persistence model extends the business model and adds database concerns. The repository factory selects the right repository flavour.
class ItemDocument(Item, BaseDocument): # BaseDocument adds id/created_at/...\n pass\n\ndef create_item_repo(*, owner_repo=None) -> ItemRepository:\n return repo_factory(\n collection=\"items\",\n document=ItemDocument,\n populate_rules=[PopulateRule(field=\"owner\", collection=\"users\")],\n cache=None, # or a CachedBaseRepository config\n )\n
"},{"location":"05_crud_template/#portsservicepy","title":"ports/service.py","text":"The service is built through the DI library, injecting the repository. The generic CRUD base provides create/read/update/patch/delete/search/....
class ItemService(CrudsServiceAdaptor[Item, ItemDocument]):\n model = Item\n schema = ItemDocument\n\ndef get_item_service(repo) -> ItemService:\n return build(ItemService, instances={\"repo\": repo})\n
"},{"location":"05_crud_template/#naming-conventions","title":"Naming conventions","text":"Concept Convention collection plural snake case (items) document <Resource>Document repository factory create_<resource>_repo service <Resource>Service service factory get_<resource>_service resource key <resource>_svc operationId list_items, create_item, ..."},{"location":"05_crud_template/#optional-files","title":"Optional files","text":" converters.py \u2014 pure functions mapping between domain types. ports/enrichment.py \u2014 adapts a pipeline into an async callable the service can invoke (used when a resource is populated by a pipeline).
"},{"location":"06_pipeline_scaffold/","title":"Pipeline Scaffold","text":""},{"location":"06_pipeline_scaffold/#summary","title":"Summary","text":"Data-driven processing is expressed as a composition graph. The graph is built by a DI library; execution is driven by a DAG engine. Stages are pure and individually testable.
"},{"location":"06_pipeline_scaffold/#when-to-use-a-pipeline","title":"When to use a pipeline","text":"Use a pipeline when a resource is produced by a multi-step transformation of an input (a document, a file, a message) rather than by a simple CRUD write.
"},{"location":"06_pipeline_scaffold/#structure-of-a-family","title":"Structure of a family","text":"A family is one input shape (e.g. a file format or a message kind). Each family has the same skeleton:
pipelines/<family>/\n abc.py # port contracts (stage interfaces + root interface)\n impl.py # concrete stages + root implementation\n pipeline.yaml # graph spec: nodes, edges, roots, schema\n resources/\n pipeline.yaml # runtime spec consumed by the loader\n patterns/*.yaml # rule fragments (pattern registry)\n banks/<variant>.py # optional variants that re-pin a few slots\n
"},{"location":"06_pipeline_scaffold/#shared-scaffolding","title":"Shared scaffolding","text":"pipelines/shared/\n ports.py # RegistryPort, FinalizePort, etc.\n loader.py # build(root_cls) -> wires the graph + engine\n registry.py # resolve_root(...) -> root class for a (kind, variant)\n
"},{"location":"06_pipeline_scaffold/#the-loader","title":"The loader","text":"def build(root_cls, *, instances=None, config=None):\n # merge resources/patterns/*.yaml into a pattern registry\n # bind registry/config into each graph node by id\n # build the schema (state) class and the engine\n root = root_cls()\n root._engine = Engine(graph, state_cls, payload_cls)\n return root\n
"},{"location":"06_pipeline_scaffold/#the-registry","title":"The registry","text":"A single lookup replaces hand-rolled if/else selection:
_FAMILIES = {\n \"csv\": {\"acme\": {\"heuristic\": AcmeCsvPipeline}},\n \"json\": {\"globex\": {\"heuristic\": GlobexJsonPipeline}},\n}\n\ndef resolve_root(kind, variant, mode=\"heuristic\"):\n return _FAMILIES[kind][variant][mode]\n
Adding a new variant means adding one class and one registry entry.
"},{"location":"06_pipeline_scaffold/#driving-a-pipeline","title":"Driving a pipeline","text":"Stages are driven explicitly so intermediate state can be persisted:
pipeline = build(resolve_root(kind, variant))\ncontent = pipeline.ingest.materialize(raw_bytes)\nrecords = pipeline.reader.read(content)\nitems = pipeline.normalize.build(records)\n
"},{"location":"06_pipeline_scaffold/#testing","title":"Testing","text":"Because stages are pure, test each stage in isolation and test the assembled root against fixture inputs whose expected outputs are stored as golden files. A parity test replays fixtures headlessly.
"},{"location":"07_worker_and_progress/","title":"Worker & Progress","text":""},{"location":"07_worker_and_progress/#summary","title":"Summary","text":"Long-running work runs in a background worker. Jobs are durable documents that move through an explicit status machine. Progress is streamed to clients.
"},{"location":"07_worker_and_progress/#the-status-machine","title":"The status machine","text":"pending -> processing -> completed\n \\-> failed (retried up to N times)\n \\-> paused (waiting for human input)\n
pending \u2014 queued, eligible for pickup. processing \u2014 claimed by a worker. paused \u2014 blocked on a human-in-the-loop question. failed \u2014 error; retry_count increments until a max, then skipped. completed \u2014 terminal success.
"},{"location":"07_worker_and_progress/#the-poll-loop","title":"The poll loop","text":"async def poll(resources):\n backoff = MIN_POLL\n while True:\n jobs = await job_svc.search(filter={\"status\": \"pending\"}, limit=100)\n backoff = MIN_POLL if jobs else min(backoff * 2, MAX_POLL)\n for job in jobs:\n if (job.retry_count or 0) >= MAX_RETRIES:\n continue\n await job_svc.update_status(job.id, \"processing\")\n try:\n await process(job, resources)\n except PauseForResolution:\n await job_svc.update_status(job.id, \"paused\")\n except Exception as e:\n await job_svc.update_status(\n job.id, \"failed\", error=str(e),\n retry_count=(job.retry_count or 0) + 1,\n )\n await asyncio.sleep(backoff)\n
Tunables live in environment variables: WORKER_POLL_MIN, WORKER_POLL_MAX, WORKER_MAX_RETRIES.
"},{"location":"07_worker_and_progress/#progress-events","title":"Progress events","text":"A worker publishes structured progress events to a bus (a port). The HTTP edge exposes a streaming endpoint that replays events for a given job id.
class ProgressEvent(BaseModel):\n job_id: str\n step: str\n status: str # started | progress | completed | skipped | paused\n message: dict | None = None\n
The bus is an in-memory adapter by default; because it is a port, it can be swapped for a durable broker. Emit from the orchestrator at each stage:
await self.bus.publish(ProgressEvent(job_id=jid, step=\"normalize\",\n status=\"completed\", message={\"count\": n}))\n
"},{"location":"07_worker_and_progress/#streaming-endpoint","title":"Streaming endpoint","text":"async def stream_job_events(request, id):\n bus = request.app.state.db_resources.get(\"progress_bus\")\n if bus is None:\n return {\"events\": []}\n\n async def gen():\n sent = 0\n while True:\n events = bus.get_events(id)\n for e in events[sent:]:\n yield f\"data: {e.model_dump_json(exclude_none=True)}\\n\\n\"\n sent += 1\n await asyncio.sleep(0.5)\n\n return StreamingResponse(gen(), media_type=\"text/event-stream\")\n
Known limitation. The default in-memory bus is per-process. If the worker and the API run as separate processes, events published by the worker are not visible to the API. For production, back the bus with a shared broker. The port boundary makes this a one-adapter change.
"},{"location":"08_human_in_loop/","title":"Human-in-the-Loop","text":""},{"location":"08_human_in_loop/#summary","title":"Summary","text":"Some inputs are ambiguous and must be resolved by a human before processing can continue. The blueprint models this as a durable pause: an exception is raised, a question is recorded, the job pauses, and later resumes.
"},{"location":"08_human_in_loop/#the-mechanism","title":"The mechanism","text":" - During processing, a stage raises a domain exception carrying the context needed to ask the question:
raise PauseForResolution(\n job_id=jid,\n reference=\"<stable key for this decision>\",\n options=[...],\n context={...},\n)\n
- The orchestrator catches it, records a question document, and re-raises so the worker marks the job
paused. - A client lists pending questions and answers one:
GET /jobs/{id}/questions -> list pending questions\nPOST /questions/{qid}/resolve {choice} -> record the choice\n
- When the last pending question is resolved, the resolve endpoint flips the parent job back to
pending. - The worker picks it up again, rebuilds a resolution map from previously answered questions, and seeds it into the pipeline so the same decision is not asked twice.
"},{"location":"08_human_in_loop/#the-question-document","title":"The question document","text":"class Question(BaseDocument):\n job_id: str\n reference: str # stable key: stage|field|value\n status: str # pending | resolved\n options: list[dict]\n context: dict\n chosen: dict | None = None\n resolved_at: str | None = None\n
"},{"location":"08_human_in_loop/#resume-map","title":"Resume map","text":"The worker turns answered questions into a lookup keyed by reference, then injects it into the pipeline so that stage can skip re-raising:
resolved = {q.reference: q.chosen for q in await svc.search(\n filter={\"job_id\": jid, \"status\": \"resolved\"})}\n\npipeline = build(root, instances={\"pause\": handler}, config={\"resolved\": resolved})\n
"},{"location":"08_human_in_loop/#resolve-endpoint","title":"Resolve endpoint","text":"async def resolve_question(request, id, body):\n svc = request.app.state.db_resources[\"question_svc\"]\n q = await svc.read(id)\n await svc.update(id, {\"chosen\": body[\"chosen\"], \"status\": \"resolved\",\n \"resolved_at\": now()})\n\n pending = await svc.search(filter={\"job_id\": q.job_id, \"status\": \"pending\"},\n limit=1)\n if not pending:\n jobs = request.app.state.db_resources[\"job_svc\"]\n job = await jobs.read(q.job_id)\n if job and job.status == \"paused\":\n await jobs.update_status(q.job_id, \"pending\")\n return q\n
"},{"location":"08_human_in_loop/#cli-parity","title":"CLI parity","text":"The same flow is available through scripts/CLI: list, get, create, questions, resolve, retry.
"},{"location":"09_infra_adaptors/","title":"Infra Adapters","text":""},{"location":"09_infra_adaptors/#summary","title":"Summary","text":"All external systems are reached through adapters in infra/. Prefer an existing internal library over writing a new adapter; where no library exists, copy the Tier B scaffold from templates/app. See the Genericity Matrix for the Tier A / B / C split.
"},{"location":"09_infra_adaptors/#database-mongodb","title":"Database (MongoDB)","text":" - A single connection manager owns the client lifecycle.
setup_database(uri, db_name) / shutdown_database(). - A repository factory selects the repository flavour:
repo_factory(collection, document, engine=None,\n populate_rules=None, cache=None)\n
PopulateRule(field=\"owner\", collection=\"users\",\n nested_rules=[], max_depth=2, projection=None)\n
- Transactions use a session context manager provided by the library.
Tier A \u2014 Library (Mongo Ops). Do not write raw Motor calls in the domain; consult the registry and the library's MCP before touching the driver.
"},{"location":"09_infra_adaptors/#http-response-cache","title":"HTTP response cache","text":"Tier B \u2014 template. Copy from templates/app; not a standalone library.
A generic GET cache sits behind a decorator.
@cache_response(ttl=60)\nasync def list_items(request): ...\n
- Keyed on path + query + user.
- No-op when no store is wired.
invalidate(*resources) bumps a generation counter for write-triggered invalidation.
"},{"location":"09_infra_adaptors/#event-progress-bus","title":"Event / progress bus","text":"Tier B \u2014 template. Copy from templates/app.
A port with an in-memory default adapter (publish, get_events). Swap for a durable broker in production (see Worker & Progress).
"},{"location":"09_infra_adaptors/#external-clients-ingestion","title":"External clients (ingestion)","text":"External fetchers (files, mail, HTTP) are adapters that wrap ingestion libraries. Construct them in the composition root and inject into orchestrators.
Tier A \u2014 Libraries: Omniread (formats) and Mail Intake (email).
"},{"location":"09_infra_adaptors/#llm-clients","title":"LLM clients","text":"Tier B \u2014 template. Copy the pool scaffold from templates/app.
When a pipeline calls a model provider, route through a pool adaptor that handles provider selection, retries, and client lifecycle:
pool = setup_async_pool(provider=settings.llm_provider)\nresult = await pool.complete(model=\"...\", messages=[...])\n
Providers are selected by LLM_PROVIDER and configured by environment.
"},{"location":"09_infra_adaptors/#key-value-store","title":"Key-value store","text":"Tier B \u2014 template. Copy from templates/app.
For report/snapshot storage, provide a pluggable store port with a Redis adapter and an in-memory fallback.
"},{"location":"09_infra_adaptors/#rule","title":"Rule","text":" - Tier A (libraries: DB, DI, ingestion, mail, auth, docs, pipelines) \u2014 use the library; writing a private copy is a defect. Query
find_library first. - Tier B (cache, progress bus, LLM pool, KV store, composition shape, worker, tests) \u2014 these are template code, not libraries: copy them from
templates/app rather than searching the registry. - Tier C (models, business logic, pipeline families, rules) \u2014 author them yourself.
"},{"location":"12_genericity_matrix/","title":"Genericity Matrix","text":""},{"location":"12_genericity_matrix/#summary","title":"Summary","text":"This page states precisely what is library-backed, what is template-provided, and what is app-specific and must be authored. Use it to scope a new build and to know, for any given capability, whether to call a library, copy template code, or write it yourself.
"},{"location":"12_genericity_matrix/#tier-a-library-backed-use-never-reimplement","title":"Tier A \u2014 Library-backed (use; never reimplement)","text":"These capabilities already exist as internal libraries. Reach them through the library registry MCP (find_library(capability)) and read the library's own MCP for its API. Writing a private copy of any of these is a defect.
Capability Library MCP Persistence: repositories, cache, populate, transactions Mongo Ops (mongo_ops) :8007 Dependency injection / composition Hexa (hexa) :8008 Deterministic multi-step transformation engine DAG Pipe (dagpipe) :8006 Spec-first HTTP edge, client, codegen OpenAPI First (openapi_first) :8004 Format-agnostic fetch + parse (HTML, PDF, CSV, XLSX) Omniread (omniread) :8003 Read-only email ingestion Mail Intake (mail_intake) :8002 JWT auth primitives (register, login, introspect) Py JWT (jwtlib) :8005 Documentation build (MkDocs + MCP) Doc Forge (docforge) :8001"},{"location":"12_genericity_matrix/#tier-b-template-provided-copy-from-templatesapp-then-adapt","title":"Tier B \u2014 Template-provided (copy from templates/app, then adapt)","text":"These are reusable internal components shipped inside the template \u2014 they are not independent libraries. Copy them as-is, then rename/populate only what is app-specific. They give every app the same wiring shape as the reference app.
Component Template location Action Layering & dependency rule the package layout + import rules keep Composition root shape init.py (get_svc/get_db, idempotency) keep Spec-first web edge interfaces/openapi.py, thin handlers bound to operationId keep CRUD slice pattern domain/<resource>/{models.py, ports/db.py, ports/service.py, converters.py} rename + fill Generic CRUD service infra/adapters/service.py (CrudsServiceAdaptor) keep Repository factory infra/db/repos/mongo.py (_repo_factory) keep Worker poll + status loop worker.py keep Progress bus + streaming bus port + in-memory adapter + SSE endpoint keep Human-in-the-loop durable pause + question document + resolve endpoint + resume map keep HTTP response cache cache_response decorator + generation invalidation keep Key-value store KV store port + Redis adapter + in-memory fallback keep LLM client pool provider-select + retry pool scaffold keep Pipeline scaffold pipelines/shared/{ports,loader,registry}.py + pipelines/<family>/ rename + fill Operator scripts scripts/ (init() / get_svc() / shutdown()) rename Testing harness integration fakes + fixture-parity pattern keep Project scaffolding pyproject.toml, Dockerfile, docker-compose.yaml, .env.example, openapi.yaml rename + fill"},{"location":"12_genericity_matrix/#tier-c-app-specific-must-be-authored","title":"Tier C \u2014 App-specific (must be authored)","text":"Artifact Why Resource models the business shape Repository populate/cache rules the relationships Service business logic domain behaviour Pipeline families the transformation graph Pattern/rule resources data-driven policy Fixtures & golden outputs test data"},{"location":"12_genericity_matrix/#the-two-axes","title":"The two axes","text":"Everything app-specific reduces to two axes:
- Resource model \u2014 the set of resources, their fields, relationships, and the OpenAPI contract.
- Pipeline family \u2014 only if inputs require multi-step transformation.
If you can specify those two axes, the rest of the app follows mechanically: Tier A is imported, Tier B is copied, and Tier C is the only thing you author.
"},{"location":"13_library_registry/","title":"Library Registry","text":""},{"location":"13_library_registry/#summary","title":"Summary","text":"The Aetos ecosystem provides internal libraries for the shared concerns of every application. Use them. Writing a private copy of any capability below is a defect.
Each library also exposes its own documentation as an MCP server. An AI agent can read a library's full API by connecting to its MCP and reading doc://index, doc://nav, and doc://modules/{module}.
"},{"location":"13_library_registry/#catalog","title":"Catalog","text":"Library Import Purpose Use instead of MCP DAG Pipe dagpipe Deterministic state-transformation DAG engine hand-rolled orchestration/topological ordering :8006 Hexa hexa Declarative DI / composition-root tooling manual wiring, hand-written factories :8008 Mongo Ops mongo_ops Async MongoDB repositories, cache, populate, transactions raw Motor CRUD and connection lifecycle :8007 Omniread omniread Format-agnostic fetch + parse (HTML, PDF, CSV, XLSX) custom scrapers/parsers :8003 OpenAPI First openapi_first Spec-first FastAPI bootstrap, client, codegen decorator routers and hand-written clients :8004 Mail Intake mail_intake Provider-agnostic read-only email ingestion Gmail OAuth + MIME parsing :8002 Py JWT jwtlib JWT auth primitives (register/login/introspect) password hashing + token code :8005 Doc Forge docforge Docstring compiler \u2192 MkDocs + MCP docs custom doc generators :8001 Auth Server (service) Central identity service (register/login/me/introspect) building auth endpoints yourself (deferred)"},{"location":"13_library_registry/#how-to-choose","title":"How to choose","text":"You need to\u2026 Use persist and query documents Mongo Ops compose dependencies Hexa run a multi-step transformation DAG Pipe read/parse an arbitrary input format Omniread expose a contract-first HTTP API OpenAPI First ingest email Mail Intake authenticate users / verify tokens Py JWT (+ Auth Server) generate documentation Doc Forge"},{"location":"13_library_registry/#discovery","title":"Discovery","text":"A single standards MCP exposes find_library(capability), which returns the matching library, its import name, its \"use instead of\" guidance, and its MCP endpoint. Query it before writing any infrastructure.
"},{"location":"13_library_registry/#notes","title":"Notes","text":" - Hexa is a full library with its own MCP server on
:8008: read doc://index, doc://nav, and doc://modules/{module} (e.g. doc://modules/hexa.container) to discover the DI surface \u2014 build(cls, *, instances, config) and the PortNode model. - Auth Server is deferred: it has no
pyproject.toml/importable package and no MCP bundle. Do not reach for it yet; py-jwt covers token primitives. - Library MCP servers expose documentation resources (not executable tools): they tell an agent the available API surface so it does not invent code.
"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Standards Reference","text":"The reference half of the Standards description: the enforceable rules, patterns, and the internal library catalog that agents consult while coding.
- Build Recipe \u2014 the ordered, agentic checklist for a new app.
- Layering Rules \u2014 the strict request \u2192 domain \u2192 repository layering and the rules that enforce it.
- Composition Root \u2014 a single composition point wiring the whole app.
- Spec-First Routing \u2014 the OpenAPI contract as the routing source of truth.
- CRUD Template \u2014 the standardized generic resource endpoint scaffold.
- Pipeline Scaffold \u2014 the deterministic pipeline skeleton for multi-stage jobs.
- Worker & Progress \u2014 queued work, progress reporting, and visibility.
- Human-in-the-Loop \u2014 guardrails and approvals inside automations.
- Infra Adapters \u2014 how infrastructure connects without leaking into domain.
- Genericity Matrix \u2014 what is library-backed, template-provided, or authored.
- Library Registry \u2014 the catalog of internal libraries behind all of it.
Read the Wiki for the philosophy and rationale; these pages are the reference that mirrors the standards MCP bundle.
"},{"location":"01_build_recipe/","title":"Build Recipe","text":""},{"location":"01_build_recipe/#summary","title":"Summary","text":"The ordered, deterministic checklist for generating a new application of the standard shape. Every step is fixed; only the resource model (step 2) and the pipeline family (step 7) need app-specific authoring.
This page is served by the standards MCP as doc://modules/build_recipe. The start_app prompt on that server collects the two axes and hands off here; pull exact files with get_app_component.
"},{"location":"01_build_recipe/#bootstrap","title":"\ud83d\ude80 Bootstrap","text":"From the standards docs checkout:
cp -r <docs>/standards/src/templates/app apps/<your-app>\ncd apps/<your-app>\n
Install the internal libraries \u2014 editable local checkouts (preferred in development) or the private index (CI / fresh machines). Credentials are environment-only; never commit them.
# local checkouts\npip install -e ../aetos/openapi-first -e ../aetos/mongo-ops -e ../aetos/hexa \\\n -e ../aetos/dagpipe -e ../aetos/doc-forge\n# private index\npip install --extra-index-url \\\n \"https://$PIP_REGISTRY_USERNAME:$PIP_REGISTRY_PASSWORD@pip.aetoskia.com/simple\" .\n
"},{"location":"01_build_recipe/#agent-prompt-copyable","title":"\ud83d\udccb Agent prompt (copyable)","text":"# Build a new application from the Aetos Standards blueprint\n\nYou are coding a complete new backend application conforming to the Aetos\nstandard app blueprint. Execute every step in order. Only steps 2 and 3 use the\napp-specific details provided by the user; everything else is deterministic.\n\n## Before writing any code\n\nConnect to the `standards` MCP server and read:\n- `doc://nav` (the module list)\n- `doc://modules/genericity_matrix` (Tier A library / Tier B template / Tier C authored)\n- `doc://modules/layering_rules` (the dependency rules)\n\nThen:\n- call `find_library` with `db`, `di`, `routing`, `ingestion`, `pipeline`, `docs`\n and note the Tier A libraries that already exist \u2014 never reimplement one;\n- call `list_app_components()` and pull each file you need with\n `get_app_component(kind, resource=..., family=...)` instead of writing it from\n scratch.\n\n## Steps\n\n1. **Scaffold** \u2014 copy `standards/src/templates/app/` into `apps/<your-app>/`.\n Keep the tree: `domain/`, `infra/`, `interfaces/`, `pipelines/`, `tests/`,\n `scripts/`, plus `pyproject.toml`, `openapi.yaml`, `config.py`, `init.py`,\n `main.py`, `worker.py`, `Dockerfile`, `docker-compose.yaml`, `.env.example`.\n\n2. **Author the spec (app-specific)** \u2014 write `openapi.yaml`: resources,\n schemas, `operationId`s, security, and extension metadata. This is the\n contract and drives all routing.\n\n3. **Write each CRUD slice (app-specific)** \u2014 per resource create the four-file\n slice: `models.py`, `ports/db.py`, `ports/service.py`, and optional\n `converters.py`. Follow the CRUD template.\n\n4. **Wire the composition root** \u2014 in `init.py`, build leaf repositories, then\n dependent repositories, then services; add infra adapters (db, cache, stores,\n progress bus, fetchers). Expose `get_svc` overloads.\n\n5. **Add routes** \u2014 create `interfaces/<resource>/routes.py` thin handlers bound\n to `operationId`s and re-export them in `interfaces/openapi.py`. Add the cache\n decorator to read endpoints.\n\n6. **Bootstrap app and worker** \u2014 confirm `main.py` runs the OpenAPI-first app\n with its lifespan. If the app has long-running jobs, adapt `worker.py` to poll\n the job collection, run the pipeline, and publish progress.\n\n7. **Add pipeline families (only if data-driven)** \u2014 copy `pipelines/example/`\n to `pipelines/<family>/` with `abc.py`, `impl.py`, `pipeline.yaml`,\n `resources/`, and reuse `pipelines/shared/` (loader + registry). Register the\n root. Add fixtures and a parity test.\n\n8. **Add infrastructure only where needed** \u2014 Tier A capability: use the\n library. Tier B capability: copy the template. Never hand-write a Tier A\n library.\n\n9. **Tests** \u2014 integration tests with fakes at the HTTP edge; unit tests for\n domain and pipeline logic; a config-consistency test for rule-driven\n pipelines.\n\n10. **Scripts and seeds** \u2014 operator scripts under `scripts/` using\n `init()` / `get_svc()` / `shutdown()`; run as `python -m scripts.<name>`.\n\n11. **Docs** \u2014 author GSDFC docstrings, generate `.pyi` stubs, mark the package\n `py.typed`, and build the docs.\n\n**Definition of done:**\n- `python -c \"import main\"` succeeds.\n- `pytest` passes.\n- The app serves every operation in the spec; the worker drains its queue.\n- No domain module imports a web framework.\n- No Tier A library was reimplemented.\n
"},{"location":"01_build_recipe/#where-the-steps-land","title":"\ud83e\udded Where the steps land","text":"The steps map onto the layers as follows:
Step Package / file Depends on 1 \u00b7 Scaffold whole templates/app/ \u2014 2 \u00b7 Spec openapi.yaml \u2014 3 \u00b7 CRUD slices domain/<resource>/ spec, CRUD template 4 \u00b7 Wiring init.py every slice 5 \u00b7 Routes interfaces/<resource>/ services, openapi route registry 6 \u00b7 Bootstrap main.py, worker.py composition root 7 \u00b7 Pipelines pipelines/<family>/, shared/ library pipeline engine 8 \u00b7 Infra infra/ domain ports 9 \u00b7 Tests tests/ app + fixtures 10 \u00b7 Scripts scripts/ composition root 11 \u00b7 Docs docstrings + docs build code, Doc Forge"},{"location":"01_build_recipe/#definition-of-done","title":"\u26a0\ufe0f Definition of done","text":" python -c \"import main\" succeeds. pytest passes. - The app serves the spec's operations; the worker drains its queue.
- No domain module imports a web framework.
- No Tier A library was reimplemented.
"},{"location":"01_build_recipe/#related","title":"Related","text":" - 02 \u2013 Layering Rules \u00b7 05 \u2013 CRUD Template \u00b7 06 \u2013 Pipeline Scaffold \u00b7 12 \u2013 Genericity Matrix
"},{"location":"02_layering_rules/","title":"Layering Rules","text":""},{"location":"02_layering_rules/#summary","title":"Summary","text":"The dependency direction is fixed. Violating it is a design defect.
"},{"location":"02_layering_rules/#allowed-imports","title":"Allowed imports","text":"interfaces/* -> domain/* (services, ports, models)\ninterfaces/* -> infra/adapters/* (only read-only value objects)\ndomain/* -> domain/* (same layer)\ndomain/* -> pipelines/* (call a pipeline via a port)\ndomain/*/ports/* -> infra/* (bind adapters + libraries at the seam)\ninfra/adapters/* -> domain/ports/* (implement the interface)\nmain.py/init.py -> everything (composition root only)\n
"},{"location":"02_layering_rules/#forbidden-imports","title":"Forbidden imports","text":"domain/* -> interfaces/* (never)\ndomain/* -> fastapi/starlette (never)\ninfra/adapters/* -> interfaces/* (never)\ninterfaces/* -> infra/db/* (state lives in resources, not imports)\n
"},{"location":"02_layering_rules/#why","title":"Why","text":" - The business logic is testable without a server or an event loop.
- Adapters can be swapped (in-memory fake, real database) without touching business logic.
- The HTTP edge stays thin and replaceable.
"},{"location":"02_layering_rules/#the-portadapter-contract","title":"The port/adapter contract","text":"The ports/ package is the seam where the domain meets infrastructure. Each domain/<resource>/ports/ module declares the resource's models/document shape and exposes factory functions that bind a concrete adapter \u2014 so a port module is allowed to import from infra/ and from the persistence libraries. Everything outside ports/ in domain/ stays framework- and driver-free.
domain/<resource>/ports/db.py -> document model + create_<resource>_repo()\ndomain/<resource>/ports/service.py -> service + get_<resource>_service()\ninfra/db/repos/<vendor>.py -> repository adapter factory\ninfra/adapters/service.py -> generic CRUD service base\n
The composition root calls the port factories; nothing else constructs an adapter.
"},{"location":"02_layering_rules/#enforcement","title":"Enforcement","text":" domain/ must never import the web framework (fastapi / starlette) or interfaces/. Only the ports/ seam, infra/, and the composition root touch drivers. - Reviewers (and CI) reject
import fastapi inside domain/ or infra/.
"},{"location":"03_composition_root/","title":"Composition Root","text":""},{"location":"03_composition_root/#summary","title":"Summary","text":"All wiring happens in exactly one module. It builds a flat resource registry and exposes it to the HTTP layer and to scripts.
"},{"location":"03_composition_root/#the-resource-registry","title":"The resource registry","text":"init.py returns a plain dict[str, Any]. Keys follow two conventions:
<resource>_repo \u2014 a repository adapter for a resource. <resource>_svc \u2014 a service for a resource.
Plus named infrastructure adaptors:
http_cache_store \u2014 response cache backend. progress_bus \u2014 progress/event bus. <vendor>_<kind> \u2014 fetchers, clients, stores.
The API stores the dict on app.state.db_resources during lifespan startup. Handlers read from it; nothing imports concrete adapters.
"},{"location":"03_composition_root/#shape","title":"Shape","text":"_resources: dict = {}\n_initialized = False\n\nasync def init(settings=None) -> dict:\n global _initialized\n if _initialized:\n return _resources\n\n await setup_database(uri=settings.mongo_uri, db_name=settings.mongo_db)\n\n # 1. leaf repositories\n user_repo = create_user_repo()\n\n # 2. dependent repositories (dependencies passed by keyword)\n wallet_repo = create_wallet_repo(user_repo=user_repo)\n\n # 3. services (build via the DI library, injecting the repo)\n _resources[\"user_repo\"] = user_repo\n _resources[\"wallet_repo\"] = wallet_repo\n _resources[\"user_svc\"] = get_user_service(user_repo)\n _resources[\"wallet_svc\"] = get_wallet_service(wallet_repo)\n\n # 4. infrastructure adaptors (optional)\n _resources[\"http_cache_store\"] = HttpCacheStore(...)\n _resources[\"progress_bus\"] = ProgressEventBus()\n\n _initialized = True\n return _resources\n\nasync def shutdown() -> None:\n global _initialized\n await close_stores(_resources)\n await shutdown_database()\n _resources.clear()\n _initialized = False\n
"},{"location":"03_composition_root/#typed-accessors","title":"Typed accessors","text":"Expose a get_svc(name) helper with one @overload per key so scripts get typing without importing concrete classes:
@overload\ndef get_svc(name: Literal[\"user_svc\"]) -> UserService: ...\n@overload\ndef get_svc(name: Literal[\"wallet_svc\"]) -> WalletService: ...\n\ndef get_svc(name: str):\n return _resources[name]\n
Also expose get_db() (raw database handle) and pure helpers such as compute_fingerprint(payload) (canonical hash used for deduplication).
"},{"location":"03_composition_root/#rules","title":"Rules","text":" init() is idempotent. - The composition root is the only place that imports concrete adapters.
- Handlers never call
init(); they read request.app.state.db_resources. - Scripts call
init() / shutdown() around their work.
"},{"location":"04_spec_first_routing/","title":"Spec-First Routing","text":""},{"location":"04_spec_first_routing/#summary","title":"Summary","text":"The OpenAPI document is the single source of truth for the HTTP contract. Handlers are bound to it by name; there is no decorator router.
"},{"location":"04_spec_first_routing/#the-contract","title":"The contract","text":" openapi.yaml (or .json) declares paths, schemas, parameters, responses, and security. - A custom
operationId names the handler function. - The framework (
openapi_first) validates requests/responses against the spec and dispatches to the handler whose name equals operationId.
"},{"location":"04_spec_first_routing/#app-bootstrap","title":"App bootstrap","text":"# main.py\nfrom openapi_first.app import OpenAPIFirstApp\n\napp = OpenAPIFirstApp(\n openapi_path=\"openapi.yaml\",\n routes_module=interfaces.openapi, # module whose namespace holds handlers\n title=\"<App> API\",\n version=\"0.1.0\",\n lifespan=lifespan,\n)\n
lifespan calls the composition root:
@asynccontextmanager\nasync def lifespan(app):\n app.state.db_resources = await init()\n yield\n await shutdown()\n
"},{"location":"04_spec_first_routing/#handler-registry","title":"Handler registry","text":"interfaces/openapi.py is a flat re-export of every handler, so the framework can resolve operationId across the whole app:
from interfaces.item.routes import *\nfrom interfaces.wallet.routes import *\nfrom interfaces.progress.sse import *\n
"},{"location":"04_spec_first_routing/#handler-shape","title":"Handler shape","text":"Handlers are thin and synchronous with respect to business logic:
async def create_item(request: Request, body: dict):\n svc = request.app.state.db_resources[\"item_svc\"]\n return await svc.create(Item.model_validate(body))\n
"},{"location":"04_spec_first_routing/#conventions","title":"Conventions","text":" operationId naming: list_<resource>, create_<resource>, get_<resource>, update_<resource>, delete_<resource>. - Read-only endpoints may use a cache decorator keyed on path + query + user.
- Validation errors ->
400; not found -> 404; delete -> 204. - Security is declared once in the spec and enforced by the framework; handlers contain no auth code.
"},{"location":"04_spec_first_routing/#codegen","title":"Codegen","text":"The openapi_first library can scaffold models and route stubs from the spec, and ships app templates (CRUD app, model app, service). Prefer codegen over hand-writing boilerplate.
"},{"location":"05_crud_template/","title":"CRUD Template","text":""},{"location":"05_crud_template/#summary","title":"Summary","text":"Every resource follows the same four-file slice. Repetition is intentional: it makes the codebase predictable and generatable.
"},{"location":"05_crud_template/#the-slice","title":"The slice","text":"domain/<resource>/\n models.py # pure Pydantic model (no DB fields)\n ports/\n db.py # document model + repository factory\n service.py # service class + factory\n converters.py # (optional) transforms\n
"},{"location":"05_crud_template/#modelspy","title":"models.py","text":"A framework-free Pydantic model describing the business shape.
class Item(BaseModel):\n name: str\n quantity: int = 0\n
"},{"location":"05_crud_template/#portsdbpy","title":"ports/db.py","text":"The persistence model extends the business model and adds database concerns. The repository factory selects the right repository flavour.
class ItemDocument(Item, BaseDocument): # BaseDocument adds id/created_at/...\n pass\n\ndef create_item_repo(*, owner_repo=None) -> ItemRepository:\n return repo_factory(\n collection=\"items\",\n document=ItemDocument,\n populate_rules=[PopulateRule(field=\"owner\", collection=\"users\")],\n cache=None, # or a CachedBaseRepository config\n )\n
"},{"location":"05_crud_template/#portsservicepy","title":"ports/service.py","text":"The service is built through the DI library, injecting the repository. The generic CRUD base provides create/read/update/patch/delete/search/....
class ItemService(CrudsServiceAdaptor[Item, ItemDocument]):\n model = Item\n schema = ItemDocument\n\ndef get_item_service(repo) -> ItemService:\n return build(ItemService, instances={\"repo\": repo})\n
"},{"location":"05_crud_template/#naming-conventions","title":"Naming conventions","text":"Concept Convention collection plural snake case (items) document <Resource>Document repository factory create_<resource>_repo service <Resource>Service service factory get_<resource>_service resource key <resource>_svc operationId list_items, create_item, ..."},{"location":"05_crud_template/#optional-files","title":"Optional files","text":" converters.py \u2014 pure functions mapping between domain types. ports/enrichment.py \u2014 adapts a pipeline into an async callable the service can invoke (used when a resource is populated by a pipeline).
"},{"location":"06_pipeline_scaffold/","title":"Pipeline Scaffold","text":""},{"location":"06_pipeline_scaffold/#summary","title":"Summary","text":"Data-driven processing is expressed as a composition graph. The graph is built by a DI library; execution is driven by a DAG engine. Stages are pure and individually testable.
"},{"location":"06_pipeline_scaffold/#when-to-use-a-pipeline","title":"When to use a pipeline","text":"Use a pipeline when a resource is produced by a multi-step transformation of an input (a document, a file, a message) rather than by a simple CRUD write.
"},{"location":"06_pipeline_scaffold/#structure-of-a-family","title":"Structure of a family","text":"A family is one input shape (e.g. a file format or a message kind). Each family has the same skeleton:
pipelines/<family>/\n abc.py # port contracts (stage interfaces + root interface)\n impl.py # concrete stages + root implementation\n pipeline.yaml # graph spec: nodes, edges, roots, schema\n resources/\n pipeline.yaml # runtime spec consumed by the loader\n patterns/*.yaml # rule fragments (pattern registry)\n banks/<variant>.py # optional variants that re-pin a few slots\n
"},{"location":"06_pipeline_scaffold/#shared-scaffolding","title":"Shared scaffolding","text":"pipelines/shared/\n ports.py # RegistryPort, FinalizePort, etc.\n loader.py # build(root_cls) -> wires the graph + engine\n registry.py # resolve_root(...) -> root class for a (kind, variant)\n
"},{"location":"06_pipeline_scaffold/#the-loader","title":"The loader","text":"def build(root_cls, *, instances=None, config=None):\n # merge resources/patterns/*.yaml into a pattern registry\n # bind registry/config into each graph node by id\n # build the schema (state) class and the engine\n root = root_cls()\n root._engine = Engine(graph, state_cls, payload_cls)\n return root\n
"},{"location":"06_pipeline_scaffold/#the-registry","title":"The registry","text":"A single lookup replaces hand-rolled if/else selection:
_FAMILIES = {\n \"csv\": {\"acme\": {\"heuristic\": AcmeCsvPipeline}},\n \"json\": {\"globex\": {\"heuristic\": GlobexJsonPipeline}},\n}\n\ndef resolve_root(kind, variant, mode=\"heuristic\"):\n return _FAMILIES[kind][variant][mode]\n
Adding a new variant means adding one class and one registry entry.
"},{"location":"06_pipeline_scaffold/#driving-a-pipeline","title":"Driving a pipeline","text":"Stages are driven explicitly so intermediate state can be persisted:
pipeline = build(resolve_root(kind, variant))\ncontent = pipeline.ingest.materialize(raw_bytes)\nrecords = pipeline.reader.read(content)\nitems = pipeline.normalize.build(records)\n
"},{"location":"06_pipeline_scaffold/#testing","title":"Testing","text":"Because stages are pure, test each stage in isolation and test the assembled root against fixture inputs whose expected outputs are stored as golden files. A parity test replays fixtures headlessly.
"},{"location":"07_worker_and_progress/","title":"Worker & Progress","text":""},{"location":"07_worker_and_progress/#summary","title":"Summary","text":"Long-running work runs in a background worker. Jobs are durable documents that move through an explicit status machine. Progress is streamed to clients.
"},{"location":"07_worker_and_progress/#the-status-machine","title":"The status machine","text":"pending -> processing -> completed\n \\-> failed (retried up to N times)\n \\-> paused (waiting for human input)\n
pending \u2014 queued, eligible for pickup. processing \u2014 claimed by a worker. paused \u2014 blocked on a human-in-the-loop question. failed \u2014 error; retry_count increments until a max, then skipped. completed \u2014 terminal success.
"},{"location":"07_worker_and_progress/#the-poll-loop","title":"The poll loop","text":"async def poll(resources):\n backoff = MIN_POLL\n while True:\n jobs = await job_svc.search(filter={\"status\": \"pending\"}, limit=100)\n backoff = MIN_POLL if jobs else min(backoff * 2, MAX_POLL)\n for job in jobs:\n if (job.retry_count or 0) >= MAX_RETRIES:\n continue\n await job_svc.update_status(job.id, \"processing\")\n try:\n await process(job, resources)\n except PauseForResolution:\n await job_svc.update_status(job.id, \"paused\")\n except Exception as e:\n await job_svc.update_status(\n job.id, \"failed\", error=str(e),\n retry_count=(job.retry_count or 0) + 1,\n )\n await asyncio.sleep(backoff)\n
Tunables live in environment variables: WORKER_POLL_MIN, WORKER_POLL_MAX, WORKER_MAX_RETRIES.
"},{"location":"07_worker_and_progress/#progress-events","title":"Progress events","text":"A worker publishes structured progress events to a bus (a port). The HTTP edge exposes a streaming endpoint that replays events for a given job id.
class ProgressEvent(BaseModel):\n job_id: str\n step: str\n status: str # started | progress | completed | skipped | paused\n message: dict | None = None\n
The bus is an in-memory adapter by default; because it is a port, it can be swapped for a durable broker. Emit from the orchestrator at each stage:
await self.bus.publish(ProgressEvent(job_id=jid, step=\"normalize\",\n status=\"completed\", message={\"count\": n}))\n
"},{"location":"07_worker_and_progress/#streaming-endpoint","title":"Streaming endpoint","text":"async def stream_job_events(request, id):\n bus = request.app.state.db_resources.get(\"progress_bus\")\n if bus is None:\n return {\"events\": []}\n\n async def gen():\n sent = 0\n while True:\n events = bus.get_events(id)\n for e in events[sent:]:\n yield f\"data: {e.model_dump_json(exclude_none=True)}\\n\\n\"\n sent += 1\n await asyncio.sleep(0.5)\n\n return StreamingResponse(gen(), media_type=\"text/event-stream\")\n
Known limitation. The default in-memory bus is per-process. If the worker and the API run as separate processes, events published by the worker are not visible to the API. For production, back the bus with a shared broker. The port boundary makes this a one-adapter change.
"},{"location":"08_human_in_loop/","title":"Human-in-the-Loop","text":""},{"location":"08_human_in_loop/#summary","title":"Summary","text":"Some inputs are ambiguous and must be resolved by a human before processing can continue. The blueprint models this as a durable pause: an exception is raised, a question is recorded, the job pauses, and later resumes.
"},{"location":"08_human_in_loop/#the-mechanism","title":"The mechanism","text":" - During processing, a stage raises a domain exception carrying the context needed to ask the question:
raise PauseForResolution(\n job_id=jid,\n reference=\"<stable key for this decision>\",\n options=[...],\n context={...},\n)\n
- The orchestrator catches it, records a question document, and re-raises so the worker marks the job
paused. - A client lists pending questions and answers one:
GET /jobs/{id}/questions -> list pending questions\nPOST /questions/{qid}/resolve {choice} -> record the choice\n
- When the last pending question is resolved, the resolve endpoint flips the parent job back to
pending. - The worker picks it up again, rebuilds a resolution map from previously answered questions, and seeds it into the pipeline so the same decision is not asked twice.
"},{"location":"08_human_in_loop/#the-question-document","title":"The question document","text":"class Question(BaseDocument):\n job_id: str\n reference: str # stable key: stage|field|value\n status: str # pending | resolved\n options: list[dict]\n context: dict\n chosen: dict | None = None\n resolved_at: str | None = None\n
"},{"location":"08_human_in_loop/#resume-map","title":"Resume map","text":"The worker turns answered questions into a lookup keyed by reference, then injects it into the pipeline so that stage can skip re-raising:
resolved = {q.reference: q.chosen for q in await svc.search(\n filter={\"job_id\": jid, \"status\": \"resolved\"})}\n\npipeline = build(root, instances={\"pause\": handler}, config={\"resolved\": resolved})\n
"},{"location":"08_human_in_loop/#resolve-endpoint","title":"Resolve endpoint","text":"async def resolve_question(request, id, body):\n svc = request.app.state.db_resources[\"question_svc\"]\n q = await svc.read(id)\n await svc.update(id, {\"chosen\": body[\"chosen\"], \"status\": \"resolved\",\n \"resolved_at\": now()})\n\n pending = await svc.search(filter={\"job_id\": q.job_id, \"status\": \"pending\"},\n limit=1)\n if not pending:\n jobs = request.app.state.db_resources[\"job_svc\"]\n job = await jobs.read(q.job_id)\n if job and job.status == \"paused\":\n await jobs.update_status(q.job_id, \"pending\")\n return q\n
"},{"location":"08_human_in_loop/#cli-parity","title":"CLI parity","text":"The same flow is available through scripts/CLI: list, get, create, questions, resolve, retry.
"},{"location":"09_infra_adaptors/","title":"Infra Adapters","text":""},{"location":"09_infra_adaptors/#summary","title":"Summary","text":"All external systems are reached through adapters in infra/. Prefer an existing internal library over writing a new adapter; where no library exists, copy the Tier B scaffold from templates/app. See the Genericity Matrix for the Tier A / B / C split.
"},{"location":"09_infra_adaptors/#database-mongodb","title":"Database (MongoDB)","text":" - A single connection manager owns the client lifecycle.
setup_database(uri, db_name) / shutdown_database(). - A repository factory selects the repository flavour:
repo_factory(collection, document, engine=None,\n populate_rules=None, cache=None)\n
PopulateRule(field=\"owner\", collection=\"users\",\n nested_rules=[], max_depth=2, projection=None)\n
- Transactions use a session context manager provided by the library.
Tier A \u2014 Library (Mongo Ops). Do not write raw Motor calls in the domain; consult the registry and the library's MCP before touching the driver.
"},{"location":"09_infra_adaptors/#http-response-cache","title":"HTTP response cache","text":"Tier B \u2014 template. Copy from templates/app; not a standalone library.
A generic GET cache sits behind a decorator.
@cache_response(ttl=60)\nasync def list_items(request): ...\n
- Keyed on path + query + user.
- No-op when no store is wired.
invalidate(*resources) bumps a generation counter for write-triggered invalidation.
"},{"location":"09_infra_adaptors/#event-progress-bus","title":"Event / progress bus","text":"Tier B \u2014 template. Copy from templates/app.
A port with an in-memory default adapter (publish, get_events). Swap for a durable broker in production (see Worker & Progress).
"},{"location":"09_infra_adaptors/#external-clients-ingestion","title":"External clients (ingestion)","text":"External fetchers (files, mail, HTTP) are adapters that wrap ingestion libraries. Construct them in the composition root and inject into orchestrators.
Tier A \u2014 Libraries: Omniread (formats) and Mail Intake (email).
"},{"location":"09_infra_adaptors/#llm-clients","title":"LLM clients","text":"Tier B \u2014 template. Copy the pool scaffold from templates/app.
When a pipeline calls a model provider, route through a pool adaptor that handles provider selection, retries, and client lifecycle:
pool = setup_async_pool(provider=settings.llm_provider)\nresult = await pool.complete(model=\"...\", messages=[...])\n
Providers are selected by LLM_PROVIDER and configured by environment.
"},{"location":"09_infra_adaptors/#key-value-store","title":"Key-value store","text":"Tier B \u2014 template. Copy from templates/app.
For report/snapshot storage, provide a pluggable store port with a Redis adapter and an in-memory fallback.
"},{"location":"09_infra_adaptors/#rule","title":"Rule","text":" - Tier A (libraries: DB, DI, ingestion, mail, auth, docs, pipelines) \u2014 use the library; writing a private copy is a defect. Query
find_library first. - Tier B (cache, progress bus, LLM pool, KV store, composition shape, worker, tests) \u2014 these are template code, not libraries: copy them from
templates/app rather than searching the registry. - Tier C (models, business logic, pipeline families, rules) \u2014 author them yourself.
"},{"location":"12_genericity_matrix/","title":"Genericity Matrix","text":""},{"location":"12_genericity_matrix/#summary","title":"Summary","text":"This page states precisely what is library-backed, what is template-provided, and what is app-specific and must be authored. Use it to scope a new build and to know, for any given capability, whether to call a library, copy template code, or write it yourself.
"},{"location":"12_genericity_matrix/#tier-a-library-backed-use-never-reimplement","title":"Tier A \u2014 Library-backed (use; never reimplement)","text":"These capabilities already exist as internal libraries. Reach them through the library registry MCP (find_library(capability)) and read the library's own MCP for its API. Writing a private copy of any of these is a defect.
Capability Library MCP Persistence: repositories, cache, populate, transactions Mongo Ops (mongo_ops) :8007 Dependency injection / composition Hexa (hexa) :8008 Deterministic multi-step transformation engine DAG Pipe (dagpipe) :8006 Spec-first HTTP edge, client, codegen OpenAPI First (openapi_first) :8004 Format-agnostic fetch + parse (HTML, PDF, CSV, XLSX) Omniread (omniread) :8003 Read-only email ingestion Mail Intake (mail_intake) :8002 JWT auth primitives (register, login, introspect) Py JWT (jwtlib) :8005 Documentation build (MkDocs + MCP) Doc Forge (docforge) :8001"},{"location":"12_genericity_matrix/#tier-b-template-provided-copy-from-templatesapp-then-adapt","title":"Tier B \u2014 Template-provided (copy from templates/app, then adapt)","text":"These are reusable internal components shipped inside the template \u2014 they are not independent libraries. Copy them as-is, then rename/populate only what is app-specific. They give every app the same wiring shape as the reference app.
Component Template location Action Layering & dependency rule the package layout + import rules keep Composition root shape init.py (get_svc/get_db, idempotency) keep Spec-first web edge interfaces/openapi.py, thin handlers bound to operationId keep CRUD slice pattern domain/<resource>/{models.py, ports/db.py, ports/service.py, converters.py} rename + fill Generic CRUD service infra/adapters/service.py (CrudsServiceAdaptor) keep Repository factory infra/db/repos/mongo.py (_repo_factory) keep Worker poll + status loop worker.py keep Progress bus + streaming bus port + in-memory adapter + SSE endpoint keep Human-in-the-loop durable pause + question document + resolve endpoint + resume map keep HTTP response cache cache_response decorator + generation invalidation keep Key-value store KV store port + Redis adapter + in-memory fallback keep LLM client pool provider-select + retry pool scaffold keep Pipeline scaffold pipelines/shared/{ports,loader,registry}.py + pipelines/<family>/ rename + fill Operator scripts scripts/ (init() / get_svc() / shutdown()) rename Testing harness integration fakes + fixture-parity pattern keep Project scaffolding pyproject.toml, Dockerfile, docker-compose.yaml, .env.example, openapi.yaml rename + fill"},{"location":"12_genericity_matrix/#tier-c-app-specific-must-be-authored","title":"Tier C \u2014 App-specific (must be authored)","text":"Artifact Why Resource models the business shape Repository populate/cache rules the relationships Service business logic domain behaviour Pipeline families the transformation graph Pattern/rule resources data-driven policy Fixtures & golden outputs test data"},{"location":"12_genericity_matrix/#the-two-axes","title":"The two axes","text":"Everything app-specific reduces to two axes:
- Resource model \u2014 the set of resources, their fields, relationships, and the OpenAPI contract.
- Pipeline family \u2014 only if inputs require multi-step transformation.
If you can specify those two axes, the rest of the app follows mechanically: Tier A is imported, Tier B is copied, and Tier C is the only thing you author.
"},{"location":"13_library_registry/","title":"Library Registry","text":""},{"location":"13_library_registry/#summary","title":"Summary","text":"The Aetos ecosystem provides internal libraries for the shared concerns of every application. Use them. Writing a private copy of any capability below is a defect.
Each library also exposes its own documentation as an MCP server. An AI agent can read a library's full API by connecting to its MCP and reading doc://index, doc://nav, and doc://modules/{module}.
"},{"location":"13_library_registry/#catalog","title":"Catalog","text":"Library Import Purpose Use instead of MCP DAG Pipe dagpipe Deterministic state-transformation DAG engine hand-rolled orchestration/topological ordering :8006 Hexa hexa Declarative DI / composition-root tooling manual wiring, hand-written factories :8008 Mongo Ops mongo_ops Async MongoDB repositories, cache, populate, transactions raw Motor CRUD and connection lifecycle :8007 Omniread omniread Format-agnostic fetch + parse (HTML, PDF, CSV, XLSX) custom scrapers/parsers :8003 OpenAPI First openapi_first Spec-first FastAPI bootstrap, client, codegen decorator routers and hand-written clients :8004 Mail Intake mail_intake Provider-agnostic read-only email ingestion Gmail OAuth + MIME parsing :8002 Py JWT jwtlib JWT auth primitives (register/login/introspect) password hashing + token code :8005 Doc Forge docforge Docstring compiler \u2192 MkDocs + MCP docs custom doc generators :8001 Auth Server (service) Central identity service (register/login/me/introspect) building auth endpoints yourself (deferred)"},{"location":"13_library_registry/#how-to-choose","title":"How to choose","text":"You need to\u2026 Use persist and query documents Mongo Ops compose dependencies Hexa run a multi-step transformation DAG Pipe read/parse an arbitrary input format Omniread expose a contract-first HTTP API OpenAPI First ingest email Mail Intake authenticate users / verify tokens Py JWT (+ Auth Server) generate documentation Doc Forge"},{"location":"13_library_registry/#discovery","title":"Discovery","text":"A single standards MCP exposes find_library(capability), which returns the matching library, its import name, its \"use instead of\" guidance, and its MCP endpoint. Query it before writing any infrastructure.
"},{"location":"13_library_registry/#notes","title":"Notes","text":" - Hexa is a full library with its own MCP server on
:8008: read doc://index, doc://nav, and doc://modules/{module} (e.g. doc://modules/hexa.container) to discover the DI surface \u2014 build(cls, *, instances, config) and the PortNode model. - Auth Server is deferred: it has no
pyproject.toml/importable package and no MCP bundle. Do not reach for it yet; py-jwt covers token primitives. - Library MCP servers expose documentation resources (not executable tools): they tell an agent the available API surface so it does not invent code.
"}]}
\ No newline at end of file
diff --git a/standards/mcp/index.json b/standards/mcp/index.json
index b06d01e..406d7a2 100644
--- a/standards/mcp/index.json
+++ b/standards/mcp/index.json
@@ -1,6 +1,6 @@
{
"project": "standards",
"type": "standards",
- "modules_count": 11,
+ "modules_count": 12,
"source": "hand-authored"
}
\ No newline at end of file
diff --git a/standards/mcp/modules/build_recipe.json b/standards/mcp/modules/build_recipe.json
new file mode 100644
index 0000000..415bcec
--- /dev/null
+++ b/standards/mcp/modules/build_recipe.json
@@ -0,0 +1,8 @@
+{
+ "module": "build_recipe",
+ "content": {
+ "path": "build_recipe",
+ "docstring": "# Build Recipe\n\n## Summary\n\nThe ordered, deterministic checklist for generating a new application of the\nstandard shape. Every step is fixed; only the resource model (step 2) and the\npipeline family (step 7) need app-specific authoring.\n\nThis page is served by the `standards` MCP as `doc://modules/build_recipe`. The\n`start_app` prompt on that server collects the two axes and hands off here; pull\nexact files with `get_app_component`.\n\n---\n\n## 🚀 Bootstrap\n\nFrom the standards docs checkout:\n\n```bash\ncp -r /standards/src/templates/app apps/\ncd apps/\n```\n\nInstall the internal libraries — editable local checkouts (preferred in\ndevelopment) or the private index (CI / fresh machines). Credentials are\nenvironment-only; never commit them.\n\n```bash\n# local checkouts\npip install -e ../aetos/openapi-first -e ../aetos/mongo-ops -e ../aetos/hexa \\\n -e ../aetos/dagpipe -e ../aetos/doc-forge\n# private index\npip install --extra-index-url \\\n \"https://$PIP_REGISTRY_USERNAME:$PIP_REGISTRY_PASSWORD@pip.aetoskia.com/simple\" .\n```\n\n---\n\n## 📋 Agent prompt (copyable)\n\n````markdown\n# Build a new application from the Aetos Standards blueprint\n\nYou are coding a complete new backend application conforming to the Aetos\nstandard app blueprint. Execute every step in order. Only steps 2 and 3 use the\napp-specific details provided by the user; everything else is deterministic.\n\n## Before writing any code\n\nConnect to the `standards` MCP server and read:\n- `doc://nav` (the module list)\n- `doc://modules/genericity_matrix` (Tier A library / Tier B template / Tier C authored)\n- `doc://modules/layering_rules` (the dependency rules)\n\nThen:\n- call `find_library` with `db`, `di`, `routing`, `ingestion`, `pipeline`, `docs`\n and note the Tier A libraries that already exist — never reimplement one;\n- call `list_app_components()` and pull each file you need with\n `get_app_component(kind, resource=..., family=...)` instead of writing it from\n scratch.\n\n## Steps\n\n1. **Scaffold** — copy `standards/src/templates/app/` into `apps//`.\n Keep the tree: `domain/`, `infra/`, `interfaces/`, `pipelines/`, `tests/`,\n `scripts/`, plus `pyproject.toml`, `openapi.yaml`, `config.py`, `init.py`,\n `main.py`, `worker.py`, `Dockerfile`, `docker-compose.yaml`, `.env.example`.\n\n2. **Author the spec (app-specific)** — write `openapi.yaml`: resources,\n schemas, `operationId`s, security, and extension metadata. This is the\n contract and drives all routing.\n\n3. **Write each CRUD slice (app-specific)** — per resource create the four-file\n slice: `models.py`, `ports/db.py`, `ports/service.py`, and optional\n `converters.py`. Follow the CRUD template.\n\n4. **Wire the composition root** — in `init.py`, build leaf repositories, then\n dependent repositories, then services; add infra adapters (db, cache, stores,\n progress bus, fetchers). Expose `get_svc` overloads.\n\n5. **Add routes** — create `interfaces//routes.py` thin handlers bound\n to `operationId`s and re-export them in `interfaces/openapi.py`. Add the cache\n decorator to read endpoints.\n\n6. **Bootstrap app and worker** — confirm `main.py` runs the OpenAPI-first app\n with its lifespan. If the app has long-running jobs, adapt `worker.py` to poll\n the job collection, run the pipeline, and publish progress.\n\n7. **Add pipeline families (only if data-driven)** — copy `pipelines/example/`\n to `pipelines//` with `abc.py`, `impl.py`, `pipeline.yaml`,\n `resources/`, and reuse `pipelines/shared/` (loader + registry). Register the\n root. Add fixtures and a parity test.\n\n8. **Add infrastructure only where needed** — Tier A capability: use the\n library. Tier B capability: copy the template. Never hand-write a Tier A\n library.\n\n9. **Tests** — integration tests with fakes at the HTTP edge; unit tests for\n domain and pipeline logic; a config-consistency test for rule-driven\n pipelines.\n\n10. **Scripts and seeds** — operator scripts under `scripts/` using\n `init()` / `get_svc()` / `shutdown()`; run as `python -m scripts.`.\n\n11. **Docs** — author GSDFC docstrings, generate `.pyi` stubs, mark the package\n `py.typed`, and build the docs.\n\n**Definition of done:**\n- `python -c \"import main\"` succeeds.\n- `pytest` passes.\n- The app serves every operation in the spec; the worker drains its queue.\n- No domain module imports a web framework.\n- No Tier A library was reimplemented.\n````\n\n---\n\n## 🧭 Where the steps land\n\nThe steps map onto the layers as follows:\n\n| Step | Package / file | Depends on |\n|---------------------|-----------------------------------|---------------------------------------------|\n| 1 · Scaffold | whole `templates/app/` | — |\n| 2 · Spec | `openapi.yaml` | — |\n| 3 · CRUD slices | `domain//` | spec, CRUD template |\n| 4 · Wiring | `init.py` | every slice |\n| 5 · Routes | `interfaces//` | services, openapi route registry |\n| 6 · Bootstrap | `main.py`, `worker.py` | composition root |\n| 7 · Pipelines | `pipelines//`, `shared/` | library pipeline engine |\n| 8 · Infra | `infra/` | domain ports |\n| 9 · Tests | `tests/` | app + fixtures |\n| 10 · Scripts | `scripts/` | composition root |\n| 11 · Docs | docstrings + docs build | code, Doc Forge |\n\n---\n\n## ⚠️ Definition of done\n\n- `python -c \"import main\"` succeeds.\n- `pytest` passes.\n- The app serves the spec's operations; the worker drains its queue.\n- No domain module imports a web framework.\n- No Tier A library was reimplemented.\n\n---\n\n## Related\n\n- [02 – Layering Rules](02_layering_rules.md) · [05 – CRUD Template](05_crud_template.md) · [06 – Pipeline Scaffold](06_pipeline_scaffold.md) · [12 – Genericity Matrix](12_genericity_matrix.md)\n",
+ "objects": {}
+ }
+}
\ No newline at end of file
diff --git a/standards/mcp/modules/index.json b/standards/mcp/modules/index.json
index 625805d..0f880f9 100644
--- a/standards/mcp/modules/index.json
+++ b/standards/mcp/modules/index.json
@@ -2,7 +2,7 @@
"module": "index",
"content": {
"path": "index",
- "docstring": "# Standards Reference\n\nThe reference half of the Standards description: the enforceable rules,\npatterns, and the internal library catalog that agents consult while coding.\n\n- **Layering Rules** — the strict request → domain → repository layering and\n the rules that enforce it.\n- **Composition Root** — a single composition point wiring the whole app.\n- **Spec-First Routing** — the OpenAPI contract as the routing source of truth.\n- **CRUD Template** — the standardized generic resource endpoint scaffold.\n- **Pipeline Scaffold** — the deterministic pipeline skeleton for multi-stage\n jobs.\n- **Worker & Progress** — queued work, progress reporting, and visibility.\n- **Human-in-the-Loop** — guardrails and approvals inside automations.\n- **Infra Adapters** — how infrastructure connects without leaking into domain.\n- **Genericity Matrix** — what is library-backed, template-provided, or authored.\n- **Library Registry** — the catalog of internal libraries behind all of it.\n\nRead the **Wiki** for the philosophy and rationale; these pages are the\nreference that mirrors the `standards` MCP bundle.\n",
+ "docstring": "# Standards Reference\n\nThe reference half of the Standards description: the enforceable rules,\npatterns, and the internal library catalog that agents consult while coding.\n\n- **Build Recipe** — the ordered, agentic checklist for a new app.\n- **Layering Rules** — the strict request → domain → repository layering and\n the rules that enforce it.\n- **Composition Root** — a single composition point wiring the whole app.\n- **Spec-First Routing** — the OpenAPI contract as the routing source of truth.\n- **CRUD Template** — the standardized generic resource endpoint scaffold.\n- **Pipeline Scaffold** — the deterministic pipeline skeleton for multi-stage\n jobs.\n- **Worker & Progress** — queued work, progress reporting, and visibility.\n- **Human-in-the-Loop** — guardrails and approvals inside automations.\n- **Infra Adapters** — how infrastructure connects without leaking into domain.\n- **Genericity Matrix** — what is library-backed, template-provided, or authored.\n- **Library Registry** — the catalog of internal libraries behind all of it.\n\nRead the **Wiki** for the philosophy and rationale; these pages are the\nreference that mirrors the `standards` MCP bundle.\n",
"objects": {}
}
}
\ No newline at end of file
diff --git a/standards/mcp/nav.json b/standards/mcp/nav.json
index 6a08db3..5b0b62b 100644
--- a/standards/mcp/nav.json
+++ b/standards/mcp/nav.json
@@ -1,4 +1,9 @@
[
+ {
+ "module": "build_recipe",
+ "resource": "doc://modules/build_recipe",
+ "title": "Build Recipe"
+ },
{
"module": "layering_rules",
"resource": "doc://modules/layering_rules",
diff --git a/standards/wiki/01_overview/index.html b/standards/wiki/01_overview/index.html
index 3428ef7..b82f282 100644
--- a/standards/wiki/01_overview/index.html
+++ b/standards/wiki/01_overview/index.html
@@ -259,13 +259,13 @@
-
+
- Build Recipe
+ Start an App
@@ -486,12 +486,12 @@
-
+
- Build Recipe
+ Start an App
@@ -668,7 +668,7 @@ root: