1 line
67 KiB
JSON
1 line
67 KiB
JSON
{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Standards Reference","text":"<p>The reference half of the Standards description: the enforceable rules, patterns, and the internal library catalog that agents consult while coding.</p> <ul> <li>Build Recipe \u2014 the ordered, agentic checklist for a new app.</li> <li>Layering Rules \u2014 the strict request \u2192 domain \u2192 repository layering and the rules that enforce it.</li> <li>Composition Root \u2014 a single composition point wiring the whole app.</li> <li>Spec-First Routing \u2014 the OpenAPI contract as the routing source of truth.</li> <li>CRUD Template \u2014 the standardized generic resource endpoint scaffold.</li> <li>Pipeline Scaffold \u2014 the deterministic pipeline skeleton for multi-stage jobs.</li> <li>Worker & Progress \u2014 queued work, progress reporting, and visibility.</li> <li>Human-in-the-Loop \u2014 guardrails and approvals inside automations.</li> <li>Infra Adapters \u2014 how infrastructure connects without leaking into domain.</li> <li>Drone Pipelines \u2014 the two CI shapes: a PyPI package or a Docker deployment.</li> <li>Versioning & Changelog \u2014 SemVer, the tag\u2194version contract, and the release changelog.</li> <li>Genericity Matrix \u2014 what is library-backed, template-provided, or authored.</li> <li>Auth Integration \u2014 consuming the shared auth-server service (spec-driven, no auth code).</li> <li>Library Registry \u2014 the catalog of internal libraries behind all of it.</li> </ul> <p>Read the Wiki for the philosophy and rationale; these pages are the reference that mirrors the <code>standards</code> MCP bundle.</p>"},{"location":"01_build_recipe/","title":"Build Recipe","text":""},{"location":"01_build_recipe/#summary","title":"Summary","text":"<p>The ordered, deterministic checklist for generating a new application of the standard shape. Every step is fixed; only the resource model (step 2), the codegen pass (step 3), and the pipeline family (step 8) need app-specific work.</p> <p>This page is served by the <code>standards</code> MCP as <code>docs://modules/build_recipe</code>. The <code>start_app</code> prompt on that server collects the two axes and hands off here; pull exact files with <code>get_app_component</code>.</p>"},{"location":"01_build_recipe/#bootstrap","title":"\ud83d\ude80 Bootstrap","text":"<p>From the standards docs checkout:</p> <pre><code>cp -r <docs>/standards/src/templates/app apps/<your-app>\ncd apps/<your-app>\n</code></pre> <p>Install the internal libraries \u2014 editable local checkouts (preferred in development) or the private index (CI / fresh machines). Credentials are environment-only; never commit them.</p> <pre><code># local checkouts\npip install -e ../aetos/openapi-first -e ../aetos/mongo-ops -e ../aetos/hexa \\\n -e ../aetos/dagpipe -e ../aetos/doc-forge\n# private index\npip install --extra-index-url \\\n \"https://$PIP_REGISTRY_USERNAME:$PIP_REGISTRY_PASSWORD@pip.aetoskia.com/simple\" .\n</code></pre>"},{"location":"01_build_recipe/#agent-prompt-copyable","title":"\ud83d\udccb Agent prompt (copyable)","text":"<pre><code># Build a new application from the Aetos Standards blueprint\n\nYou are coding a complete new backend application conforming to the Aetos\nstandard app blueprint. Execute every step in order. Only steps 2\u20134 use the\napp-specific details provided by the user; everything else is deterministic.\n\n## Before writing any code\n\nConnect to the `standards` MCP server and read:\n- `docs://nav` (the module list)\n- `docs://modules/genericity_matrix` (Tier A library / Tier B template / Tier C authored)\n- `docs://modules/layering_rules` (the dependency rules)\n\nThen:\n- call `find_library` with `db`, `di`, `routing`, `ingestion`, `pipeline`, `docs`\n and note the Tier A libraries that already exist \u2014 never reimplement one;\n- call `list_app_components()` and pull each file you need with\n `get_app_component(kind, resource=..., family=...)` instead of writing it from\n scratch.\n\n## Steps\n\n1. **Scaffold** \u2014 copy `standards/src/templates/app/` into `apps/<your-app>/`\n (or pull the same tree via `get_app_template`). Keep the tree: `domain/`,\n `infra/`, `interfaces/`, `pipelines/`, `tests/`, `scripts/`, plus\n `pyproject.toml`, `openapi.yaml`, `config.py`, `init.py`, `main.py`,\n `worker.py`, `Dockerfile`, `docker-compose.yaml`, `.env.example`.\n\n2. **Author the spec (app-specific)** \u2014 write `openapi.yaml`: resources,\n schemas, `operationId`s, security, and extension metadata. This is the\n contract and drives all routing. Every operation MUST have an `operationId`.\n\n3. **Generate the scaffold (codegen)** \u2014 from the spec, generate the wire\n models and the exact handler signatures:\n ```bash\n openapi-first models openapi.yaml -o _gen/models.py\n openapi-first routes openapi.yaml -o _gen/routes \\\n --use-models --models-module _gen.models --inject-request --package --template-style\n ```\n `_gen/models.py` is the **wire contract** (Pydantic, one class per\n `components.schemas` entry). `_gen/routes/<resource>/routes.py` lists every\n `operationId` with its exact parameter signature. `--template-style` makes\n those stubs `async def` with a `body` parameter, matching the handlers you\n keep in `interfaces/` (without it they are sync `def` stubs taking\n `payload`; both bind the same way). Treat `_gen/` as generated\n (never hand-edit it) \u2014 the canonical handlers live in `interfaces/`.\n\n4. **Write each CRUD slice (app-specific)** \u2014 per resource create the four-file\n slice: `models.py` (domain model, **no `id`**), `ports/db.py`\n (`class XDocument(BaseDocument, X)` \u2014 id comes from `BaseDocument`),\n `ports/service.py`, and optional `converters.py`. Follow the CRUD template.\n\n5. **Wire the composition root** \u2014 in `init.py`, build leaf repositories, then\n dependent repositories, then services; add infra adapters (db, cache, stores,\n progress bus, fetchers). Expose `get_svc` overloads.\n\n6. **Add routes** \u2014 create `interfaces/<resource>/routes.py` thin handlers whose\n names match the `operationId`s (use the signatures from step 3 as your\n checklist) and re-export them in `interfaces/openapi.py`. Add the cache\n decorator to read endpoints.\n\n7. **Bootstrap app and worker** \u2014 confirm `main.py` runs the OpenAPI-first app\n with its lifespan. If the app has long-running jobs, adapt `worker.py` to poll\n the job collection, run the pipeline, and publish progress.\n\n8. **Add pipeline families (only if data-driven)** \u2014 copy `pipelines/example/`\n to `pipelines/<family>/` with `abc.py`, `impl.py`, `pipeline.yaml`,\n `resources/`, and reuse `pipelines/shared/` (loader + registry). Register the\n root. Add fixtures and a parity test.\n\n9. **Add infrastructure only where needed** \u2014 Tier A capability: use the\n library. Tier B capability: copy the template. Never hand-write a Tier A\n library.\n\n10. **Tests** \u2014 integration tests with fakes at the HTTP edge; unit tests for\n domain and pipeline logic; a config-consistency test for rule-driven\n pipelines. Include `tests/integration/test_openapi_app_builds.py` (imports\n the app, asserts `import main` binds every operation).\n\n11. **Scripts and seeds** \u2014 operator scripts under `scripts/` using\n `init()` / `get_svc()` / `shutdown()`; run as `python -m scripts.<name>`.\n\n12. **Docs** \u2014 author GSDFC docstrings, generate `.pyi` stubs, mark the package\n `py.typed`, and build the docs.\n\n**Definition of done:**\n- `python -c \"import main\"` succeeds.\n- `pytest` passes.\n- The app serves every operation in the spec; the worker drains its queue.\n- No domain module imports a web framework.\n- No Tier A library was reimplemented.\n</code></pre>"},{"location":"01_build_recipe/#where-the-steps-land","title":"\ud83e\udded Where the steps land","text":"<p>The steps map onto the layers as follows:</p> Step Package / file Depends on 1 \u00b7 Scaffold whole <code>templates/app/</code> \u2014 2 \u00b7 Spec <code>openapi.yaml</code> \u2014 3 \u00b7 Codegen <code>_gen/</code> (generated) spec, openapi-first 4 \u00b7 CRUD slices <code>domain/<resource>/</code> spec, CRUD template 5 \u00b7 Wiring <code>init.py</code> every slice 6 \u00b7 Routes <code>interfaces/<resource>/</code> services, openapi route registry 7 \u00b7 Bootstrap <code>main.py</code>, <code>worker.py</code> composition root 8 \u00b7 Pipelines <code>pipelines/<family>/</code>, <code>shared/</code> library pipeline engine 9 \u00b7 Infra <code>infra/</code> domain ports 10 \u00b7 Tests <code>tests/</code> app + fixtures 11 \u00b7 Scripts <code>scripts/</code> composition root 12 \u00b7 Docs docstrings + docs build code, Doc Forge"},{"location":"01_build_recipe/#the-codegen-fast-path-in-detail","title":"\ud83d\udd27 The codegen fast path, in detail","text":"<p><code>openapi-first</code> is spec \u2192 code only (it never generates a spec). Use it to erase the two mechanical parts of the build:</p> Generated artifact Replaces Notes <code>_gen/models.py</code> hand-writing Pydantic wire models one class per <code>components.schemas</code> entry <code>_gen/routes/<resource>/routes.py</code> hand-typing handler signatures names == <code>operationId</code>s; <code>request: Request</code> injected <p>Rules of the road:</p> <ul> <li>Wire models are not domain models. <code>_gen/models.py</code> is the HTTP contract; <code>domain/<resource>/models.py</code> is your business shape. If the spec's schema has a required <code>id: str</code>, keep it off the domain model and let the document get its id from <code>BaseDocument</code>.</li> <li>Use <code>--template-style</code>. Stubs are then <code>async def ...(request, body: ...)</code> matching <code>interfaces/</code>; without it they are sync <code>def ...(payload: ...)</code>. Either binds (the router introspects the signature), but <code>--template-style</code> keeps the scaffold consistent with the code you keep.</li> <li>Resource names are sanitized. <code>/fetch-requests</code> \u2192 <code>fetch_requests.py</code>.</li> <li>Multipart bodies expand to <code>UploadFile = File(...)</code> / <code>Form(...)</code> params.</li> <li><code>_gen/</code> is disposable. Regenerate on every spec change; <code>git diff</code> shows exactly what moved.</li> </ul>"},{"location":"01_build_recipe/#definition-of-done","title":"\u26a0\ufe0f Definition of done","text":"<ul> <li><code>python -c \"import main\"</code> succeeds.</li> <li><code>pytest</code> passes.</li> <li>The app serves the spec's operations; the worker drains its queue.</li> <li>No domain module imports a web framework.</li> <li>No Tier A library was reimplemented.</li> </ul>"},{"location":"01_build_recipe/#related","title":"Related","text":"<ul> <li>02 \u2013 Layering Rules \u00b7 05 \u2013 CRUD Template \u00b7 06 \u2013 Pipeline Scaffold \u00b7 12 \u2013 Genericity Matrix</li> </ul>"},{"location":"02_layering_rules/","title":"Layering Rules","text":""},{"location":"02_layering_rules/#summary","title":"Summary","text":"<p>The dependency direction is fixed. Violating it is a design defect.</p>"},{"location":"02_layering_rules/#allowed-imports","title":"Allowed imports","text":"<pre><code>interfaces/* -> domain/* (services, ports, models)\ninterfaces/* -> infra/adapters/* (only read-only value objects)\ndomain/* -> domain/* (same layer)\ndomain/* -> pipelines/* (call a pipeline via a port)\ndomain/*/ports/* -> infra/* (bind adapters + libraries at the seam)\ninfra/adapters/* -> domain/ports/* (implement the interface)\nmain.py/init.py -> everything (composition root only)\n</code></pre>"},{"location":"02_layering_rules/#forbidden-imports","title":"Forbidden imports","text":"<pre><code>domain/* -> interfaces/* (never)\ndomain/* -> fastapi/starlette (never)\ninfra/adapters/* -> interfaces/* (never)\ninterfaces/* -> infra/db/* (state lives in resources, not imports)\n</code></pre>"},{"location":"02_layering_rules/#why","title":"Why","text":"<ul> <li>The business logic is testable without a server or an event loop.</li> <li>Adapters can be swapped (in-memory fake, real database) without touching business logic.</li> <li>The HTTP edge stays thin and replaceable.</li> </ul>"},{"location":"02_layering_rules/#the-portadapter-contract","title":"The port/adapter contract","text":"<p>The <code>ports/</code> package is the seam where the domain meets infrastructure. Each <code>domain/<resource>/ports/</code> module declares the resource's models/document shape and exposes factory functions that bind a concrete adapter \u2014 so a port module is allowed to import from <code>infra/</code> and from the persistence libraries. Everything outside <code>ports/</code> in <code>domain/</code> stays framework- and driver-free.</p> <pre><code>domain/<resource>/ports/db.py -> document model + create_<resource>_repo()\ndomain/<resource>/ports/service.py -> service + get_<resource>_service()\ninfra/db/repos/<vendor>.py -> repository adapter factory\ninfra/adapters/service.py -> generic CRUD service base\n</code></pre> <p>The composition root calls the port factories; nothing else constructs an adapter.</p>"},{"location":"02_layering_rules/#enforcement","title":"Enforcement","text":"<ul> <li><code>domain/</code> must never import the web framework (<code>fastapi</code> / <code>starlette</code>) or <code>interfaces/</code>. Only the <code>ports/</code> seam, <code>infra/</code>, and the composition root touch drivers.</li> <li>Reviewers (and CI) reject <code>import fastapi</code> inside <code>domain/</code> or <code>infra/</code>.</li> </ul>"},{"location":"03_composition_root/","title":"Composition Root","text":""},{"location":"03_composition_root/#summary","title":"Summary","text":"<p>All wiring happens in exactly one module. It builds a flat resource registry and exposes it to the HTTP layer and to scripts.</p>"},{"location":"03_composition_root/#the-resource-registry","title":"The resource registry","text":"<p><code>init.py</code> returns a plain <code>dict[str, Any]</code>. Keys follow two conventions:</p> <ul> <li><code><resource>_repo</code> \u2014 a repository adapter for a resource.</li> <li><code><resource>_svc</code> \u2014 a service for a resource.</li> </ul> <p>Plus named infrastructure adaptors:</p> <ul> <li><code>http_cache_store</code> \u2014 response cache backend.</li> <li><code>progress_bus</code> \u2014 progress/event bus.</li> <li><code><vendor>_<kind></code> \u2014 fetchers, clients, stores.</li> </ul> <p>The API stores the dict on <code>app.state.db_resources</code> during lifespan startup. Handlers read from it; nothing imports concrete adapters.</p>"},{"location":"03_composition_root/#shape","title":"Shape","text":"<pre><code>_resources: dict = {}\n_initialized = False\n\nasync def init(settings=None) -> dict:\n global _initialized\n if _initialized:\n return _resources\n\n await setup_database(uri=settings.mongo_uri, db_name=settings.mongo_db)\n\n # 1. leaf repositories\n user_repo = create_user_repo()\n\n # 2. dependent repositories (dependencies passed by keyword)\n wallet_repo = create_wallet_repo(user_repo=user_repo)\n\n # 3. services (build via the DI library, injecting the repo)\n _resources[\"user_repo\"] = user_repo\n _resources[\"wallet_repo\"] = wallet_repo\n _resources[\"user_svc\"] = get_user_service(user_repo)\n _resources[\"wallet_svc\"] = get_wallet_service(wallet_repo)\n\n # 4. infrastructure adaptors (optional)\n _resources[\"http_cache_store\"] = HttpCacheStore(...)\n _resources[\"progress_bus\"] = ProgressEventBus()\n\n _initialized = True\n return _resources\n\nasync def shutdown() -> None:\n global _initialized\n await close_stores(_resources)\n await shutdown_database()\n _resources.clear()\n _initialized = False\n</code></pre>"},{"location":"03_composition_root/#typed-accessors","title":"Typed accessors","text":"<p>Expose a <code>get_svc(name)</code> helper with one <code>@overload</code> per key so scripts get typing without importing concrete classes:</p> <pre><code>@overload\ndef get_svc(name: Literal[\"user_svc\"]) -> UserService: ...\n@overload\ndef get_svc(name: Literal[\"wallet_svc\"]) -> WalletService: ...\n\ndef get_svc(name: str):\n return _resources[name]\n</code></pre> <p>Also expose <code>get_db()</code> (raw database handle) and pure helpers such as <code>compute_fingerprint(payload)</code> (canonical hash used for deduplication).</p>"},{"location":"03_composition_root/#rules","title":"Rules","text":"<ul> <li><code>init()</code> is idempotent.</li> <li>The composition root is the only place that imports concrete adapters.</li> <li>Handlers never call <code>init()</code>; they read <code>request.app.state.db_resources</code>.</li> <li>Scripts call <code>init()</code> / <code>shutdown()</code> around their work.</li> </ul>"},{"location":"04_spec_first_routing/","title":"Spec-First Routing","text":""},{"location":"04_spec_first_routing/#summary","title":"Summary","text":"<p>The OpenAPI document is the single source of truth for the HTTP contract. Handlers are bound to it by name; there is no decorator router.</p>"},{"location":"04_spec_first_routing/#the-contract","title":"The contract","text":"<ul> <li><code>openapi.yaml</code> (or <code>.json</code>) declares paths, schemas, parameters, responses, and security.</li> <li>A custom <code>operationId</code> names the handler function.</li> <li>The framework (<code>openapi_first</code>) validates requests/responses against the spec and dispatches to the handler whose name equals <code>operationId</code>.</li> </ul>"},{"location":"04_spec_first_routing/#app-bootstrap","title":"App bootstrap","text":"<pre><code># main.py\nfrom openapi_first.app import OpenAPIFirstApp\n\napp = OpenAPIFirstApp(\n openapi_path=\"openapi.yaml\",\n routes_module=interfaces.openapi, # module whose namespace holds handlers\n title=\"<App> API\",\n version=\"0.1.0\",\n lifespan=lifespan,\n)\n</code></pre> <p><code>lifespan</code> calls the composition root:</p> <pre><code>@asynccontextmanager\nasync def lifespan(app):\n app.state.db_resources = await init()\n yield\n await shutdown()\n</code></pre>"},{"location":"04_spec_first_routing/#handler-registry","title":"Handler registry","text":"<p><code>interfaces/openapi.py</code> is a flat re-export of every handler, so the framework can resolve <code>operationId</code> across the whole app:</p> <pre><code>from interfaces.item.routes import *\nfrom interfaces.wallet.routes import *\nfrom interfaces.progress.sse import *\n</code></pre>"},{"location":"04_spec_first_routing/#handler-shape","title":"Handler shape","text":"<p>Handlers are thin and synchronous with respect to business logic:</p> <pre><code>async def create_item(request: Request, body: dict):\n svc = request.app.state.db_resources[\"item_svc\"]\n return await svc.create(Item.model_validate(body))\n</code></pre> <p>Generated stubs. <code>openapi-first routes</code> emits this exact shape (<code>async def</code> handlers taking <code>body</code>) when given <code>--template-style</code>; without that flag the stub is <code>def create_item(payload: ItemInput)</code> \u2014 sync, with a <code>payload</code> parameter. Both shapes bind identically: the router resolves handlers by <code>operationId</code> and introspects the FastAPI signature, and it never inspects the request body. Pass <code>--template-style</code> so the generated signatures match the handlers you keep in <code>interfaces/</code>.</p>"},{"location":"04_spec_first_routing/#conventions","title":"Conventions","text":"<ul> <li><code>operationId</code> naming: <code>list_<resource></code>, <code>create_<resource></code>, <code>get_<resource></code>, <code>update_<resource></code>, <code>delete_<resource></code>.</li> <li>Read-only endpoints may use a cache decorator keyed on path + query + user.</li> <li>Validation errors -> <code>400</code>; not found -> <code>404</code>; delete -> <code>204</code>.</li> <li>Security is declared once in the spec and enforced by the framework; handlers contain no auth code.</li> </ul>"},{"location":"04_spec_first_routing/#authless-apps","title":"Authless apps","text":"<p>Auth is deferred in the standard: the same spec-first app runs with or without it, because security lives entirely in the spec, never in handlers. For an app that needs no auth yet, do not invent per-handler auth checks \u2014 just omit the security keys:</p> <ul> <li>Remove the top-level <code>security:</code> block.</li> <li>Remove <code>components.securitySchemes</code>.</li> <li>Do not add <code>security: []</code> to operations. An explicit empty array is a per-operation override that means \"public\", and it only matters when a global <code>security</code> block exists. Leaving every operation without a <code>security</code> key is the cleanest authless spec.</li> </ul> <p>Profile endpoints (<code>/users/me</code>) and profile headers (<code>x-profile</code>) still exist in an authless app. Define a stand-in identity in the composition root \u2014 for example <code>get_current_user</code> returns or lazily creates a fixed \"Demo User\" \u2014 so handlers stay auth-agnostic and only the wiring changes when real auth arrives.</p> <p>The handlers, services, and composition root are unchanged. When auth is needed later, add the global <code>security</code> block and an introspecting dependency; only handlers that must opt out get <code>security: []</code>.</p>"},{"location":"04_spec_first_routing/#codegen","title":"Codegen","text":"<p><code>openapi_first</code> scaffolds models and route stubs from the spec (see the Build Recipe, step 3):</p> <pre><code>openapi-first models openapi.yaml -o _gen/models.py\nopenapi-first routes openapi.yaml -o _gen/routes \\\n --use-models --models-module _gen.models --inject-request --package --template-style\n</code></pre> <p>Prefer codegen over hand-writing boilerplate. The result is the wire contract and a set of <code>operationId</code>-named signatures \u2014 not the canonical handlers.</p>"},{"location":"05_crud_template/","title":"CRUD Template","text":""},{"location":"05_crud_template/#summary","title":"Summary","text":"<p>Every resource follows the same four-file slice. Repetition is intentional: it makes the codebase predictable and generatable.</p>"},{"location":"05_crud_template/#the-slice","title":"The slice","text":"<pre><code>domain/<resource>/\n models.py # pure Pydantic model (no DB fields)\n ports/\n db.py # document model + repository factory\n service.py # service class + factory\n converters.py # (optional) transforms\n</code></pre>"},{"location":"05_crud_template/#the-id-rule-model-vs-document","title":"The id rule (model vs. document)","text":"<p>Two different objects, two different jobs:</p> Object Lives in Has <code>id</code>? Purpose Domain model <code>domain/<resource>/models.py</code> No business shape, framework-free Document <code>domain/<resource>/ports/db.py</code> Yes (from <code>BaseDocument</code>) Mongo persistence shape <p><code>BaseDocument</code> (mongo-ops) supplies <code>id: Optional[PyObjectId]</code> (aliased to <code>_id</code>) plus <code>created_at</code> / <code>updated_at</code>. The domain model must not declare <code>id</code>; Mongo owns it.</p> <p>Trap \u2014 generated wire models. <code>openapi-first models</code> produces the HTTP contract (see the Build Recipe, step 3). Response schemas frequently declare a required <code>id: str</code>, so a generated class is not a domain model. If you write <code>class ItemDocument(Item, BaseDocument)</code> and <code>Item</code> re-exports a required <code>id: str</code>, that field shadows <code>BaseDocument.id</code> and no Mongo <code>_id</code> can ever be set. Fix it one of three ways:</p> <ol> <li>Preferred: keep the domain model id-free (hand-author it from the CRUD template), and keep the required <code>id</code> only on the wire model.</li> <li>Order the bases so the document wins: <code>class ItemDocument(BaseDocument, Item)</code>.</li> <li>Keep a separate document type and translate with <code>ports/converters.py</code>.</li> </ol>"},{"location":"05_crud_template/#modelspy","title":"models.py","text":"<p>A framework-free Pydantic model describing the business shape (no <code>id</code>).</p> <pre><code>class Item(BaseModel):\n name: str\n quantity: int = 0\n</code></pre>"},{"location":"05_crud_template/#portsdbpy","title":"ports/db.py","text":"<p>The persistence model extends the business model and adds database concerns. The repository factory selects the right repository flavour.</p> <pre><code># Item is id-free; BaseDocument adds id/created_at/updated_at.\nclass ItemDocument(Item, BaseDocument):\n pass\n\ndef create_item_repo(*, owner_repo=None) -> ItemRepository:\n return repo_factory(\n collection=\"items\",\n document=ItemDocument,\n populate_rules=[PopulateRule(field_name=\"owner\", collection_name=\"users\")],\n cache=None, # or a CachedBaseRepository config\n )\n</code></pre>"},{"location":"05_crud_template/#nested-references","title":"Nested references","text":"<p>The wire contract often declares a reference as an expanded object (<code>expense.entity: Entity</code>), while the domain model stays id-free. Pick one convention and apply it consistently:</p> <ul> <li>Embed (default for small, stable refs) \u2014 store the object as a <code>dict</code> / <code>list[dict]</code> on the document; the API returns it verbatim. No populate wiring.</li> <li>Populate \u2014 store an id and declare a <code>PopulateRule</code>; mongo-ops expands it on read. The referenced repo must be registered with the <code>PopulationEngine</code>.</li> <li>Id only \u2014 store the id and let the client resolve it.</li> </ul> <p>For populate-backed repositories change a reference with <code>update</code>, not <code>patch</code>: <code>patch</code> rejects changes to populated FK fields by design.</p>"},{"location":"05_crud_template/#portsservicepy","title":"ports/service.py","text":"<p>The service is built through the DI library, injecting the repository. The generic CRUD base provides <code>create/read/update/patch/delete/search/...</code>.</p> <pre><code>class ItemService(CrudsServiceAdaptor[Item, ItemDocument]):\n model = Item\n schema = ItemDocument\n\ndef get_item_service(repo) -> ItemService:\n return build(ItemService, instances={\"repo\": repo})\n</code></pre>"},{"location":"05_crud_template/#naming-conventions","title":"Naming conventions","text":"Concept Convention collection plural snake case (<code>items</code>) document <code><Resource>Document</code> repository factory <code>create_<resource>_repo</code> service <code><Resource>Service</code> service factory <code>get_<resource>_service</code> resource key <code><resource>_svc</code> operationId <code>list_items</code>, <code>create_item</code>, ..."},{"location":"05_crud_template/#optional-files","title":"Optional files","text":"<ul> <li><code>converters.py</code> \u2014 pure functions mapping between domain types.</li> <li><code>ports/enrichment.py</code> \u2014 adapts a pipeline into an async callable the service can invoke (used when a resource is populated by a pipeline).</li> </ul>"},{"location":"06_pipeline_scaffold/","title":"Pipeline Scaffold","text":""},{"location":"06_pipeline_scaffold/#summary","title":"Summary","text":"<p>Data-driven processing is expressed as a composition graph. The graph is built by a DI library; execution is driven by a DAG engine. Stages are pure and individually testable.</p>"},{"location":"06_pipeline_scaffold/#when-to-use-a-pipeline","title":"When to use a pipeline","text":"<p>Use a pipeline when a resource is produced by a multi-step transformation of an input (a document, a file, a message) rather than by a simple CRUD write.</p>"},{"location":"06_pipeline_scaffold/#structure-of-a-family","title":"Structure of a family","text":"<p>A family is one input shape (e.g. a file format or a message kind). Each family has the same skeleton:</p> <pre><code>pipelines/<family>/\n abc.py # port contracts (stage interfaces + root interface)\n impl.py # concrete stages + root implementation\n pipeline.yaml # graph spec: nodes, edges, roots, schema\n resources/\n pipeline.yaml # runtime spec consumed by the loader\n patterns/*.yaml # rule fragments (pattern registry)\n banks/<variant>.py # optional variants that re-pin a few slots\n</code></pre>"},{"location":"06_pipeline_scaffold/#shared-scaffolding","title":"Shared scaffolding","text":"<pre><code>pipelines/shared/\n ports.py # RegistryPort, FinalizePort, etc.\n loader.py # build(root_cls) -> wires the graph + engine\n registry.py # resolve_root(...) -> root class for a (kind, variant)\n</code></pre>"},{"location":"06_pipeline_scaffold/#the-loader","title":"The loader","text":"<pre><code>def build(root_cls, *, instances=None, config=None):\n # merge resources/patterns/*.yaml into a pattern registry\n # bind registry/config into each graph node by id\n # build the schema (state) class and the engine\n root = root_cls()\n root._engine = Engine(graph, state_cls, payload_cls)\n return root\n</code></pre>"},{"location":"06_pipeline_scaffold/#the-registry","title":"The registry","text":"<p>A single lookup replaces hand-rolled <code>if/else</code> selection:</p> <pre><code>_FAMILIES = {\n \"csv\": {\"acme\": {\"heuristic\": AcmeCsvPipeline}},\n \"json\": {\"globex\": {\"heuristic\": GlobexJsonPipeline}},\n}\n\ndef resolve_root(kind, variant, mode=\"heuristic\"):\n return _FAMILIES[kind][variant][mode]\n</code></pre> <p>Adding a new variant means adding one class and one registry entry.</p>"},{"location":"06_pipeline_scaffold/#driving-a-pipeline","title":"Driving a pipeline","text":"<p>Stages are driven explicitly so intermediate state can be persisted:</p> <pre><code>pipeline = build(resolve_root(kind, variant))\ncontent = pipeline.ingest.materialize(raw_bytes)\nrecords = pipeline.reader.read(content)\nitems = pipeline.normalize.build(records)\n</code></pre>"},{"location":"06_pipeline_scaffold/#testing","title":"Testing","text":"<p>Because stages are pure, test each stage in isolation and test the assembled root against fixture inputs whose expected outputs are stored as golden files. A parity test replays fixtures headlessly.</p>"},{"location":"07_worker_and_progress/","title":"Worker & Progress","text":""},{"location":"07_worker_and_progress/#summary","title":"Summary","text":"<p>Long-running work runs in a background worker. Jobs are durable documents that move through an explicit status machine. Progress is streamed to clients.</p>"},{"location":"07_worker_and_progress/#the-status-machine","title":"The status machine","text":"<pre><code>pending -> processing -> completed\n \\-> failed (retried up to N times)\n \\-> paused (waiting for human input)\n</code></pre> <ul> <li><code>pending</code> \u2014 queued, eligible for pickup.</li> <li><code>processing</code> \u2014 claimed by a worker.</li> <li><code>paused</code> \u2014 blocked on a human-in-the-loop question.</li> <li><code>failed</code> \u2014 error; <code>retry_count</code> increments until a max, then skipped.</li> <li><code>completed</code> \u2014 terminal success.</li> </ul>"},{"location":"07_worker_and_progress/#the-poll-loop","title":"The poll loop","text":"<pre><code>async def poll(resources):\n backoff = MIN_POLL\n while True:\n jobs = await job_svc.search(filter={\"status\": \"pending\"}, limit=100)\n backoff = MIN_POLL if jobs else min(backoff * 2, MAX_POLL)\n for job in jobs:\n if (job.retry_count or 0) >= MAX_RETRIES:\n continue\n await job_svc.update_status(job.id, \"processing\")\n try:\n await process(job, resources)\n except PauseForResolution:\n await job_svc.update_status(job.id, \"paused\")\n except Exception as e:\n await job_svc.update_status(\n job.id, \"failed\", error=str(e),\n retry_count=(job.retry_count or 0) + 1,\n )\n await asyncio.sleep(backoff)\n</code></pre> <p>Tunables live in environment variables: <code>WORKER_POLL_MIN</code>, <code>WORKER_POLL_MAX</code>, <code>WORKER_MAX_RETRIES</code>.</p>"},{"location":"07_worker_and_progress/#progress-events","title":"Progress events","text":"<p>A worker publishes structured progress events to a bus (a port). The HTTP edge exposes a streaming endpoint that replays events for a given job id.</p> <pre><code>class ProgressEvent(BaseModel):\n job_id: str\n step: str\n status: str # started | progress | completed | skipped | paused\n message: dict | None = None\n</code></pre> <p>The bus is an in-memory adapter by default; because it is a port, it can be swapped for a durable broker. Emit from the orchestrator at each stage:</p> <pre><code>await self.bus.publish(ProgressEvent(job_id=jid, step=\"normalize\",\n status=\"completed\", message={\"count\": n}))\n</code></pre>"},{"location":"07_worker_and_progress/#streaming-endpoint","title":"Streaming endpoint","text":"<pre><code>async def stream_job_events(request, id):\n bus = request.app.state.db_resources.get(\"progress_bus\")\n if bus is None:\n return {\"events\": []}\n\n async def gen():\n sent = 0\n while True:\n events = bus.get_events(id)\n for e in events[sent:]:\n yield f\"data: {e.model_dump_json(exclude_none=True)}\\n\\n\"\n sent += 1\n await asyncio.sleep(0.5)\n\n return StreamingResponse(gen(), media_type=\"text/event-stream\")\n</code></pre> <p>Known limitation. The default in-memory bus is per-process. If the worker and the API run as separate processes, events published by the worker are not visible to the API. For production, back the bus with a shared broker. The port boundary makes this a one-adapter change.</p>"},{"location":"08_human_in_loop/","title":"Human-in-the-Loop","text":""},{"location":"08_human_in_loop/#summary","title":"Summary","text":"<p>Some inputs are ambiguous and must be resolved by a human before processing can continue. The blueprint models this as a durable pause: an exception is raised, a question is recorded, the job pauses, and later resumes.</p>"},{"location":"08_human_in_loop/#the-mechanism","title":"The mechanism","text":"<ol> <li>During processing, a stage raises a domain exception carrying the context needed to ask the question:</li> </ol> <pre><code>raise PauseForResolution(\n job_id=jid,\n reference=\"<stable key for this decision>\",\n options=[...],\n context={...},\n)\n</code></pre> <ol> <li>The orchestrator catches it, records a question document, and re-raises so the worker marks the job <code>paused</code>.</li> <li>A client lists pending questions and answers one:</li> </ol> <pre><code>GET /jobs/{id}/questions -> list pending questions\nPOST /questions/{qid}/resolve {choice} -> record the choice\n</code></pre> <ol> <li>When the last pending question is resolved, the resolve endpoint flips the parent job back to <code>pending</code>.</li> <li>The worker picks it up again, rebuilds a resolution map from previously answered questions, and seeds it into the pipeline so the same decision is not asked twice.</li> </ol>"},{"location":"08_human_in_loop/#the-question-document","title":"The question document","text":"<pre><code>class Question(BaseDocument):\n job_id: str\n reference: str # stable key: stage|field|value\n status: str # pending | resolved\n options: list[dict]\n context: dict\n chosen: dict | None = None\n resolved_at: str | None = None\n</code></pre>"},{"location":"08_human_in_loop/#resume-map","title":"Resume map","text":"<p>The worker turns answered questions into a lookup keyed by <code>reference</code>, then injects it into the pipeline so that stage can skip re-raising:</p> <pre><code>resolved = {q.reference: q.chosen for q in await svc.search(\n filter={\"job_id\": jid, \"status\": \"resolved\"})}\n\npipeline = build(root, instances={\"pause\": handler}, config={\"resolved\": resolved})\n</code></pre>"},{"location":"08_human_in_loop/#resolve-endpoint","title":"Resolve endpoint","text":"<pre><code>async def resolve_question(request, id, body):\n svc = request.app.state.db_resources[\"question_svc\"]\n q = await svc.read(id)\n await svc.update(id, {\"chosen\": body[\"chosen\"], \"status\": \"resolved\",\n \"resolved_at\": now()})\n\n pending = await svc.search(filter={\"job_id\": q.job_id, \"status\": \"pending\"},\n limit=1)\n if not pending:\n jobs = request.app.state.db_resources[\"job_svc\"]\n job = await jobs.read(q.job_id)\n if job and job.status == \"paused\":\n await jobs.update_status(q.job_id, \"pending\")\n return q\n</code></pre>"},{"location":"08_human_in_loop/#cli-parity","title":"CLI parity","text":"<p>The same flow is available through scripts/CLI: <code>list</code>, <code>get</code>, <code>create</code>, <code>questions</code>, <code>resolve</code>, <code>retry</code>.</p>"},{"location":"09_infra_adaptors/","title":"Infra Adapters","text":""},{"location":"09_infra_adaptors/#summary","title":"Summary","text":"<p>All external systems are reached through adapters in <code>infra/</code>. Prefer an existing internal library over writing a new adapter; where no library exists, copy the Tier B scaffold from <code>templates/app</code>. See the Genericity Matrix for the Tier A / B / C split.</p>"},{"location":"09_infra_adaptors/#database-mongodb","title":"Database (MongoDB)","text":"<ul> <li>A single connection manager owns the client lifecycle. <code>setup_database(uri, db_name)</code> / <code>shutdown_database()</code>.</li> <li>A repository factory selects the repository flavour:</li> </ul> <pre><code>repo_factory(collection, document, engine=None,\n populate_rules=None, cache=None)\n</code></pre> <ul> <li>base repository \u2014 plain CRUD;</li> <li>populating repository \u2014 resolves referenced documents (foreign keys);</li> <li> <p>cached repository \u2014 an LRU/TTL cache in front of reads.</p> </li> <li> <p>Referenced documents are described declaratively:</p> </li> </ul> <pre><code>PopulateRule(field_name=\"owner\", collection_name=\"users\",\n nested_rules=[], max_depth=2, projection=None)\n</code></pre> <ul> <li>Transactions use a session context manager provided by the library.</li> </ul> <p>Tier A \u2014 Library (Mongo Ops). Do not write raw Motor calls in the domain; consult the registry and the library's MCP before touching the driver.</p>"},{"location":"09_infra_adaptors/#http-response-cache","title":"HTTP response cache","text":"<p>Tier B \u2014 template. Copy from <code>templates/app</code>; not a standalone library.</p> <p>A generic GET cache sits behind a decorator.</p> <pre><code>@cache_response(ttl=60)\nasync def list_items(request): ...\n</code></pre> <ul> <li>Keyed on path + query + user.</li> <li>No-op when no store is wired.</li> <li><code>invalidate(*resources)</code> bumps a generation counter for write-triggered invalidation.</li> </ul>"},{"location":"09_infra_adaptors/#event-progress-bus","title":"Event / progress bus","text":"<p>Tier B \u2014 template. Copy from <code>templates/app</code>.</p> <p>A port with an in-memory default adapter (<code>publish</code>, <code>get_events</code>). Swap for a durable broker in production (see Worker & Progress).</p>"},{"location":"09_infra_adaptors/#external-clients-ingestion","title":"External clients (ingestion)","text":"<p>External fetchers (files, mail, HTTP) are adapters that wrap ingestion libraries. Construct them in the composition root and inject into orchestrators.</p> <p>Tier A \u2014 Libraries: Omniread (formats) and Mail Intake (email).</p>"},{"location":"09_infra_adaptors/#llm-clients","title":"LLM clients","text":"<p>Tier B \u2014 template. Copy the pool scaffold from <code>templates/app</code>.</p> <p>When a pipeline calls a model provider, route through a pool adaptor that handles provider selection, retries, and client lifecycle:</p> <pre><code>pool = setup_async_pool(provider=settings.llm_provider)\nresult = await pool.complete(model=\"...\", messages=[...])\n</code></pre> <p>Providers are selected by <code>LLM_PROVIDER</code> and configured by environment.</p>"},{"location":"09_infra_adaptors/#key-value-store","title":"Key-value store","text":"<p>Tier B \u2014 template. Copy from <code>templates/app</code>.</p> <p>For report/snapshot storage, provide a pluggable store port with a Redis adapter and an in-memory fallback.</p>"},{"location":"09_infra_adaptors/#rule","title":"Rule","text":"<ul> <li>Tier A (libraries: DB, DI, ingestion, mail, auth, docs, pipelines) \u2014 use the library; writing a private copy is a defect. Query <code>find_library</code> first.</li> <li>Tier B (cache, progress bus, LLM pool, KV store, composition shape, worker, tests) \u2014 these are template code, not libraries: copy them from <code>templates/app</code> rather than searching the registry.</li> <li>Tier C (models, business logic, pipeline families, rules) \u2014 author them yourself.</li> </ul>"},{"location":"10_drone_pipelines/","title":"Drone Pipelines","text":""},{"location":"10_drone_pipelines/#summary","title":"Summary","text":"<p>Every Aetos repository ships a Drone CI definition at <code>.drone.yml</code>. There is exactly one standard per repository, and there are two types of file, chosen by what the repository produces:</p> Type File Produces Repos A \u2014 PyPI package <code>templates/drone/python-lib.yaml</code> A wheel + sdist on the private index <code>dagpipe</code>, <code>doc-forge</code>, <code>hexa</code>, <code>mail-intake</code>, <code>mongo-ops</code>, <code>omniread</code>, <code>openapi-first</code>, <code>py-jwt</code> B \u2014 Docker deployment <code>templates/drone/service-docker.yaml</code> An image pushed to the registry and run as a container <code>auth-server</code>, <code>docs</code>, <code>mcp</code> <p>Both templates run on <code>platform: linux / arm64</code> with the workspace at <code>/drone/src</code>. Copy the matching template to <code><repo>/.drone.yml</code>; the Type A file needs no edits, the Type B file needs its placeholders filled in (below).</p> <p>Conventions: <code>.drone.yml</code> (not <code>.yaml</code>), one file per repo, secrets injected with <code>from_secret</code> \u2014 never commit credentials.</p>"},{"location":"10_drone_pipelines/#type-a-pypi-package-python-libyaml","title":"Type A \u2014 PyPI package (<code>python-lib.yaml</code>)","text":"<p>For internal libraries. The file contains two pipelines:</p> <ol> <li><code>build-and-publish-pypi</code> \u2014 triggered on <code>tag</code>:</li> <li><code>check-version</code> \u2014 guard: fail unless <code>$DRONE_TAG</code> equals the version in <code>pyproject.toml</code>; then read <code>name</code> / <code>version</code> and query <code>$PIP_REPO_URL/simple/<package>/</code>; if that version already exists, exit <code>78</code> and skip the whole build (see 11 \u2013 Versioning & Changelog).</li> <li><code>quality-gate</code> \u2014 install the package with its <code>[dev]</code> extras from the private index, then <code>black --check</code>, <code>ruff check</code>, <code>mypy</code>, <code>pydoclint</code>, <code>pytest</code>.</li> <li><code>build-package</code> \u2014 <code>python -m build</code> into <code>dist/</code>.</li> <li><code>upload-to-private-pypi</code> \u2014 <code>twine upload</code> to <code>$PIP_REPO_URL</code>.</li> <li><code>backfill-pypi-from-tags</code> \u2014 triggered on <code>custom</code>:</li> <li><code>fetch-tags</code> \u2014 <code>git fetch --tags --force</code>.</li> <li><code>build-and-upload-missing</code> \u2014 walk every tag in version order, skip the ones already on the index, <code>git checkout</code> the rest, verify <code>tag == pyproject version</code> at that commit, build, and upload. This is the disaster-recovery / replay path.</li> </ol> <p>Secrets: <code>PIP_REPO_URL</code>, <code>PIP_USERNAME</code>, <code>PIP_PASSWORD</code>.</p> <ul> <li>Ship as-is. Package name and version come from <code>pyproject.toml</code>, so the same file is reused verbatim across every library.</li> </ul>"},{"location":"10_drone_pipelines/#type-b-docker-deployment-service-dockeryaml","title":"Type B \u2014 Docker deployment (<code>service-docker.yaml</code>)","text":"<p>For services and sites that ship a container image. The file contains two pipelines, each mirroring a repo family:</p> <ol> <li><code>build-and-deploy-service</code> \u2014 triggered on <code>tag</code> (a long-running API/worker service, e.g. <code>auth-server</code>):</li> <li><code>fetch-tags</code> \u2014 resolve the latest tag into <code>LATEST_TAG.txt</code>; fail if there are no tags.</li> <li><code>check-version</code> \u2014 guard: fail unless <code>$DRONE_TAG</code> equals the repo-root <code>VERSION</code> file (see 11 \u2013 Versioning & Changelog).</li> <li><code>check-remote-image</code> \u2014 if <code>aetos/<app>:<tag></code> already exists locally, exit <code>78</code> and skip the build.</li> <li><code>quality-gate</code> \u2014 install from <code>requirements.txt</code> and run <code>black</code> / <code>ruff</code> / <code>mypy</code> / <code>pytest</code>.</li> <li><code>build-image</code> \u2014 <code>docker build</code> with the private-index build args, tagged <code>aetos/<app>:<tag></code> and <code>:latest</code>.</li> <li><code>push-image</code> \u2014 login, re-tag with <code>$REGISTRY_HOST</code>, push both tags.</li> <li><code>stop-old</code> \u2014 <code>docker rm -f <app></code>.</li> <li><code>run-container</code> \u2014 <code>docker run -d --name <app> -p <host>:<container> --restart always</code> with the service's <code>-e</code> environment.</li> <li><code>build-and-deploy-site</code> \u2014 triggered on <code>push</code> (a static/HTTP site, e.g. <code>docs</code>, <code>mcp</code>): <code>build-image</code> \u2192 <code>push-image</code> \u2192 <code>stop-old</code> \u2192 <code>run-container</code>, with no tag resolution, no quality gate, and no version guard (it ships <code>:latest</code> only).</li> </ol> <p>Secrets: <code>PIP_REPO_URL</code>, <code>PIP_USERNAME</code>, <code>PIP_PASSWORD</code> (service build args), <code>REGISTRY_HOST</code>, <code>REGISTRY_USER</code>, <code>REGISTRY_PASS</code>, plus the service runtime env (for example <code>MONGO_*</code>, <code>JWT_SECRET</code>).</p> <p>Placeholders to fill in per repo:</p> Placeholder Meaning Example <code><app></code> service image / container name <code>auth-server</code> <code>aetos/<app></code> image repository on the registry <code>aetos/auth-server</code> <code><host-port></code> / <code><container-port></code> published : in-image port <code>9003:8000</code> <code><site></code> site image / container name <code>docs</code> <code><ENV_VAR></code> runtime environment variable <code>MONGO_HOST</code>"},{"location":"10_drone_pipelines/#what-not-to-do","title":"What not to do","text":"<ul> <li>Do not hand-roll a third shape. A repo is either Type A or Type B, never both.</li> <li>Do not inline secrets; always use <code>from_secret</code>.</li> <li>Do not drop the <code>check-version</code> / <code>check-remote-image</code> skip \u2014 it guards the index and the running container against duplicate publishes.</li> <li>Do not drop the tag\u2194version guard. A package needs its <code>pyproject.toml</code> version to equal the tag; a service needs a root <code>VERSION</code> file that equals the tag. See 11 \u2013 Versioning & Changelog.</li> <li>Do not rename the pipelines; tooling and runbooks key off <code>build-and-publish-pypi</code>, <code>build-and-deploy-service</code>, and <code>build-and-deploy-site</code>.</li> </ul>"},{"location":"10_drone_pipelines/#related","title":"Related","text":"<ul> <li>01 \u2013 Build Recipe \u00b7 09 \u2013 Infra Adapters \u00b7 11 \u2013 Versioning & Changelog \u00b7 13 \u2013 Library Registry</li> </ul>"},{"location":"11_versioning_and_changelog/","title":"Versioning & Changelog","text":""},{"location":"11_versioning_and_changelog/#summary","title":"Summary","text":"<p>Every Aetos artifact \u2014 library, service, or site \u2014 is versioned with Semantic Versioning and documents its releases in a Keep a Changelog file. The version file is the source of truth, the git tag must equal it, and a Drone guard enforces that equality before anything is published. Releases are cut manually; a tag is what triggers the pipeline.</p> <p>This page pairs with 10 \u2013 Drone Pipelines: the version and tag contract is exactly what those two CI shapes consume.</p>"},{"location":"11_versioning_and_changelog/#the-version-contract","title":"The version contract","text":"Artifact Source of truth Notes Library (PyPI package) <code>version</code> in <code>pyproject.toml</code> <code>check-version</code> also reads it to skip duplicates Service / site (Docker) <code>VERSION</code> file at the repo root one line, <code>x.y.z</code> <p>Rules:</p> <ol> <li>One version per repository. No per-module versions.</li> <li>Tag = version. The git tag is bare SemVer (<code>0.0.6</code>), no <code>v</code> prefix, and it MUST match the version file byte-for-byte. The Drone guard fails the build otherwise \u2014 this is what stops a tag from silently publishing nothing.</li> <li>A release is a tag. Bump the version file, promote the changelog, commit, tag, push. Everything else is automated by Drone.</li> <li>Never reuse a tag. If a release is wrong, cut a new patch version.</li> </ol>"},{"location":"11_versioning_and_changelog/#pre-10-0yz","title":"Pre-1.0 (0.y.z)","text":"<p>While a package is <code>0.y.z</code>, <code>0.y</code> is treated as the major line:</p> <ul> <li>breaking change \u2192 <code>0.(y+1).0</code></li> <li>new feature \u2192 <code>0.y.(z+1)</code></li> <li>bug fix \u2192 <code>0.y.(z+1)</code></li> </ul> <p>Once a package reaches <code>1.0.0</code>, normal SemVer applies. Do not ship <code>1.0.0</code> until the public API is considered stable.</p>"},{"location":"11_versioning_and_changelog/#the-changelog","title":"The changelog","text":"<p><code>CHANGELOG.md</code> at the repo root, in Keep a Changelog form. It always has exactly one open <code>[Unreleased]</code> section with the change categories that apply:</p> <pre><code>## [Unreleased]\n\n### Added\n### Changed\n### Deprecated\n### Removed\n### Fixed\n### Security\n</code></pre> <p>On release you promote it \u2014 never delete history:</p> <pre><code>## [0.0.6] - 2026-03-07\n\n### Added\n- ...\n</code></pre> <p>and add a link reference at the bottom:</p> <pre><code>[Unreleased]: https://git.aetoskia.com/aetos/<repo>/compare/0.0.6...HEAD\n[0.0.6]: https://git.aetoskia.com/aetos/<repo>/compare/0.0.5...0.0.6\n</code></pre> <p>Conventions:</p> <ul> <li>Write entries for humans: one line each, imperative, referencing the user impact, not the commit hash.</li> <li>Categories: Added, Changed, Deprecated, Removed, Fixed, Security. Omit empty categories.</li> <li>Security fixes get their own entry even if the change is otherwise a bug fix.</li> <li>The <code>[Unreleased]</code> heading is never renamed; copy its content into the new version heading and clear it.</li> </ul>"},{"location":"11_versioning_and_changelog/#release-checklist","title":"Release checklist","text":"<ol> <li>Confirm the quality gate is green on <code>main</code>.</li> <li>Move every <code>[Unreleased]</code> entry under the new version heading with today's date (<code>## [x.y.z] - YYYY-MM-DD</code>); restore an empty <code>[Unreleased]</code> on top.</li> <li>Add the two compare-link references at the bottom of the changelog.</li> <li>Set the version file: <code>pyproject.toml</code> (<code>version = \"x.y.z\"</code>) or <code>VERSION</code>.</li> <li>Commit: <code>chore(release): x.y.z</code>.</li> <li>Tag: <code>git tag x.y.z</code> and push the tag (<code>git push origin x.y.z</code>).</li> <li>Watch Drone: Type A publishes the wheel/sdist, Type B builds and deploys the image. Both first assert <code>tag == version</code>.</li> <li>Do not reuse the tag if the run fails \u2014 fix and cut <code>x.y.(z+1)</code>.</li> </ol>"},{"location":"11_versioning_and_changelog/#why-the-guard-matters","title":"Why the guard matters","text":"<p>The version file and the tag are two facts that must agree; nothing in git enforces that by itself. A mismatch is not a theoretical risk \u2014 a package can be tagged <code>0.0.2</code> while its <code>pyproject.toml</code> still says <code>0.0.1</code>, and the publish step (which compares the file against the index) then skips the build entirely, so <code>0.0.2</code> never reaches the index while looking like a release. The Drone guard turns that silent no-op into a hard failure.</p>"},{"location":"11_versioning_and_changelog/#related","title":"Related","text":"<ul> <li>10 \u2013 Drone Pipelines \u00b7 01 \u2013 Build Recipe \u00b7 13 \u2013 Library Registry</li> </ul>"},{"location":"12_genericity_matrix/","title":"Genericity Matrix","text":""},{"location":"12_genericity_matrix/#summary","title":"Summary","text":"<p>This page states precisely what is library-backed, what is template-provided, and what is app-specific and must be authored. Use it to scope a new build and to know, for any given capability, whether to call a library, copy template code, or write it yourself.</p>"},{"location":"12_genericity_matrix/#tier-a-library-backed-use-never-reimplement","title":"Tier A \u2014 Library-backed (use; never reimplement)","text":"<p>These capabilities already exist as internal libraries. Reach them through the library registry MCP (<code>find_library(capability)</code>) and read the library's own MCP for its API. Writing a private copy of any of these is a defect.</p> Capability Library MCP Persistence: repositories, cache, populate, transactions Mongo Ops (<code>mongo_ops</code>) <code>:8007</code> Dependency injection / composition Hexa (<code>hexa</code>) <code>:8008</code> Deterministic multi-step transformation engine DAG Pipe (<code>dagpipe</code>) <code>:8006</code> Spec-first HTTP edge, client, codegen OpenAPI First (<code>openapi_first</code>) <code>:8004</code> Format-agnostic fetch + parse (HTML, PDF, CSV, XLSX) Omniread (<code>omniread</code>) <code>:8003</code> Read-only email ingestion Mail Intake (<code>mail_intake</code>) <code>:8002</code> JWT auth primitives (register, login, introspect) Py JWT (<code>jwtlib</code>) <code>:8005</code> Documentation build (MkDocs + MCP) Doc Forge (<code>docforge</code>) <code>:8001</code>"},{"location":"12_genericity_matrix/#tier-b-template-provided-copy-from-templatesapp-then-adapt","title":"Tier B \u2014 Template-provided (copy from <code>templates/app</code>, then adapt)","text":"<p>These are reusable internal components shipped inside the template \u2014 they are not independent libraries. Copy them as-is, then rename/populate only what is app-specific. They give every app the same wiring shape as the reference app. Fetch them by kind with <code>get_app_component</code>, or the whole tree with <code>get_app_template</code>.</p> Component Template location Action Shipped Layering & dependency rule the package layout + import rules keep \u2705 Composition root shape <code>init.py</code> (<code>get_svc</code>/<code>get_db</code>, idempotency) keep \u2705 Spec-first web edge <code>interfaces/openapi.py</code>, thin handlers bound to <code>operationId</code> keep \u2705 CRUD slice pattern <code>domain/<resource>/{models.py, ports/db.py, ports/service.py, converters.py}</code> rename + fill \u2705 Generic CRUD service <code>infra/adapters/service.py</code> (<code>CrudsServiceAdaptor</code>) keep \u2705 Repository factory <code>infra/db/repos/mongo.py</code> (<code>_repo_factory</code>) keep \u2705 Worker poll + status loop <code>worker.py</code> keep \u2705 Progress bus <code>infra/adapters/sse.py</code> (keyed in-memory bus) + an app SSE endpoint keep \u2705 HTTP response cache <code>infra/http_cache.py</code> (<code>cache_response</code> + generation invalidation) keep \u2705 Key-value store <code>infra/adapters/kv.py</code> (port + in-memory + Redis adaptor) keep \u2705 Pipeline scaffold <code>pipelines/shared/{ports,loader,registry}.py</code> + <code>pipelines/<family>/</code> rename + fill \u2705 Operator scripts <code>scripts/</code> (<code>init()</code> / <code>get_svc()</code> / <code>shutdown()</code>) rename \u2705 Testing harness integration fakes + fixture-parity pattern keep \u2705 Project scaffolding <code>pyproject.toml</code>, <code>Dockerfile</code>, <code>docker-compose.yaml</code>, <code>.env.example</code>, <code>openapi.yaml</code> rename + fill \u2705"},{"location":"12_genericity_matrix/#tier-b-patterns-port-from-the-reference-app","title":"Tier B (patterns) \u2014 port from the reference app","text":"<p>A few Tier B capabilities are real and reusable, but are shaped by the app or the provider, so the template does not ship them verbatim \u2014 copying them blindly would carry app-specific names. Treat them as patterns: lift the shape from the reference app and rename per app.</p> Component Reference location Action Shipped Human-in-the-loop durable pause + question document + resolve endpoint + resume map port + reshape \u2717 LLM client pool provider-select + retry/rotation pool scaffold port + reshape \u2717"},{"location":"12_genericity_matrix/#tier-c-app-specific-must-be-authored","title":"Tier C \u2014 App-specific (must be authored)","text":"Artifact Why Resource models the business shape Repository populate/cache rules the relationships Service business logic domain behaviour Pipeline families the transformation graph Pattern/rule resources data-driven policy Fixtures & golden outputs test data"},{"location":"12_genericity_matrix/#the-two-axes","title":"The two axes","text":"<p>Everything app-specific reduces to two axes:</p> <ol> <li>Resource model \u2014 the set of resources, their fields, relationships, and the OpenAPI contract.</li> <li>Pipeline family \u2014 only if inputs require multi-step transformation.</li> </ol> <p>If you can specify those two axes, the rest of the app follows mechanically: Tier A is imported, Tier B is copied, and Tier C is the only thing you author.</p>"},{"location":"13_library_registry/","title":"Library Registry","text":""},{"location":"13_library_registry/#summary","title":"Summary","text":"<p>The Aetos ecosystem provides internal libraries for the shared concerns of every application. Use them. Writing a private copy of any capability below is a defect.</p> <p>Each library also exposes its own documentation as an MCP server. An AI agent can read a library's full API by connecting to its MCP and reading <code>docs://index</code>, <code>docs://nav</code>, and <code>docs://modules/{module}</code>.</p>"},{"location":"13_library_registry/#catalog","title":"Catalog","text":"Library Import Purpose Use instead of MCP DAG Pipe <code>dagpipe</code> Deterministic state-transformation DAG engine hand-rolled orchestration/topological ordering <code>:8006</code> Hexa <code>hexa</code> Declarative DI / composition-root tooling manual wiring, hand-written factories <code>:8008</code> Mongo Ops <code>mongo_ops</code> Async MongoDB repositories, cache, populate, transactions raw Motor CRUD and connection lifecycle <code>:8007</code> Omniread <code>omniread</code> Format-agnostic fetch + parse (HTML, PDF, CSV, XLSX) custom scrapers/parsers <code>:8003</code> OpenAPI First <code>openapi_first</code> Spec-first FastAPI bootstrap, client, codegen decorator routers and hand-written clients <code>:8004</code> Mail Intake <code>mail_intake</code> Provider-agnostic read-only email ingestion Gmail OAuth + MIME parsing <code>:8002</code> Py JWT <code>jwtlib</code> JWT auth primitives (register/login/introspect) password hashing + token code <code>:8005</code> Doc Forge <code>docforge</code> Docstring compiler \u2192 MkDocs + MCP docs custom doc generators <code>:8001</code> Auth Server (service) Central identity service (register/login/me/logout/introspect) building auth endpoints yourself spec: <code>docs://spec</code>"},{"location":"13_library_registry/#how-to-choose","title":"How to choose","text":"You need to\u2026 Use persist and query documents Mongo Ops compose dependencies Hexa run a multi-step transformation DAG Pipe read/parse an arbitrary input format Omniread expose a contract-first HTTP API OpenAPI First ingest email Mail Intake authenticate users / verify tokens Py JWT (+ Auth Server) generate documentation Doc Forge"},{"location":"13_library_registry/#discovery","title":"Discovery","text":"<p>A single standards MCP exposes <code>find_library(capability)</code>, which returns the matching library, its import name, its \"use instead of\" guidance, and its MCP endpoint (or, for the auth-server service, a spec pointer). It also serves the pinned auth-server OpenAPI contract at <code>docs://spec</code>. Query it before writing any infrastructure.</p>"},{"location":"13_library_registry/#notes","title":"Notes","text":"<ul> <li>Hexa is a full library with its own MCP server on <code>:8008</code>: read <code>docs://index</code>, <code>docs://nav</code>, and <code>docs://modules/{module}</code> (e.g. <code>docs://modules/hexa.container</code>) to discover the DI surface \u2014 <code>build(cls, *, instances, config)</code> and the <code>PortNode</code> model.</li> <li>Auth Server is a service, not an importable library: it has no MCP bundle by design. Read its client contract from the pinned OpenAPI document via <code>docs://spec</code> and wire it by declaring a <code>bearerAuth</code> security scheme in the app's own spec (see Auth Integration); <code>py-jwt</code> covers the low-level token path.</li> <li>Library MCP servers expose documentation resources (not executable tools): they tell an agent the available API surface so it does not invent code.</li> </ul>"},{"location":"14_auth_integration/","title":"Auth Integration","text":""},{"location":"14_auth_integration/#summary","title":"Summary","text":"<p>Authentication is a service, not a library. Every Aetos application shares a single identity service \u2014 the auth-server \u2014 and consumes it through its HTTP contract. An app never implements registration, login, password hashing, or token issuance itself.</p> <p>The agent does not need an MCP bundle for the auth-server. It reads the pinned OpenAPI contract (below) and declares a bearer security scheme in its own spec; <code>openapi_first</code> then generates the verifying dependency. No auth code is written.</p>"},{"location":"14_auth_integration/#read-the-contract","title":"Read the contract","text":"<p>The auth-server OpenAPI document is pinned in the standards bundle and served as an MCP resource:</p> <pre><code>docs://spec\n</code></pre> <p>It is the authoritative client contract (OpenAPI 3.1). Read it to learn the endpoints and the request/response schemas:</p> Endpoint Purpose <code>POST /register</code> Create a user (201, returns <code>PublicUser</code>) <code>POST /login</code> Authenticate and issue a JWT (<code>LoginResponse</code>) <code>GET /me</code> Current user's <code>PublicUser</code> (bearer required) <code>POST /logout</code> Stateless logout (<code>LogoutResponse</code>) <code>POST /introspect</code> Internal token introspection (<code>active</code>, <code>user</code>) <p>The pinned snapshot is versioned (currently <code>0.0.5</code>). If the caller has a live auth-server and needs the current contract, they can pass the full endpoint (<code>{AUTH_SERVER}/openapi.json</code>) and read it directly instead of the pin.</p>"},{"location":"14_auth_integration/#wire-it-up-declare-dont-code","title":"Wire it up: declare, don't code","text":"<p>Authentication is added by editing the app's own <code>openapi.yaml</code> \u2014 nothing else. Declare a global bearer requirement and point the scheme at the auth server (values come from the environment):</p> <pre><code>security:\n - bearerAuth: []\npaths:\n # ...\ncomponents:\n securitySchemes:\n bearerAuth:\n type: http\n scheme: bearer\n bearerFormat: JWT\n x-server-url: \"{AUTH_SERVER}\"\n x-introspect-path: \"/introspect\"\n x-login-path: \"/login\"\n x-register-path: \"/register\"\n x-logout-path: \"/logout\"\n x-me-path: \"/me\"\n</code></pre> <p><code>openapi_first</code> reads <code>securitySchemes</code> + <code>security</code> and generates a FastAPI dependency that extracts the bearer token, POSTs <code>{\"token\": <jwt>}</code> to <code>{AUTH_SERVER}/introspect</code>, and returns the user when <code>active == true</code>. It raises <code>401</code> on an invalid or inactive token and <code>503</code> when the auth server is unreachable. The generated handlers receive the authenticated user \u2014 you author no authentication code.</p>"},{"location":"14_auth_integration/#public-operations","title":"Public operations","text":"<p>A route that must stay open opts out by overriding <code>security</code> with an empty list:</p> <pre><code>paths:\n /health:\n get:\n operationId: health\n security: [] # overrides the global requirement\n responses:\n \"200\":\n description: OK\n</code></pre>"},{"location":"14_auth_integration/#environment","title":"Environment","text":"<pre><code>AUTH_SERVER=http://localhost:8000 # auth-server base URL\nJWT_SERVER=http://localhost:8000 # jwtlib client base URL (seeding/CLI)\n</code></pre>"},{"location":"14_auth_integration/#path-1-jwtlib-client-no-generator","title":"Path 1: <code>jwtlib</code> client (no generator)","text":"<p>Apps that do not use a generated dependency verify requests with the <code>py-jwt</code> client:</p> <pre><code>from jwtlib.introspection import authenticate_request\n\nauthorized = await authenticate_request(\n should_skip_authentication, # allow public paths\n method,\n path,\n authorization_token, # \"Bearer <jwt>\" or None\n)\n</code></pre> <p><code>authenticate_request</code> POSTs to <code>{JWT_SERVER}/introspect</code> (3 second timeout) and treats <code>{\"active\": true, \"user\": {...}}</code> as valid. Use this for seeding scripts and tests (<code>jwtlib.register_user</code>, <code>jwtlib.login_user</code>).</p>"},{"location":"14_auth_integration/#golden-rules","title":"Golden rules","text":"<ol> <li>Never trust an unverified token. Decode through the generated dependency or <code>jwtlib</code> before handling the request.</li> <li>Treat <code>active: false</code> as unauthenticated, even if the JWT decodes.</li> <li>Do not implement your own JWT parsing. Revocation and issuer changes must stay centralized in the auth-server.</li> <li>Do not add auth until the app's outcome is validated. Ship the app authless first; layer the security scheme once the core behaviour works.</li> </ol>"},{"location":"14_auth_integration/#notes","title":"Notes","text":"<ul> <li>The default template is authless on purpose. Add the <code>securitySchemes</code> block above only when auth is required. Pull the shared slice with <code>get_app_component(\"auth\")</code>.</li> <li>The auth-server is deployed as a service (no importable package); the client surface is the spec, plus <code>jwtlib</code> for the low-level path.</li> </ul>"}]} |