1 line
41 KiB
JSON
1 line
41 KiB
JSON
{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"The Aetos Standard App Blueprint","text":"<p>A single, reusable architecture for backend applications in the Aetos ecosystem.</p> <p>Every application built with this blueprint has the same shape:</p> <ul> <li>a spec-first HTTP layer driven by an OpenAPI document;</li> <li>a domain layer of pure models, ports, and services;</li> <li>an infrastructure layer of adapters (MongoDB, cache, workers, LLM, mail);</li> <li>a composition root that wires everything in exactly one place;</li> <li>an optional pipeline layer for data-driven extraction/enrichment;</li> <li>an optional background worker that runs long-running jobs and reports progress.</li> </ul> <p>This blueprints is deliberately app-agnostic. It never names a concrete business domain. Any application \u2014 expenses, health metrics, inventory, analytics \u2014 is produced by filling in two axes only: the resource model and (if needed) the pipeline family.</p>"},{"location":"#how-to-read-this","title":"How to read this","text":"Page What it covers Overview The philosophy and the one-way dependency rule Layering Rules Which package may import which Composition Root The single wiring point Spec-First Routing OpenAPI as the source of truth CRUD Template The canonical resource slice Pipeline Scaffold Data-driven processing families Worker & Progress Async jobs, status machine, streaming progress Human-in-the-Loop Pausing a job to ask a question Infra Adapters MongoDB, cache, stores, LLM, mail Testing Integration + unit conventions Build Recipe The ordered checklist to generate a new app Genericity Matrix What is reusable vs what you must write"},{"location":"#for-ai-agents","title":"For AI agents","text":"<p>This content is served as MCP resources under <code>docs://index</code>, <code>docs://nav</code>, and <code>docs://modules/{module}</code>. To build a new app:</p> <ol> <li>Read <code>build_recipe</code> first.</li> <li>Read <code>genericity_matrix</code> to know what you must author.</li> <li>Query the library registry (<code>find_library</code>) before writing any infrastructure yourself.</li> <li>Copy the <code>templates/</code> skeleton and fill the two app-specific axes.</li> </ol>"},{"location":"01_overview/","title":"Overview","text":""},{"location":"01_overview/#summary","title":"Summary","text":"<p>The blueprint is a hexagonal (ports and adapters) application with a spec-first web edge and a single composition root. It favors explicit wiring over implicit magic, and it prefers existing internal libraries over hand-written infrastructure.</p>"},{"location":"01_overview/#the-three-layers","title":"The three layers","text":"<pre><code>interfaces/ \u2190 HTTP edge: thin handlers bound to an OpenAPI operationId\ndomain/ \u2190 business core: models, ports (interfaces), services, orchestrators\ninfra/ \u2190 adapters: databases, caches, external clients, hardware\n</code></pre> <p>Plus two optional layers:</p> <pre><code>pipelines/ \u2190 data-driven transformation graphs (extraction, enrichment)\nscripts/ \u2190 operator scripts that reuse the composition root\n</code></pre>"},{"location":"01_overview/#core-principles","title":"Core principles","text":"<ol> <li>One-way dependencies. <code>interfaces -> domain -> infra</code> is allowed; the reverse is forbidden. The domain never imports a framework it does not own.</li> <li>Ports in the domain, adapters in infra. A service depends on an abstract port; the composition root injects a concrete adapter.</li> <li>One composition root. Nothing constructs its own dependencies. All wiring lives in one module (<code>init.py</code>) and is exposed as a flat resource registry.</li> <li>Spec-first. The HTTP contract is declared in an OpenAPI document. Route handlers are looked up by <code>operationId</code>. There are no decorator routers.</li> <li>Reuse the libraries. Pipeline orchestration, DI, Mongo repositories, ingestion, auth, and docs are provided by internal libraries. Do not reimplement them \u2014 see the library registry.</li> <li>Explicit state machines. Long-running work moves through explicit, durable statuses and can pause for human input.</li> </ol>"},{"location":"01_overview/#runtime-processes","title":"Runtime processes","text":"<p>A standard app may run as up to three processes that share the same composition root:</p> Process Entry point Responsibility API <code>main.py</code> serve HTTP; start/stop resources via lifespan Worker <code>worker.py</code> poll durable jobs, run pipelines, publish progress Scripts <code>python -m scripts.<name></code> one-shot operator tasks"},{"location":"02_layering_rules/","title":"Layering Rules","text":""},{"location":"02_layering_rules/#summary","title":"Summary","text":"<p>The dependency direction is fixed. Violating it is a design defect.</p>"},{"location":"02_layering_rules/#allowed-imports","title":"Allowed imports","text":"<pre><code>interfaces/* -> domain/* (services, ports, models)\ninterfaces/* -> infra/adapters/* (only read-only value objects)\ndomain/* -> domain/* (same layer)\ndomain/* -> pipelines/* (call a pipeline via a port)\ninfra/adapters/* -> domain/ports/* (implement the interface)\nmain.py/init.py -> everything (composition root only)\n</code></pre>"},{"location":"02_layering_rules/#forbidden-imports","title":"Forbidden imports","text":"<pre><code>domain/* -> interfaces/* (never)\ndomain/* -> fastapi/starlette (never)\ninfra/adapters/* -> interfaces/* (never)\ninterfaces/* -> infra/db/* (state lives in resources, not imports)\n</code></pre>"},{"location":"02_layering_rules/#why","title":"Why","text":"<ul> <li>A pure domain is testable without a server, a database, or an event loop.</li> <li>Adapters can be swapped (in-memory fake, real database) without touching business logic.</li> <li>The HTTP edge stays thin and replaceable.</li> </ul>"},{"location":"02_layering_rules/#the-portadapter-contract","title":"The port/adapter contract","text":"<p>A port is an abstract class or protocol declared in <code>domain/<x>/ports/</code>. An adapter implements it in <code>infra/adapters/</code>. The composition root binds the two.</p> <pre><code>domain/<resource>/ports/db.py -> repository port + document model\ndomain/<resource>/ports/service.py -> service + factory\ninfra/db/repos/<vendor>.py -> repository adapter factory\ninfra/adapters/service.py -> generic CRUD service base\n</code></pre>"},{"location":"02_layering_rules/#enforcement","title":"Enforcement","text":"<ul> <li>Keep every domain module importable with only the standard library and Pydantic. If a domain import pulls in <code>fastapi</code>, the layering is broken.</li> <li>Reviewers (and CI) reject <code>import fastapi</code> inside <code>domain/</code> or <code>infra/</code>.</li> </ul>"},{"location":"03_composition_root/","title":"Composition Root","text":""},{"location":"03_composition_root/#summary","title":"Summary","text":"<p>All wiring happens in exactly one module. It builds a flat resource registry and exposes it to the HTTP layer and to scripts.</p>"},{"location":"03_composition_root/#the-resource-registry","title":"The resource registry","text":"<p><code>init.py</code> returns a plain <code>dict[str, Any]</code>. Keys follow two conventions:</p> <ul> <li><code><resource>_repo</code> \u2014 a repository adapter for a resource.</li> <li><code><resource>_svc</code> \u2014 a service for a resource.</li> </ul> <p>Plus named infrastructure adaptors:</p> <ul> <li><code>http_cache_store</code> \u2014 response cache backend.</li> <li><code>progress_bus</code> \u2014 progress/event bus.</li> <li><code><vendor>_<kind></code> \u2014 fetchers, clients, stores.</li> </ul> <p>The API stores the dict on <code>app.state.resources</code> during lifespan startup. Handlers read from it; nothing imports concrete adapters.</p>"},{"location":"03_composition_root/#shape","title":"Shape","text":"<pre><code>_resources: dict = {}\n_initialized = False\n\nasync def init(settings=None) -> dict:\n global _initialized\n if _initialized:\n return _resources\n\n await setup_database(uri=settings.mongo_uri, db_name=settings.mongo_db)\n\n # 1. leaf repositories\n user_repo = create_user_repo()\n\n # 2. dependent repositories (dependencies passed by keyword)\n wallet_repo = create_wallet_repo(user_repo=user_repo)\n\n # 3. services (build via the DI library, injecting the repo)\n _resources[\"user_repo\"] = user_repo\n _resources[\"wallet_repo\"] = wallet_repo\n _resources[\"user_svc\"] = get_user_service(user_repo)\n _resources[\"wallet_svc\"] = get_wallet_service(wallet_repo)\n\n # 4. infrastructure adaptors (optional)\n _resources[\"http_cache_store\"] = HttpCacheStore(...)\n _resources[\"progress_bus\"] = ProgressEventBus()\n\n _initialized = True\n return _resources\n\nasync def shutdown() -> None:\n global _initialized\n await close_stores(_resources)\n await shutdown_database()\n _resources.clear()\n _initialized = False\n</code></pre>"},{"location":"03_composition_root/#typed-accessors","title":"Typed accessors","text":"<p>Expose a <code>get_svc(name)</code> helper with one <code>@overload</code> per key so scripts get typing without importing concrete classes:</p> <pre><code>@overload\ndef get_svc(name: Literal[\"user_svc\"]) -> UserService: ...\n@overload\ndef get_svc(name: Literal[\"wallet_svc\"]) -> WalletService: ...\n\ndef get_svc(name: str):\n return _resources[name]\n</code></pre> <p>Also expose <code>get_db()</code> (raw database handle) and pure helpers such as <code>compute_fingerprint(payload)</code> (canonical hash used for deduplication).</p>"},{"location":"03_composition_root/#rules","title":"Rules","text":"<ul> <li><code>init()</code> is idempotent.</li> <li>The composition root is the only place that imports concrete adapters.</li> <li>Handlers never call <code>init()</code>; they read <code>request.app.state.resources</code>.</li> <li>Scripts call <code>init()</code> / <code>shutdown()</code> around their work.</li> </ul>"},{"location":"04_spec_first_routing/","title":"Spec-First Routing","text":""},{"location":"04_spec_first_routing/#summary","title":"Summary","text":"<p>The OpenAPI document is the single source of truth for the HTTP contract. Handlers are bound to it by name; there is no decorator router.</p>"},{"location":"04_spec_first_routing/#the-contract","title":"The contract","text":"<ul> <li><code>openapi.yaml</code> (or <code>.json</code>) declares paths, schemas, parameters, responses, and security.</li> <li>A custom <code>operationId</code> names the handler function.</li> <li>The framework (<code>openapi_first</code>) validates requests/responses against the spec and dispatches to the handler whose name equals <code>operationId</code>.</li> </ul>"},{"location":"04_spec_first_routing/#app-bootstrap","title":"App bootstrap","text":"<pre><code># main.py\nfrom openapi_first.app import OpenAPIFirstApp\n\napp = OpenAPIFirstApp(\n openapi_path=\"openapi.yaml\",\n routes_module=interfaces.openapi, # module whose namespace holds handlers\n title=\"<App> API\",\n version=\"0.1.0\",\n lifespan=lifespan,\n)\n</code></pre> <p><code>lifespan</code> calls the composition root:</p> <pre><code>@asynccontextmanager\nasync def lifespan(app):\n app.state.resources = await init()\n yield\n await shutdown()\n</code></pre>"},{"location":"04_spec_first_routing/#handler-registry","title":"Handler registry","text":"<p><code>interfaces/openapi.py</code> is a flat re-export of every handler, so the framework can resolve <code>operationId</code> across the whole app:</p> <pre><code>from interfaces.item.routes import *\nfrom interfaces.wallet.routes import *\nfrom interfaces.progress.sse import *\n</code></pre>"},{"location":"04_spec_first_routing/#handler-shape","title":"Handler shape","text":"<p>Handlers are thin and synchronous with respect to business logic:</p> <pre><code>async def create_item(request: Request, body: dict):\n svc = request.app.state.resources[\"item_svc\"]\n return await svc.create(Item.model_validate(body))\n</code></pre>"},{"location":"04_spec_first_routing/#conventions","title":"Conventions","text":"<ul> <li><code>operationId</code> naming: <code>list_<resource></code>, <code>create_<resource></code>, <code>get_<resource></code>, <code>update_<resource></code>, <code>delete_<resource></code>.</li> <li>Read-only endpoints may use a cache decorator keyed on path + query + user.</li> <li>Validation errors -> <code>400</code>; not found -> <code>404</code>; delete -> <code>204</code>.</li> <li>Security is declared once in the spec and enforced by the framework; handlers contain no auth code.</li> </ul>"},{"location":"04_spec_first_routing/#codegen","title":"Codegen","text":"<p>The <code>openapi_first</code> library can scaffold models and route stubs from the spec, and ships app templates (CRUD app, model app, service). Prefer codegen over hand-writing boilerplate.</p>"},{"location":"05_crud_template/","title":"CRUD Template","text":""},{"location":"05_crud_template/#summary","title":"Summary","text":"<p>Every resource follows the same four-file slice. Repetition is intentional: it makes the codebase predictable and generatable.</p>"},{"location":"05_crud_template/#the-slice","title":"The slice","text":"<pre><code>domain/<resource>/\n models.py # pure Pydantic model (no DB fields)\n ports/\n db.py # document model + repository factory\n service.py # service class + factory\n converters.py # (optional) transforms\n</code></pre>"},{"location":"05_crud_template/#modelspy","title":"models.py","text":"<p>A framework-free Pydantic model describing the business shape.</p> <pre><code>class Item(BaseModel):\n name: str\n quantity: int = 0\n</code></pre>"},{"location":"05_crud_template/#portsdbpy","title":"ports/db.py","text":"<p>The persistence model extends the business model and adds database concerns. The repository factory selects the right repository flavour.</p> <pre><code>class ItemDocument(Item, BaseDocument): # BaseDocument adds id/created_at/...\n pass\n\ndef create_item_repo(*, owner_repo=None) -> ItemRepository:\n return repo_factory(\n collection=\"items\",\n document=ItemDocument,\n populate_rules=[PopulateRule(field=\"owner\", collection=\"users\")],\n cache=None, # or a CachedBaseRepository config\n )\n</code></pre>"},{"location":"05_crud_template/#portsservicepy","title":"ports/service.py","text":"<p>The service is built through the DI library, injecting the repository. The generic CRUD base provides <code>create/read/update/patch/delete/search/...</code>.</p> <pre><code>class ItemService(CrudsServiceAdaptor[Item, ItemDocument]):\n model = Item\n schema = ItemDocument\n\ndef get_item_service(repo) -> ItemService:\n return build(ItemService, instances={\"repo\": repo})\n</code></pre>"},{"location":"05_crud_template/#naming-conventions","title":"Naming conventions","text":"Concept Convention collection plural snake case (<code>items</code>) document <code><Resource>Document</code> repository factory <code>create_<resource>_repo</code> service <code><Resource>Service</code> service factory <code>get_<resource>_service</code> resource key <code><resource>_svc</code> operationId <code>list_items</code>, <code>create_item</code>, ..."},{"location":"05_crud_template/#optional-files","title":"Optional files","text":"<ul> <li><code>converters.py</code> \u2014 pure functions mapping between domain types.</li> <li><code>ports/enrichment.py</code> \u2014 adapts a pipeline into an async callable the service can invoke (used when a resource is populated by a pipeline).</li> </ul>"},{"location":"06_pipeline_scaffold/","title":"Pipeline Scaffold","text":""},{"location":"06_pipeline_scaffold/#summary","title":"Summary","text":"<p>Data-driven processing is expressed as a composition graph. The graph is built by a DI library; execution is driven by a DAG engine. Stages are pure and individually testable.</p>"},{"location":"06_pipeline_scaffold/#when-to-use-a-pipeline","title":"When to use a pipeline","text":"<p>Use a pipeline when a resource is produced by a multi-step transformation of an input (a document, a file, a message) rather than by a simple CRUD write.</p>"},{"location":"06_pipeline_scaffold/#structure-of-a-family","title":"Structure of a family","text":"<p>A family is one input shape (e.g. a file format or a message kind). Each family has the same skeleton:</p> <pre><code>pipelines/<family>/\n abc.py # port contracts (stage interfaces + root interface)\n impl.py # concrete stages + root implementation\n pipeline.yaml # graph spec: nodes, edges, roots, schema\n resources/\n pipeline.yaml # runtime spec consumed by the loader\n patterns/*.yaml # rule fragments (pattern registry)\n banks/<variant>.py # optional variants that re-pin a few slots\n</code></pre>"},{"location":"06_pipeline_scaffold/#shared-scaffolding","title":"Shared scaffolding","text":"<pre><code>pipelines/shared/\n ports.py # RegistryPort, FinalizePort, etc.\n loader.py # build(root_cls) -> wires the graph + engine\n registry.py # resolve_root(...) -> root class for a (kind, variant)\n</code></pre>"},{"location":"06_pipeline_scaffold/#the-loader","title":"The loader","text":"<pre><code>def build(root_cls, *, instances=None, config=None):\n # merge resources/patterns/*.yaml into a pattern registry\n # bind registry/config into each graph node by id\n # build the schema (state) class and the engine\n root = root_cls()\n root._engine = Engine(graph, state_cls, payload_cls)\n return root\n</code></pre>"},{"location":"06_pipeline_scaffold/#the-registry","title":"The registry","text":"<p>A single lookup replaces hand-rolled <code>if/else</code> selection:</p> <pre><code>_FAMILIES = {\n \"csv\": {\"acme\": {\"heuristic\": AcmeCsvPipeline}},\n \"json\": {\"globex\": {\"heuristic\": GlobexJsonPipeline}},\n}\n\ndef resolve_root(kind, variant, mode=\"heuristic\"):\n return _FAMILIES[kind][variant][mode]\n</code></pre> <p>Adding a new variant means adding one class and one registry entry.</p>"},{"location":"06_pipeline_scaffold/#driving-a-pipeline","title":"Driving a pipeline","text":"<p>Stages are driven explicitly so intermediate state can be persisted:</p> <pre><code>pipeline = build(resolve_root(kind, variant))\ncontent = pipeline.ingest.materialize(raw_bytes)\nrecords = pipeline.reader.read(content)\nitems = pipeline.normalize.build(records)\n</code></pre>"},{"location":"06_pipeline_scaffold/#testing","title":"Testing","text":"<p>Because stages are pure, test each stage in isolation and test the assembled root against fixture inputs whose expected outputs are stored as golden files. A parity test replays fixtures headlessly.</p>"},{"location":"07_worker_and_progress/","title":"Worker & Progress","text":""},{"location":"07_worker_and_progress/#summary","title":"Summary","text":"<p>Long-running work runs in a background worker. Jobs are durable documents that move through an explicit status machine. Progress is streamed to clients.</p>"},{"location":"07_worker_and_progress/#the-status-machine","title":"The status machine","text":"<pre><code>pending -> processing -> completed\n \\-> failed (retried up to N times)\n \\-> paused (waiting for human input)\n</code></pre> <ul> <li><code>pending</code> \u2014 queued, eligible for pickup.</li> <li><code>processing</code> \u2014 claimed by a worker.</li> <li><code>paused</code> \u2014 blocked on a human-in-the-loop question.</li> <li><code>failed</code> \u2014 error; <code>retry_count</code> increments until a max, then skipped.</li> <li><code>completed</code> \u2014 terminal success.</li> </ul>"},{"location":"07_worker_and_progress/#the-poll-loop","title":"The poll loop","text":"<pre><code>async def poll(resources):\n backoff = MIN_POLL\n while True:\n jobs = await job_svc.search(filter={\"status\": \"pending\"}, limit=100)\n backoff = MIN_POLL if jobs else min(backoff * 2, MAX_POLL)\n for job in jobs:\n if (job.retry_count or 0) >= MAX_RETRIES:\n continue\n await job_svc.update_status(job.id, \"processing\")\n try:\n await process(job, resources)\n except PauseForResolution:\n await job_svc.update_status(job.id, \"paused\")\n except Exception as e:\n await job_svc.update_status(\n job.id, \"failed\", error=str(e),\n retry_count=(job.retry_count or 0) + 1,\n )\n await asyncio.sleep(backoff)\n</code></pre> <p>Tunables live in environment variables: <code>WORKER_POLL_MIN</code>, <code>WORKER_POLL_MAX</code>, <code>WORKER_MAX_RETRIES</code>.</p>"},{"location":"07_worker_and_progress/#progress-events","title":"Progress events","text":"<p>A worker publishes structured progress events to a bus (a port). The HTTP edge exposes a streaming endpoint that replays events for a given job id.</p> <pre><code>class ProgressEvent(BaseModel):\n job_id: str\n step: str\n status: str # started | progress | completed | skipped | paused\n message: dict | None = None\n</code></pre> <p>The bus is an in-memory adapter by default; because it is a port, it can be swapped for a durable broker. Emit from the orchestrator at each stage:</p> <pre><code>await self.bus.publish(ProgressEvent(job_id=jid, step=\"normalize\",\n status=\"completed\", message={\"count\": n}))\n</code></pre>"},{"location":"07_worker_and_progress/#streaming-endpoint","title":"Streaming endpoint","text":"<pre><code>async def stream_job_events(request, id):\n bus = request.app.state.resources.get(\"progress_bus\")\n if bus is None:\n return {\"events\": []}\n\n async def gen():\n sent = 0\n while True:\n events = bus.get_events(id)\n for e in events[sent:]:\n yield f\"data: {e.model_dump_json(exclude_none=True)}\\n\\n\"\n sent += 1\n await asyncio.sleep(0.5)\n\n return StreamingResponse(gen(), media_type=\"text/event-stream\")\n</code></pre> <p>Known limitation. The default in-memory bus is per-process. If the worker and the API run as separate processes, events published by the worker are not visible to the API. For production, back the bus with a shared broker. The port boundary makes this a one-adapter change.</p>"},{"location":"08_human_in_loop/","title":"Human-in-the-Loop","text":""},{"location":"08_human_in_loop/#summary","title":"Summary","text":"<p>Some inputs are ambiguous and must be resolved by a human before processing can continue. The blueprint models this as a durable pause: an exception is raised, a question is recorded, the job pauses, and later resumes.</p>"},{"location":"08_human_in_loop/#the-mechanism","title":"The mechanism","text":"<ol> <li>During processing, a stage raises a domain exception carrying the context needed to ask the question:</li> </ol> <pre><code>raise PauseForResolution(\n job_id=jid,\n reference=\"<stable key for this decision>\",\n options=[...],\n context={...},\n)\n</code></pre> <ol> <li>The orchestrator catches it, records a question document, and re-raises so the worker marks the job <code>paused</code>.</li> <li>A client lists pending questions and answers one:</li> </ol> <pre><code>GET /jobs/{id}/questions -> list pending questions\nPOST /questions/{qid}/resolve {choice} -> record the choice\n</code></pre> <ol> <li>When the last pending question is resolved, the resolve endpoint flips the parent job back to <code>pending</code>.</li> <li>The worker picks it up again, rebuilds a resolution map from previously answered questions, and seeds it into the pipeline so the same decision is not asked twice.</li> </ol>"},{"location":"08_human_in_loop/#the-question-document","title":"The question document","text":"<pre><code>class Question(BaseDocument):\n job_id: str\n reference: str # stable key: stage|field|value\n status: str # pending | resolved\n options: list[dict]\n context: dict\n chosen: dict | None = None\n resolved_at: str | None = None\n</code></pre>"},{"location":"08_human_in_loop/#resume-map","title":"Resume map","text":"<p>The worker turns answered questions into a lookup keyed by <code>reference</code>, then injects it into the pipeline so that stage can skip re-raising:</p> <pre><code>resolved = {q.reference: q.chosen for q in await svc.search(\n filter={\"job_id\": jid, \"status\": \"resolved\"})}\n\npipeline = build(root, instances={\"pause\": handler}, config={\"resolved\": resolved})\n</code></pre>"},{"location":"08_human_in_loop/#resolve-endpoint","title":"Resolve endpoint","text":"<pre><code>async def resolve_question(request, id, body):\n svc = request.app.state.resources[\"question_svc\"]\n q = await svc.read(id)\n await svc.update(id, {\"chosen\": body[\"chosen\"], \"status\": \"resolved\",\n \"resolved_at\": now()})\n\n pending = await svc.search(filter={\"job_id\": q.job_id, \"status\": \"pending\"},\n limit=1)\n if not pending:\n jobs = request.app.state.resources[\"job_svc\"]\n job = await jobs.read(q.job_id)\n if job and job.status == \"paused\":\n await jobs.update_status(q.job_id, \"pending\")\n return q\n</code></pre>"},{"location":"08_human_in_loop/#cli-parity","title":"CLI parity","text":"<p>The same flow is available through scripts/CLI: <code>list</code>, <code>get</code>, <code>create</code>, <code>questions</code>, <code>resolve</code>, <code>retry</code>.</p>"},{"location":"09_infra_adaptors/","title":"Infra Adapters","text":""},{"location":"09_infra_adaptors/#summary","title":"Summary","text":"<p>All external systems are reached through adapters in <code>infra/</code>. Prefer an existing internal library over writing a new adapter.</p>"},{"location":"09_infra_adaptors/#database-mongodb","title":"Database (MongoDB)","text":"<ul> <li>A single connection manager owns the client lifecycle. <code>setup_database(uri, db_name)</code> / <code>shutdown_database()</code>.</li> <li>A repository factory selects the repository flavour:</li> </ul> <pre><code>repo_factory(collection, document, engine=None,\n populate_rules=None, cache=None)\n</code></pre> <ul> <li>base repository \u2014 plain CRUD;</li> <li>populating repository \u2014 resolves referenced documents (foreign keys);</li> <li> <p>cached repository \u2014 an LRU/TTL cache in front of reads.</p> </li> <li> <p>Referenced documents are described declaratively:</p> </li> </ul> <pre><code>PopulateRule(field=\"owner\", collection=\"users\",\n nested_rules=[], max_depth=2, projection=None)\n</code></pre> <ul> <li>Transactions use a session context manager provided by the library.</li> </ul> <p>Library: the Mongo operations library (see registry). Do not write raw Motor calls in the domain.</p>"},{"location":"09_infra_adaptors/#http-response-cache","title":"HTTP response cache","text":"<p>A generic GET cache sits behind a decorator.</p> <pre><code>@cache_response(ttl=60)\nasync def list_items(request): ...\n</code></pre> <ul> <li>Keyed on path + query + user.</li> <li>No-op when no store is wired.</li> <li><code>invalidate(*resources)</code> bumps a generation counter for write-triggered invalidation.</li> </ul>"},{"location":"09_infra_adaptors/#event-progress-bus","title":"Event / progress bus","text":"<p>A port with an in-memory default adapter (<code>publish</code>, <code>get_events</code>). Swap for a durable broker in production (see Worker & Progress).</p>"},{"location":"09_infra_adaptors/#external-clients-ingestion","title":"External clients (ingestion)","text":"<p>External fetchers (files, mail, HTTP) are adapters that wrap ingestion libraries. Construct them in the composition root and inject into orchestrators.</p> <p>Libraries: the ingestion library (formats) and the mail library (email).</p>"},{"location":"09_infra_adaptors/#llm-clients","title":"LLM clients","text":"<p>When a pipeline calls a model provider, route through a pool adaptor that handles provider selection, retries, and client lifecycle:</p> <pre><code>pool = setup_async_pool(provider=settings.llm_provider)\nresult = await pool.complete(model=\"...\", messages=[...])\n</code></pre> <p>Providers are selected by <code>LLM_PROVIDER</code> and configured by environment.</p>"},{"location":"09_infra_adaptors/#key-value-store","title":"Key-value store","text":"<p>For report/snapshot storage, provide a pluggable store port with a Redis adapter and an in-memory fallback.</p>"},{"location":"09_infra_adaptors/#rule","title":"Rule","text":"<p>If a capability already exists as a library (DB, DI, ingestion, mail, auth, docs), use it. Writing a private copy of a library is a defect.</p>"},{"location":"10_testing/","title":"Testing","text":""},{"location":"10_testing/#summary","title":"Summary","text":"<p>Two test suites: integration (the HTTP edge with fakes) and unit (pure domain and pipeline logic). Both run under pytest with asyncio auto mode.</p>"},{"location":"10_testing/#layout","title":"Layout","text":"<pre><code>tests/\n integration/ # TestClient(app) + mocked resources\n unit/ # models, converters, pipeline stages/roots\n conftest.py # fixtures\n</code></pre>"},{"location":"10_testing/#integration-tests","title":"Integration tests","text":"<ul> <li>Build the app with <code>TestClient</code>.</li> <li>Replace <code>app.state.resources</code> with fake repositories/services so no database is required.</li> <li>Patch the auth introspection call (<code>httpx.AsyncClient.post</code>) to return a canned identity.</li> <li>Assert status codes and bodies for list/get/create/update/delete, plus <code>400</code>, <code>404</code>, <code>204</code>, and the streaming endpoint's fallback.</li> </ul> <pre><code>client = TestClient(app)\napp.state.resources = {\"item_svc\": FakeItemService(), ...}\n\ndef test_create_item():\n r = client.post(\"/items\", json={\"name\": \"x\"})\n assert r.status_code == 200\n</code></pre>"},{"location":"10_testing/#unit-tests","title":"Unit tests","text":"<ul> <li>Domain models and converters: pure, no I/O.</li> <li>Pipeline stages: feed a fixture state, assert the transformed state.</li> <li>Assembled pipeline roots: fixture parity \u2014 replay a stored fixture input and compare against a golden output file.</li> </ul>"},{"location":"10_testing/#config-consistency-tests","title":"Config-consistency tests","text":"<p>For rule-driven pipelines, add a test that cross-references rules against the declared pattern registry: every flag emitted must be defined; every defined flag must be reachable. This catches drift without running the pipeline.</p>"},{"location":"10_testing/#conventions","title":"Conventions","text":"<ul> <li><code>asyncio_mode = \"auto\"</code>; no manual event loops.</li> <li>Fixtures are checked into the repo (inputs + golden outputs).</li> <li>Tests skip (not fail) when a required fixture is absent.</li> <li>Do not mock the domain; mock only the infrastructure edge.</li> </ul>"},{"location":"11_build_recipe/","title":"Build Recipe","text":""},{"location":"11_build_recipe/#summary","title":"Summary","text":"<p>Follow this ordered checklist to generate a new application of the standard shape. Every step is deterministic; only steps 2 and 7 require app-specific authoring.</p>"},{"location":"11_build_recipe/#0-query-the-registry","title":"0. Query the registry","text":"<p>Before writing anything, read the library registry and note which internal libraries provide DB, DI, routing, ingestion, auth, and docs. Do not reimplement them.</p>"},{"location":"11_build_recipe/#1-scaffold-the-repository","title":"1. Scaffold the repository","text":"<p>Copy the <code>templates/</code> skeleton. It provides: <code>pyproject.toml</code>, <code>openapi.yaml</code>, <code>main.py</code>, <code>init.py</code>, <code>worker.py</code>, <code>config.py</code>, <code>Dockerfile</code>, <code>docker-compose.yaml</code>, <code>.env.example</code>, and the package tree (<code>domain/ infra/ interfaces/ pipelines/ tests/ scripts/</code>).</p>"},{"location":"11_build_recipe/#2-author-the-spec-app-specific","title":"2. Author the spec (app-specific)","text":"<p>Write <code>openapi.yaml</code>: resources, schemas, <code>operationId</code>s, security, and any extension metadata. This is the contract.</p>"},{"location":"11_build_recipe/#3-write-each-crud-slice-app-specific","title":"3. Write each CRUD slice (app-specific)","text":"<p>For each resource, create the four-file slice (see CRUD Template): <code>models.py</code>, <code>ports/db.py</code>, <code>ports/service.py</code>, optional <code>converters.py</code>.</p>"},{"location":"11_build_recipe/#4-wire-the-composition-root","title":"4. Wire the composition root","text":"<p>In <code>init.py</code>: build leaf repositories, then dependent repositories, then services; add infrastructure adaptors (cache store, progress bus, stores, fetchers). Add <code>get_svc</code> overloads.</p>"},{"location":"11_build_recipe/#5-add-routes","title":"5. Add routes","text":"<p>Create <code>interfaces/<resource>/routes.py</code> (thin handlers) and re-export them in <code>interfaces/openapi.py</code>. Add a cache decorator to read endpoints.</p>"},{"location":"11_build_recipe/#6-bootstrap-the-app-and-worker","title":"6. Bootstrap the app and worker","text":"<p>Confirm <code>main.py</code> wires the OpenAPI-first app with the lifespan. If the app has long-running jobs, adapt <code>worker.py</code> to poll the relevant job collection.</p>"},{"location":"11_build_recipe/#7-add-a-pipeline-family-only-if-data-driven","title":"7. Add a pipeline family (only if data-driven)","text":"<p>Create <code>pipelines/<family>/</code> with <code>abc.py</code>, <code>impl.py</code>, <code>pipeline.yaml</code>, <code>resources/</code>, and register the root in <code>registry.py</code>. Add fixtures and a parity test.</p>"},{"location":"11_build_recipe/#8-add-infrastructure-as-needed","title":"8. Add infrastructure as needed","text":"<p>Only where the standard adaptors do not cover the need. Prefer library-backed adapters.</p>"},{"location":"11_build_recipe/#9-tests","title":"9. Tests","text":"<p>Add integration tests with fakes and unit tests for domain/pipeline logic; add a config-consistency test if using rule-driven pipelines.</p>"},{"location":"11_build_recipe/#10-scripts-and-seeds","title":"10. Scripts and seeds","text":"<p>Add operator scripts under <code>scripts/</code> that use <code>init()</code> / <code>get_svc()</code> / <code>shutdown()</code>.</p>"},{"location":"11_build_recipe/#11-docs","title":"11. Docs","text":"<p>Author GSDFC docstrings, generate the type stubs (<code>.pyi</code>), mark the package <code>py.typed</code>, and build docs.</p>"},{"location":"11_build_recipe/#definition-of-done","title":"Definition of done","text":"<ul> <li><code>python -c \"import main\"</code> succeeds.</li> <li><code>pytest</code> passes.</li> <li>The app serves the spec's operations; the worker drains its queue.</li> <li>No domain module imports a web framework.</li> <li>No infrastructure was reimplemented that a library already provides.</li> </ul>"},{"location":"12_genericity_matrix/","title":"Genericity Matrix","text":""},{"location":"12_genericity_matrix/#summary","title":"Summary","text":"<p>This page states precisely what is reusable as-is, what is a template to adapt, and what is app-specific and must be authored. Use it to scope a new build.</p>"},{"location":"12_genericity_matrix/#reusable-as-is-do-not-rewrite","title":"Reusable as-is (do not rewrite)","text":"Area Artifact Layering & dependency rule the package layout and import rules Composition root the <code>init.py</code> shape, idempotency, <code>get_svc</code> overloading Web edge <code>OpenAPIFirstApp</code> bootstrap, handler-name = <code>operationId</code>, thin handlers CRUD slice the four-file pattern (model / ports-db / ports-service / converters) Generic CRUD service the base service adaptor Repository factory base / populating / cached selection Worker the poll loop, backoff, retry, status machine Progress bus the bus port + in-memory adapter + streaming endpoint Human-in-the-loop the durable pause + question document + resolve endpoint + resume map Infra adaptors HTTP cache decorator, KV store port, LLM pool scaffold Testing integration harness with fakes, fixture-parity pattern Docs GSDFC + generated stubs + MCP bundle pipeline"},{"location":"12_genericity_matrix/#template-to-adapt-copy-then-change-names","title":"Template to adapt (copy, then change names)","text":"Artifact Change <code>pyproject.toml</code> project name, dependencies <code>Dockerfile</code> / <code>docker-compose.yaml</code> service name, ports, services <code>.env.example</code> connection strings, provider keys <code>openapi.yaml</code> every path/schema/operationId <code>scripts/</code> seed and operator tasks"},{"location":"12_genericity_matrix/#app-specific-must-be-authored","title":"App-specific (must be authored)","text":"Artifact Why Resource models the business shape Repository populate/cache rules the relationships Service business logic domain behaviour Pipeline families (if any) the transformation graph Pattern/rule resources data-driven policy Fixtures & golden outputs test data"},{"location":"12_genericity_matrix/#the-two-axes","title":"The two axes","text":"<p>Everything app-specific reduces to two axes:</p> <ol> <li>Resource model \u2014 the set of resources, their fields, relationships, and the OpenAPI contract.</li> <li>Pipeline family \u2014 only if inputs require multi-step transformation.</li> </ol> <p>If you can specify those two axes, the rest of the app follows mechanically.</p>"},{"location":"13_library_registry/","title":"Library Registry","text":""},{"location":"13_library_registry/#summary","title":"Summary","text":"<p>The Aetos ecosystem provides internal libraries for the shared concerns of every application. Use them. Writing a private copy of any capability below is a defect.</p> <p>Each library also exposes its own documentation as an MCP server. An AI agent can read a library's full API by connecting to its MCP and reading <code>docs://index</code>, <code>docs://nav</code>, and <code>docs://modules/{module}</code>.</p>"},{"location":"13_library_registry/#catalog","title":"Catalog","text":"Library Import Purpose Use instead of MCP DAG Pipe <code>dagpipe</code> Deterministic state-transformation DAG engine hand-rolled orchestration/topological ordering <code>:8006</code> Hexa <code>hexa</code> Declarative DI / composition-root tooling manual wiring, hand-written factories (see note) Mongo Ops <code>mongo_ops</code> Async MongoDB repositories, cache, populate, transactions raw Motor CRUD and connection lifecycle <code>:8007</code> Omniread <code>omniread</code> Format-agnostic fetch + parse (HTML, PDF, CSV, XLSX) custom scrapers/parsers <code>:8003</code> OpenAPI First <code>openapi_first</code> Spec-first FastAPI bootstrap, client, codegen decorator routers and hand-written clients <code>:8004</code> Mail Intake <code>mail_intake</code> Provider-agnostic read-only email ingestion Gmail OAuth + MIME parsing <code>:8002</code> Py JWT <code>jwtlib</code> JWT auth primitives (register/login/introspect) password hashing + token code <code>:8005</code> Doc Forge <code>docforge</code> Docstring compiler \u2192 MkDocs + MCP docs custom doc generators <code>:8001</code> Auth Server (service) Central identity service (register/login/me/introspect) building auth endpoints yourself \u2014"},{"location":"13_library_registry/#how-to-choose","title":"How to choose","text":"You need to\u2026 Use persist and query documents Mongo Ops compose dependencies Hexa run a multi-step transformation DAG Pipe read/parse an arbitrary input format Omniread expose a contract-first HTTP API OpenAPI First ingest email Mail Intake authenticate users / verify tokens Py JWT (+ Auth Server) generate documentation Doc Forge"},{"location":"13_library_registry/#discovery","title":"Discovery","text":"<p>A single standards MCP exposes <code>find_library(capability)</code>, which returns the matching library, its import name, its \"use instead of\" guidance, and its MCP endpoint. Query it before writing any infrastructure.</p>"},{"location":"13_library_registry/#notes","title":"Notes","text":"<ul> <li>Hexa ships an MCP bundle but is not yet assigned a dedicated port in the hub config; it is reachable via the standards registry entry and will be wired to a port.</li> <li>Library MCP servers expose documentation resources (not executable tools): they tell an agent the available API surface so it does not invent code.</li> </ul>"}]} |