feat: add standards wiki, MCP registry, and app template
This commit is contained in:
69
standards_src/mkdocs.yml
Normal file
69
standards_src/mkdocs.yml
Normal file
@@ -0,0 +1,69 @@
|
||||
site_name: Aetos Standards
|
||||
site_description: The standardized application blueprint and library registry.
|
||||
docs_dir: wiki
|
||||
site_dir: ../standards
|
||||
use_directory_urls: true
|
||||
|
||||
theme:
|
||||
name: material
|
||||
palette:
|
||||
- scheme: slate
|
||||
primary: deep purple
|
||||
accent: cyan
|
||||
font:
|
||||
text: Inter
|
||||
code: JetBrains Mono
|
||||
features:
|
||||
- navigation.sections
|
||||
- navigation.expand
|
||||
- navigation.top
|
||||
- navigation.instant
|
||||
- navigation.tracking
|
||||
- navigation.indexes
|
||||
- content.code.copy
|
||||
- content.code.annotate
|
||||
- content.tabs.link
|
||||
- content.action.edit
|
||||
- search.highlight
|
||||
- search.share
|
||||
- search.suggest
|
||||
|
||||
markdown_extensions:
|
||||
- pymdownx.superfences
|
||||
- pymdownx.inlinehilite
|
||||
- pymdownx.snippets
|
||||
- admonition
|
||||
- pymdownx.details
|
||||
- pymdownx.highlight:
|
||||
linenums: true
|
||||
anchor_linenums: true
|
||||
line_spans: __span
|
||||
pygments_lang_class: true
|
||||
- pymdownx.tabbed:
|
||||
alternate_style: true
|
||||
- pymdownx.tasklist:
|
||||
custom_checkbox: true
|
||||
- tables
|
||||
- footnotes
|
||||
- pymdownx.caret
|
||||
- pymdownx.tilde
|
||||
- pymdownx.mark
|
||||
|
||||
plugins:
|
||||
- search
|
||||
|
||||
nav:
|
||||
- Home: index.md
|
||||
- Overview: 01_overview.md
|
||||
- Layering Rules: 02_layering_rules.md
|
||||
- Composition Root: 03_composition_root.md
|
||||
- Spec-First Routing: 04_spec_first_routing.md
|
||||
- CRUD Template: 05_crud_template.md
|
||||
- Pipeline Scaffold: 06_pipeline_scaffold.md
|
||||
- Worker & Progress: 07_worker_and_progress.md
|
||||
- Human-in-the-Loop: 08_human_in_loop.md
|
||||
- Infra Adapters: 09_infra_adaptors.md
|
||||
- Testing: 10_testing.md
|
||||
- Build Recipe: 11_build_recipe.md
|
||||
- Genericity Matrix: 12_genericity_matrix.md
|
||||
- Library Registry: 13_library_registry.md
|
||||
54
standards_src/wiki/01_overview.md
Normal file
54
standards_src/wiki/01_overview.md
Normal file
@@ -0,0 +1,54 @@
|
||||
# Overview
|
||||
|
||||
## Summary
|
||||
|
||||
The blueprint is a **hexagonal (ports and adapters) application** with a
|
||||
**spec-first** web edge and a **single composition root**. It favors explicit
|
||||
wiring over implicit magic, and it prefers existing internal libraries over
|
||||
hand-written infrastructure.
|
||||
|
||||
---
|
||||
|
||||
## The three layers
|
||||
|
||||
```
|
||||
interfaces/ ← HTTP edge: thin handlers bound to an OpenAPI operationId
|
||||
domain/ ← business core: models, ports (interfaces), services, orchestrators
|
||||
infra/ ← adapters: databases, caches, external clients, hardware
|
||||
```
|
||||
|
||||
Plus two optional layers:
|
||||
|
||||
```
|
||||
pipelines/ ← data-driven transformation graphs (extraction, enrichment)
|
||||
scripts/ ← operator scripts that reuse the composition root
|
||||
```
|
||||
|
||||
## Core principles
|
||||
|
||||
1. **One-way dependencies.** `interfaces -> domain -> infra` is allowed;
|
||||
the reverse is forbidden. The domain never imports a framework it does not
|
||||
own.
|
||||
2. **Ports in the domain, adapters in infra.** A service depends on an
|
||||
abstract port; the composition root injects a concrete adapter.
|
||||
3. **One composition root.** Nothing constructs its own dependencies. All
|
||||
wiring lives in one module (`init.py`) and is exposed as a flat resource
|
||||
registry.
|
||||
4. **Spec-first.** The HTTP contract is declared in an OpenAPI document. Route
|
||||
handlers are looked up by `operationId`. There are no decorator routers.
|
||||
5. **Reuse the libraries.** Pipeline orchestration, DI, Mongo repositories,
|
||||
ingestion, auth, and docs are provided by internal libraries. Do not
|
||||
reimplement them — see the library registry.
|
||||
6. **Explicit state machines.** Long-running work moves through explicit,
|
||||
durable statuses and can pause for human input.
|
||||
|
||||
## Runtime processes
|
||||
|
||||
A standard app may run as up to three processes that share the same
|
||||
composition root:
|
||||
|
||||
| Process | Entry point | Responsibility |
|
||||
|---|---|---|
|
||||
| API | `main.py` | serve HTTP; start/stop resources via lifespan |
|
||||
| Worker | `worker.py` | poll durable jobs, run pipelines, publish progress |
|
||||
| Scripts | `python -m scripts.<name>` | one-shot operator tasks |
|
||||
53
standards_src/wiki/02_layering_rules.md
Normal file
53
standards_src/wiki/02_layering_rules.md
Normal file
@@ -0,0 +1,53 @@
|
||||
# Layering Rules
|
||||
|
||||
## Summary
|
||||
|
||||
The dependency direction is fixed. Violating it is a design defect.
|
||||
|
||||
---
|
||||
|
||||
## Allowed imports
|
||||
|
||||
```
|
||||
interfaces/* -> domain/* (services, ports, models)
|
||||
interfaces/* -> infra/adapters/* (only read-only value objects)
|
||||
domain/* -> domain/* (same layer)
|
||||
domain/* -> pipelines/* (call a pipeline via a port)
|
||||
infra/adapters/* -> domain/ports/* (implement the interface)
|
||||
main.py/init.py -> everything (composition root only)
|
||||
```
|
||||
|
||||
## Forbidden imports
|
||||
|
||||
```
|
||||
domain/* -> interfaces/* (never)
|
||||
domain/* -> fastapi/starlette (never)
|
||||
infra/adapters/* -> interfaces/* (never)
|
||||
interfaces/* -> infra/db/* (state lives in resources, not imports)
|
||||
```
|
||||
|
||||
## Why
|
||||
|
||||
- A pure domain is testable without a server, a database, or an event loop.
|
||||
- Adapters can be swapped (in-memory fake, real database) without touching
|
||||
business logic.
|
||||
- The HTTP edge stays thin and replaceable.
|
||||
|
||||
## The port/adapter contract
|
||||
|
||||
A **port** is an abstract class or protocol declared in `domain/<x>/ports/`.
|
||||
An **adapter** implements it in `infra/adapters/`. The composition root binds
|
||||
the two.
|
||||
|
||||
```
|
||||
domain/<resource>/ports/db.py -> repository port + document model
|
||||
domain/<resource>/ports/service.py -> service + factory
|
||||
infra/db/repos/<vendor>.py -> repository adapter factory
|
||||
infra/adapters/service.py -> generic CRUD service base
|
||||
```
|
||||
|
||||
## Enforcement
|
||||
|
||||
- Keep every domain module importable with only the standard library and
|
||||
Pydantic. If a domain import pulls in `fastapi`, the layering is broken.
|
||||
- Reviewers (and CI) reject `import fastapi` inside `domain/` or `infra/`.
|
||||
89
standards_src/wiki/03_composition_root.md
Normal file
89
standards_src/wiki/03_composition_root.md
Normal file
@@ -0,0 +1,89 @@
|
||||
# Composition Root
|
||||
|
||||
## Summary
|
||||
|
||||
All wiring happens in exactly one module. It builds a **flat resource
|
||||
registry** and exposes it to the HTTP layer and to scripts.
|
||||
|
||||
---
|
||||
|
||||
## The resource registry
|
||||
|
||||
`init.py` returns a plain `dict[str, Any]`. Keys follow two conventions:
|
||||
|
||||
- `<resource>_repo` — a repository adapter for a resource.
|
||||
- `<resource>_svc` — a service for a resource.
|
||||
|
||||
Plus named infrastructure adaptors:
|
||||
|
||||
- `http_cache_store` — response cache backend.
|
||||
- `progress_bus` — progress/event bus.
|
||||
- `<vendor>_<kind>` — fetchers, clients, stores.
|
||||
|
||||
The API stores the dict on `app.state.resources` during lifespan startup.
|
||||
Handlers read from it; nothing imports concrete adapters.
|
||||
|
||||
## Shape
|
||||
|
||||
```python
|
||||
_resources: dict = {}
|
||||
_initialized = False
|
||||
|
||||
async def init(settings=None) -> dict:
|
||||
global _initialized
|
||||
if _initialized:
|
||||
return _resources
|
||||
|
||||
await setup_database(uri=settings.mongo_uri, db_name=settings.mongo_db)
|
||||
|
||||
# 1. leaf repositories
|
||||
user_repo = create_user_repo()
|
||||
|
||||
# 2. dependent repositories (dependencies passed by keyword)
|
||||
wallet_repo = create_wallet_repo(user_repo=user_repo)
|
||||
|
||||
# 3. services (build via the DI library, injecting the repo)
|
||||
_resources["user_repo"] = user_repo
|
||||
_resources["wallet_repo"] = wallet_repo
|
||||
_resources["user_svc"] = get_user_service(user_repo)
|
||||
_resources["wallet_svc"] = get_wallet_service(wallet_repo)
|
||||
|
||||
# 4. infrastructure adaptors (optional)
|
||||
_resources["http_cache_store"] = HttpCacheStore(...)
|
||||
_resources["progress_bus"] = ProgressEventBus()
|
||||
|
||||
_initialized = True
|
||||
return _resources
|
||||
|
||||
async def shutdown() -> None:
|
||||
global _initialized
|
||||
await close_stores(_resources)
|
||||
await shutdown_database()
|
||||
_resources.clear()
|
||||
_initialized = False
|
||||
```
|
||||
|
||||
## Typed accessors
|
||||
|
||||
Expose a `get_svc(name)` helper with one `@overload` per key so scripts get
|
||||
typing without importing concrete classes:
|
||||
|
||||
```python
|
||||
@overload
|
||||
def get_svc(name: Literal["user_svc"]) -> UserService: ...
|
||||
@overload
|
||||
def get_svc(name: Literal["wallet_svc"]) -> WalletService: ...
|
||||
|
||||
def get_svc(name: str):
|
||||
return _resources[name]
|
||||
```
|
||||
|
||||
Also expose `get_db()` (raw database handle) and pure helpers such as
|
||||
`compute_fingerprint(payload)` (canonical hash used for deduplication).
|
||||
|
||||
## Rules
|
||||
|
||||
- `init()` is **idempotent**.
|
||||
- The composition root is the only place that imports concrete adapters.
|
||||
- Handlers never call `init()`; they read `request.app.state.resources`.
|
||||
- Scripts call `init()` / `shutdown()` around their work.
|
||||
77
standards_src/wiki/04_spec_first_routing.md
Normal file
77
standards_src/wiki/04_spec_first_routing.md
Normal file
@@ -0,0 +1,77 @@
|
||||
# Spec-First Routing
|
||||
|
||||
## Summary
|
||||
|
||||
The OpenAPI document is the single source of truth for the HTTP contract.
|
||||
Handlers are bound to it by name; there is no decorator router.
|
||||
|
||||
---
|
||||
|
||||
## The contract
|
||||
|
||||
- `openapi.yaml` (or `.json`) declares paths, schemas, parameters, responses,
|
||||
and security.
|
||||
- A custom `operationId` names the handler function.
|
||||
- The framework (`openapi_first`) validates requests/responses against the
|
||||
spec and dispatches to the handler whose name equals `operationId`.
|
||||
|
||||
## App bootstrap
|
||||
|
||||
```python
|
||||
# main.py
|
||||
from openapi_first.app import OpenAPIFirstApp
|
||||
|
||||
app = OpenAPIFirstApp(
|
||||
openapi_path="openapi.yaml",
|
||||
routes_module=interfaces.openapi, # module whose namespace holds handlers
|
||||
title="<App> API",
|
||||
version="0.1.0",
|
||||
lifespan=lifespan,
|
||||
)
|
||||
```
|
||||
|
||||
`lifespan` calls the composition root:
|
||||
|
||||
```python
|
||||
@asynccontextmanager
|
||||
async def lifespan(app):
|
||||
app.state.resources = await init()
|
||||
yield
|
||||
await shutdown()
|
||||
```
|
||||
|
||||
## Handler registry
|
||||
|
||||
`interfaces/openapi.py` is a flat re-export of every handler, so the framework
|
||||
can resolve `operationId` across the whole app:
|
||||
|
||||
```python
|
||||
from interfaces.item.routes import *
|
||||
from interfaces.wallet.routes import *
|
||||
from interfaces.progress.sse import *
|
||||
```
|
||||
|
||||
## Handler shape
|
||||
|
||||
Handlers are thin and synchronous with respect to business logic:
|
||||
|
||||
```python
|
||||
async def create_item(request: Request, body: dict):
|
||||
svc = request.app.state.resources["item_svc"]
|
||||
return await svc.create(Item.model_validate(body))
|
||||
```
|
||||
|
||||
## Conventions
|
||||
|
||||
- `operationId` naming: `list_<resource>`, `create_<resource>`,
|
||||
`get_<resource>`, `update_<resource>`, `delete_<resource>`.
|
||||
- Read-only endpoints may use a cache decorator keyed on path + query + user.
|
||||
- Validation errors -> `400`; not found -> `404`; delete -> `204`.
|
||||
- Security is declared once in the spec and enforced by the framework; handlers
|
||||
contain no auth code.
|
||||
|
||||
## Codegen
|
||||
|
||||
The `openapi_first` library can scaffold models and route stubs from the spec,
|
||||
and ships app templates (CRUD app, model app, service). Prefer codegen over
|
||||
hand-writing boilerplate.
|
||||
79
standards_src/wiki/05_crud_template.md
Normal file
79
standards_src/wiki/05_crud_template.md
Normal file
@@ -0,0 +1,79 @@
|
||||
# CRUD Template
|
||||
|
||||
## Summary
|
||||
|
||||
Every resource follows the same four-file slice. Repetition is intentional: it
|
||||
makes the codebase predictable and generatable.
|
||||
|
||||
---
|
||||
|
||||
## The slice
|
||||
|
||||
```
|
||||
domain/<resource>/
|
||||
models.py # pure Pydantic model (no DB fields)
|
||||
ports/
|
||||
db.py # document model + repository factory
|
||||
service.py # service class + factory
|
||||
converters.py # (optional) transforms
|
||||
```
|
||||
|
||||
## models.py
|
||||
|
||||
A framework-free Pydantic model describing the business shape.
|
||||
|
||||
```python
|
||||
class Item(BaseModel):
|
||||
name: str
|
||||
quantity: int = 0
|
||||
```
|
||||
|
||||
## ports/db.py
|
||||
|
||||
The persistence model extends the business model and adds database concerns.
|
||||
The repository factory selects the right repository flavour.
|
||||
|
||||
```python
|
||||
class ItemDocument(Item, BaseDocument): # BaseDocument adds id/created_at/...
|
||||
pass
|
||||
|
||||
def create_item_repo(*, owner_repo=None) -> ItemRepository:
|
||||
return repo_factory(
|
||||
collection="items",
|
||||
document=ItemDocument,
|
||||
populate_rules=[PopulateRule(field="owner", collection="users")],
|
||||
cache=None, # or a CachedBaseRepository config
|
||||
)
|
||||
```
|
||||
|
||||
## ports/service.py
|
||||
|
||||
The service is built through the DI library, injecting the repository. The
|
||||
generic CRUD base provides `create/read/update/patch/delete/search/...`.
|
||||
|
||||
```python
|
||||
class ItemService(CrudsServiceAdaptor[Item, ItemDocument]):
|
||||
model = Item
|
||||
schema = ItemDocument
|
||||
|
||||
def get_item_service(repo) -> ItemService:
|
||||
return build(ItemService, instances={"repo": repo})
|
||||
```
|
||||
|
||||
## Naming conventions
|
||||
|
||||
| Concept | Convention |
|
||||
|---|---|
|
||||
| collection | plural snake case (`items`) |
|
||||
| document | `<Resource>Document` |
|
||||
| repository factory | `create_<resource>_repo` |
|
||||
| service | `<Resource>Service` |
|
||||
| service factory | `get_<resource>_service` |
|
||||
| resource key | `<resource>_svc` |
|
||||
| operationId | `list_items`, `create_item`, ... |
|
||||
|
||||
## Optional files
|
||||
|
||||
- `converters.py` — pure functions mapping between domain types.
|
||||
- `ports/enrichment.py` — adapts a pipeline into an async callable the service
|
||||
can invoke (used when a resource is populated by a pipeline).
|
||||
84
standards_src/wiki/06_pipeline_scaffold.md
Normal file
84
standards_src/wiki/06_pipeline_scaffold.md
Normal file
@@ -0,0 +1,84 @@
|
||||
# Pipeline Scaffold
|
||||
|
||||
## Summary
|
||||
|
||||
Data-driven processing is expressed as a **composition graph**. The graph is
|
||||
built by a DI library; execution is driven by a DAG engine. Stages are pure and
|
||||
individually testable.
|
||||
|
||||
---
|
||||
|
||||
## When to use a pipeline
|
||||
|
||||
Use a pipeline when a resource is produced by a **multi-step transformation**
|
||||
of an input (a document, a file, a message) rather than by a simple CRUD write.
|
||||
|
||||
## Structure of a family
|
||||
|
||||
A **family** is one input shape (e.g. a file format or a message kind). Each
|
||||
family has the same skeleton:
|
||||
|
||||
```
|
||||
pipelines/<family>/
|
||||
abc.py # port contracts (stage interfaces + root interface)
|
||||
impl.py # concrete stages + root implementation
|
||||
pipeline.yaml # graph spec: nodes, edges, roots, schema
|
||||
resources/
|
||||
pipeline.yaml # runtime spec consumed by the loader
|
||||
patterns/*.yaml # rule fragments (pattern registry)
|
||||
banks/<variant>.py # optional variants that re-pin a few slots
|
||||
```
|
||||
|
||||
## Shared scaffolding
|
||||
|
||||
```
|
||||
pipelines/shared/
|
||||
ports.py # RegistryPort, FinalizePort, etc.
|
||||
loader.py # build(root_cls) -> wires the graph + engine
|
||||
registry.py # resolve_root(...) -> root class for a (kind, variant)
|
||||
```
|
||||
|
||||
## The loader
|
||||
|
||||
```python
|
||||
def build(root_cls, *, instances=None, config=None):
|
||||
# merge resources/patterns/*.yaml into a pattern registry
|
||||
# bind registry/config into each graph node by id
|
||||
# build the schema (state) class and the engine
|
||||
root = root_cls()
|
||||
root._engine = Engine(graph, state_cls, payload_cls)
|
||||
return root
|
||||
```
|
||||
|
||||
## The registry
|
||||
|
||||
A single lookup replaces hand-rolled `if/else` selection:
|
||||
|
||||
```python
|
||||
_FAMILIES = {
|
||||
"csv": {"acme": {"heuristic": AcmeCsvPipeline}},
|
||||
"json": {"globex": {"heuristic": GlobexJsonPipeline}},
|
||||
}
|
||||
|
||||
def resolve_root(kind, variant, mode="heuristic"):
|
||||
return _FAMILIES[kind][variant][mode]
|
||||
```
|
||||
|
||||
Adding a new variant means adding one class and one registry entry.
|
||||
|
||||
## Driving a pipeline
|
||||
|
||||
Stages are driven explicitly so intermediate state can be persisted:
|
||||
|
||||
```python
|
||||
pipeline = build(resolve_root(kind, variant))
|
||||
content = pipeline.ingest.materialize(raw_bytes)
|
||||
records = pipeline.reader.read(content)
|
||||
items = pipeline.normalize.build(records)
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
Because stages are pure, test each stage in isolation and test the assembled
|
||||
root against **fixture inputs** whose expected outputs are stored as golden
|
||||
files. A parity test replays fixtures headlessly.
|
||||
95
standards_src/wiki/07_worker_and_progress.md
Normal file
95
standards_src/wiki/07_worker_and_progress.md
Normal file
@@ -0,0 +1,95 @@
|
||||
# Worker & Progress
|
||||
|
||||
## Summary
|
||||
|
||||
Long-running work runs in a background worker. Jobs are durable documents that
|
||||
move through an explicit status machine. Progress is streamed to clients.
|
||||
|
||||
---
|
||||
|
||||
## The status machine
|
||||
|
||||
```
|
||||
pending -> processing -> completed
|
||||
\-> failed (retried up to N times)
|
||||
\-> paused (waiting for human input)
|
||||
```
|
||||
|
||||
- `pending` — queued, eligible for pickup.
|
||||
- `processing` — claimed by a worker.
|
||||
- `paused` — blocked on a human-in-the-loop question.
|
||||
- `failed` — error; `retry_count` increments until a max, then skipped.
|
||||
- `completed` — terminal success.
|
||||
|
||||
## The poll loop
|
||||
|
||||
```python
|
||||
async def poll(resources):
|
||||
backoff = MIN_POLL
|
||||
while True:
|
||||
jobs = await job_svc.search(filter={"status": "pending"}, limit=100)
|
||||
backoff = MIN_POLL if jobs else min(backoff * 2, MAX_POLL)
|
||||
for job in jobs:
|
||||
if (job.retry_count or 0) >= MAX_RETRIES:
|
||||
continue
|
||||
await job_svc.update_status(job.id, "processing")
|
||||
try:
|
||||
await process(job, resources)
|
||||
except PauseForResolution:
|
||||
await job_svc.update_status(job.id, "paused")
|
||||
except Exception as e:
|
||||
await job_svc.update_status(
|
||||
job.id, "failed", error=str(e),
|
||||
retry_count=(job.retry_count or 0) + 1,
|
||||
)
|
||||
await asyncio.sleep(backoff)
|
||||
```
|
||||
|
||||
Tunables live in environment variables: `WORKER_POLL_MIN`, `WORKER_POLL_MAX`,
|
||||
`WORKER_MAX_RETRIES`.
|
||||
|
||||
## Progress events
|
||||
|
||||
A worker publishes structured progress events to a **bus** (a port). The HTTP
|
||||
edge exposes a streaming endpoint that replays events for a given job id.
|
||||
|
||||
```python
|
||||
class ProgressEvent(BaseModel):
|
||||
job_id: str
|
||||
step: str
|
||||
status: str # started | progress | completed | skipped | paused
|
||||
message: dict | None = None
|
||||
```
|
||||
|
||||
The bus is an in-memory adapter by default; because it is a port, it can be
|
||||
swapped for a durable broker. Emit from the orchestrator at each stage:
|
||||
|
||||
```python
|
||||
await self.bus.publish(ProgressEvent(job_id=jid, step="normalize",
|
||||
status="completed", message={"count": n}))
|
||||
```
|
||||
|
||||
## Streaming endpoint
|
||||
|
||||
```python
|
||||
async def stream_job_events(request, id):
|
||||
bus = request.app.state.resources.get("progress_bus")
|
||||
if bus is None:
|
||||
return {"events": []}
|
||||
|
||||
async def gen():
|
||||
sent = 0
|
||||
while True:
|
||||
events = bus.get_events(id)
|
||||
for e in events[sent:]:
|
||||
yield f"data: {e.model_dump_json(exclude_none=True)}\n\n"
|
||||
sent += 1
|
||||
await asyncio.sleep(0.5)
|
||||
|
||||
return StreamingResponse(gen(), media_type="text/event-stream")
|
||||
```
|
||||
|
||||
> **Known limitation.** The default in-memory bus is per-process. If the worker
|
||||
> and the API run as separate processes, events published by the worker are not
|
||||
> visible to the API. For production, back the bus with a shared broker. The
|
||||
> port boundary makes this a one-adapter change.
|
||||
87
standards_src/wiki/08_human_in_loop.md
Normal file
87
standards_src/wiki/08_human_in_loop.md
Normal file
@@ -0,0 +1,87 @@
|
||||
# Human-in-the-Loop
|
||||
|
||||
## Summary
|
||||
|
||||
Some inputs are ambiguous and must be resolved by a human before processing can
|
||||
continue. The blueprint models this as a **durable pause**: an exception is
|
||||
raised, a question is recorded, the job pauses, and later resumes.
|
||||
|
||||
---
|
||||
|
||||
## The mechanism
|
||||
|
||||
1. During processing, a stage raises a domain exception carrying the context
|
||||
needed to ask the question:
|
||||
|
||||
```python
|
||||
raise PauseForResolution(
|
||||
job_id=jid,
|
||||
reference="<stable key for this decision>",
|
||||
options=[...],
|
||||
context={...},
|
||||
)
|
||||
```
|
||||
|
||||
2. The orchestrator catches it, records a **question document**, and re-raises
|
||||
so the worker marks the job `paused`.
|
||||
3. A client lists pending questions and answers one:
|
||||
|
||||
```
|
||||
GET /jobs/{id}/questions -> list pending questions
|
||||
POST /questions/{qid}/resolve {choice} -> record the choice
|
||||
```
|
||||
|
||||
4. When the **last** pending question is resolved, the resolve endpoint flips
|
||||
the parent job back to `pending`.
|
||||
5. The worker picks it up again, rebuilds a **resolution map** from previously
|
||||
answered questions, and seeds it into the pipeline so the same decision is
|
||||
not asked twice.
|
||||
|
||||
## The question document
|
||||
|
||||
```python
|
||||
class Question(BaseDocument):
|
||||
job_id: str
|
||||
reference: str # stable key: stage|field|value
|
||||
status: str # pending | resolved
|
||||
options: list[dict]
|
||||
context: dict
|
||||
chosen: dict | None = None
|
||||
resolved_at: str | None = None
|
||||
```
|
||||
|
||||
## Resume map
|
||||
|
||||
The worker turns answered questions into a lookup keyed by `reference`, then
|
||||
injects it into the pipeline so that stage can skip re-raising:
|
||||
|
||||
```python
|
||||
resolved = {q.reference: q.chosen for q in await svc.search(
|
||||
filter={"job_id": jid, "status": "resolved"})}
|
||||
|
||||
pipeline = build(root, instances={"pause": handler}, config={"resolved": resolved})
|
||||
```
|
||||
|
||||
## Resolve endpoint
|
||||
|
||||
```python
|
||||
async def resolve_question(request, id, body):
|
||||
svc = request.app.state.resources["question_svc"]
|
||||
q = await svc.read(id)
|
||||
await svc.update(id, {"chosen": body["chosen"], "status": "resolved",
|
||||
"resolved_at": now()})
|
||||
|
||||
pending = await svc.search(filter={"job_id": q.job_id, "status": "pending"},
|
||||
limit=1)
|
||||
if not pending:
|
||||
jobs = request.app.state.resources["job_svc"]
|
||||
job = await jobs.read(q.job_id)
|
||||
if job and job.status == "paused":
|
||||
await jobs.update_status(q.job_id, "pending")
|
||||
return q
|
||||
```
|
||||
|
||||
## CLI parity
|
||||
|
||||
The same flow is available through scripts/CLI: `list`, `get`, `create`,
|
||||
`questions`, `resolve`, `retry`.
|
||||
84
standards_src/wiki/09_infra_adaptors.md
Normal file
84
standards_src/wiki/09_infra_adaptors.md
Normal file
@@ -0,0 +1,84 @@
|
||||
# Infra Adapters
|
||||
|
||||
## Summary
|
||||
|
||||
All external systems are reached through adapters in `infra/`. Prefer an
|
||||
existing internal library over writing a new adapter.
|
||||
|
||||
---
|
||||
|
||||
## Database (MongoDB)
|
||||
|
||||
- A single connection manager owns the client lifecycle.
|
||||
`setup_database(uri, db_name)` / `shutdown_database()`.
|
||||
- A repository **factory** selects the repository flavour:
|
||||
|
||||
```python
|
||||
repo_factory(collection, document, engine=None,
|
||||
populate_rules=None, cache=None)
|
||||
```
|
||||
|
||||
- base repository — plain CRUD;
|
||||
- populating repository — resolves referenced documents (foreign keys);
|
||||
- cached repository — an LRU/TTL cache in front of reads.
|
||||
|
||||
- Referenced documents are described declaratively:
|
||||
|
||||
```python
|
||||
PopulateRule(field="owner", collection="users",
|
||||
nested_rules=[], max_depth=2, projection=None)
|
||||
```
|
||||
|
||||
- Transactions use a session context manager provided by the library.
|
||||
|
||||
**Library:** the Mongo operations library (see registry). Do not write raw
|
||||
Motor calls in the domain.
|
||||
|
||||
## HTTP response cache
|
||||
|
||||
A generic GET cache sits behind a decorator.
|
||||
|
||||
```python
|
||||
@cache_response(ttl=60)
|
||||
async def list_items(request): ...
|
||||
```
|
||||
|
||||
- Keyed on path + query + user.
|
||||
- No-op when no store is wired.
|
||||
- `invalidate(*resources)` bumps a generation counter for write-triggered
|
||||
invalidation.
|
||||
|
||||
## Event / progress bus
|
||||
|
||||
A port with an in-memory default adapter (`publish`, `get_events`). Swap for a
|
||||
durable broker in production (see Worker & Progress).
|
||||
|
||||
## External clients (ingestion)
|
||||
|
||||
External fetchers (files, mail, HTTP) are adapters that wrap ingestion
|
||||
libraries. Construct them in the composition root and inject into
|
||||
orchestrators.
|
||||
|
||||
**Libraries:** the ingestion library (formats) and the mail library (email).
|
||||
|
||||
## LLM clients
|
||||
|
||||
When a pipeline calls a model provider, route through a pool adaptor that
|
||||
handles provider selection, retries, and client lifecycle:
|
||||
|
||||
```python
|
||||
pool = setup_async_pool(provider=settings.llm_provider)
|
||||
result = await pool.complete(model="...", messages=[...])
|
||||
```
|
||||
|
||||
Providers are selected by `LLM_PROVIDER` and configured by environment.
|
||||
|
||||
## Key-value store
|
||||
|
||||
For report/snapshot storage, provide a pluggable store port with a Redis
|
||||
adapter and an in-memory fallback.
|
||||
|
||||
## Rule
|
||||
|
||||
If a capability already exists as a library (DB, DI, ingestion, mail, auth,
|
||||
docs), **use it**. Writing a private copy of a library is a defect.
|
||||
56
standards_src/wiki/10_testing.md
Normal file
56
standards_src/wiki/10_testing.md
Normal file
@@ -0,0 +1,56 @@
|
||||
# Testing
|
||||
|
||||
## Summary
|
||||
|
||||
Two test suites: **integration** (the HTTP edge with fakes) and **unit** (pure
|
||||
domain and pipeline logic). Both run under pytest with asyncio auto mode.
|
||||
|
||||
---
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
tests/
|
||||
integration/ # TestClient(app) + mocked resources
|
||||
unit/ # models, converters, pipeline stages/roots
|
||||
conftest.py # fixtures
|
||||
```
|
||||
|
||||
## Integration tests
|
||||
|
||||
- Build the app with `TestClient`.
|
||||
- Replace `app.state.resources` with fake repositories/services so no database
|
||||
is required.
|
||||
- Patch the auth introspection call (`httpx.AsyncClient.post`) to return a
|
||||
canned identity.
|
||||
- Assert status codes and bodies for list/get/create/update/delete, plus
|
||||
`400`, `404`, `204`, and the streaming endpoint's fallback.
|
||||
|
||||
```python
|
||||
client = TestClient(app)
|
||||
app.state.resources = {"item_svc": FakeItemService(), ...}
|
||||
|
||||
def test_create_item():
|
||||
r = client.post("/items", json={"name": "x"})
|
||||
assert r.status_code == 200
|
||||
```
|
||||
|
||||
## Unit tests
|
||||
|
||||
- Domain models and converters: pure, no I/O.
|
||||
- Pipeline stages: feed a fixture state, assert the transformed state.
|
||||
- Assembled pipeline roots: **fixture parity** — replay a stored fixture input
|
||||
and compare against a golden output file.
|
||||
|
||||
## Config-consistency tests
|
||||
|
||||
For rule-driven pipelines, add a test that cross-references rules against the
|
||||
declared pattern registry: every flag emitted must be defined; every defined
|
||||
flag must be reachable. This catches drift without running the pipeline.
|
||||
|
||||
## Conventions
|
||||
|
||||
- `asyncio_mode = "auto"`; no manual event loops.
|
||||
- Fixtures are checked into the repo (inputs + golden outputs).
|
||||
- Tests skip (not fail) when a required fixture is absent.
|
||||
- Do not mock the domain; mock only the infrastructure edge.
|
||||
82
standards_src/wiki/11_build_recipe.md
Normal file
82
standards_src/wiki/11_build_recipe.md
Normal file
@@ -0,0 +1,82 @@
|
||||
# Build Recipe
|
||||
|
||||
## Summary
|
||||
|
||||
Follow this ordered checklist to generate a new application of the standard
|
||||
shape. Every step is deterministic; only steps 2 and 7 require app-specific
|
||||
authoring.
|
||||
|
||||
---
|
||||
|
||||
## 0. Query the registry
|
||||
|
||||
Before writing anything, read the **library registry** and note which internal
|
||||
libraries provide DB, DI, routing, ingestion, auth, and docs. Do not
|
||||
reimplement them.
|
||||
|
||||
## 1. Scaffold the repository
|
||||
|
||||
Copy the `templates/` skeleton. It provides: `pyproject.toml`, `openapi.yaml`,
|
||||
`main.py`, `init.py`, `worker.py`, `config.py`, `Dockerfile`,
|
||||
`docker-compose.yaml`, `.env.example`, and the package tree
|
||||
(`domain/ infra/ interfaces/ pipelines/ tests/ scripts/`).
|
||||
|
||||
## 2. Author the spec *(app-specific)*
|
||||
|
||||
Write `openapi.yaml`: resources, schemas, `operationId`s, security, and any
|
||||
extension metadata. This is the contract.
|
||||
|
||||
## 3. Write each CRUD slice *(app-specific)*
|
||||
|
||||
For each resource, create the four-file slice (see **CRUD Template**):
|
||||
`models.py`, `ports/db.py`, `ports/service.py`, optional `converters.py`.
|
||||
|
||||
## 4. Wire the composition root
|
||||
|
||||
In `init.py`: build leaf repositories, then dependent repositories, then
|
||||
services; add infrastructure adaptors (cache store, progress bus, stores,
|
||||
fetchers). Add `get_svc` overloads.
|
||||
|
||||
## 5. Add routes
|
||||
|
||||
Create `interfaces/<resource>/routes.py` (thin handlers) and re-export them in
|
||||
`interfaces/openapi.py`. Add a cache decorator to read endpoints.
|
||||
|
||||
## 6. Bootstrap the app and worker
|
||||
|
||||
Confirm `main.py` wires the OpenAPI-first app with the lifespan. If the app has
|
||||
long-running jobs, adapt `worker.py` to poll the relevant job collection.
|
||||
|
||||
## 7. Add a pipeline family *(only if data-driven)*
|
||||
|
||||
Create `pipelines/<family>/` with `abc.py`, `impl.py`, `pipeline.yaml`,
|
||||
`resources/`, and register the root in `registry.py`. Add fixtures and a parity
|
||||
test.
|
||||
|
||||
## 8. Add infrastructure as needed
|
||||
|
||||
Only where the standard adaptors do not cover the need. Prefer library-backed
|
||||
adapters.
|
||||
|
||||
## 9. Tests
|
||||
|
||||
Add integration tests with fakes and unit tests for domain/pipeline logic; add
|
||||
a config-consistency test if using rule-driven pipelines.
|
||||
|
||||
## 10. Scripts and seeds
|
||||
|
||||
Add operator scripts under `scripts/` that use `init()` / `get_svc()` /
|
||||
`shutdown()`.
|
||||
|
||||
## 11. Docs
|
||||
|
||||
Author GSDFC docstrings, generate the type stubs (`.pyi`), mark the package
|
||||
`py.typed`, and build docs.
|
||||
|
||||
## Definition of done
|
||||
|
||||
- `python -c "import main"` succeeds.
|
||||
- `pytest` passes.
|
||||
- The app serves the spec's operations; the worker drains its queue.
|
||||
- No domain module imports a web framework.
|
||||
- No infrastructure was reimplemented that a library already provides.
|
||||
57
standards_src/wiki/12_genericity_matrix.md
Normal file
57
standards_src/wiki/12_genericity_matrix.md
Normal file
@@ -0,0 +1,57 @@
|
||||
# Genericity Matrix
|
||||
|
||||
## Summary
|
||||
|
||||
This page states precisely what is **reusable as-is**, what is a **template to
|
||||
adapt**, and what is **app-specific and must be authored**. Use it to scope a
|
||||
new build.
|
||||
|
||||
---
|
||||
|
||||
## Reusable as-is (do not rewrite)
|
||||
|
||||
| Area | Artifact |
|
||||
|---|---|
|
||||
| Layering & dependency rule | the package layout and import rules |
|
||||
| Composition root | the `init.py` shape, idempotency, `get_svc` overloading |
|
||||
| Web edge | `OpenAPIFirstApp` bootstrap, handler-name = `operationId`, thin handlers |
|
||||
| CRUD slice | the four-file pattern (model / ports-db / ports-service / converters) |
|
||||
| Generic CRUD service | the base service adaptor |
|
||||
| Repository factory | base / populating / cached selection |
|
||||
| Worker | the poll loop, backoff, retry, status machine |
|
||||
| Progress bus | the bus port + in-memory adapter + streaming endpoint |
|
||||
| Human-in-the-loop | the durable pause + question document + resolve endpoint + resume map |
|
||||
| Infra adaptors | HTTP cache decorator, KV store port, LLM pool scaffold |
|
||||
| Testing | integration harness with fakes, fixture-parity pattern |
|
||||
| Docs | GSDFC + generated stubs + MCP bundle pipeline |
|
||||
|
||||
## Template to adapt (copy, then change names)
|
||||
|
||||
| Artifact | Change |
|
||||
|---|---|
|
||||
| `pyproject.toml` | project name, dependencies |
|
||||
| `Dockerfile` / `docker-compose.yaml` | service name, ports, services |
|
||||
| `.env.example` | connection strings, provider keys |
|
||||
| `openapi.yaml` | every path/schema/operationId |
|
||||
| `scripts/` | seed and operator tasks |
|
||||
|
||||
## App-specific (must be authored)
|
||||
|
||||
| Artifact | Why |
|
||||
|---|---|
|
||||
| Resource models | the business shape |
|
||||
| Repository populate/cache rules | the relationships |
|
||||
| Service business logic | domain behaviour |
|
||||
| Pipeline families (if any) | the transformation graph |
|
||||
| Pattern/rule resources | data-driven policy |
|
||||
| Fixtures & golden outputs | test data |
|
||||
|
||||
## The two axes
|
||||
|
||||
Everything app-specific reduces to **two axes**:
|
||||
|
||||
1. **Resource model** — the set of resources, their fields, relationships, and
|
||||
the OpenAPI contract.
|
||||
2. **Pipeline family** — only if inputs require multi-step transformation.
|
||||
|
||||
If you can specify those two axes, the rest of the app follows mechanically.
|
||||
54
standards_src/wiki/13_library_registry.md
Normal file
54
standards_src/wiki/13_library_registry.md
Normal file
@@ -0,0 +1,54 @@
|
||||
# Library Registry
|
||||
|
||||
## Summary
|
||||
|
||||
The Aetos ecosystem provides internal libraries for the shared concerns of
|
||||
every application. **Use them.** Writing a private copy of any capability below
|
||||
is a defect.
|
||||
|
||||
Each library also exposes its own documentation as an MCP server. An AI agent
|
||||
can read a library's full API by connecting to its MCP and reading
|
||||
`docs://index`, `docs://nav`, and `docs://modules/{module}`.
|
||||
|
||||
---
|
||||
|
||||
## Catalog
|
||||
|
||||
| Library | Import | Purpose | Use instead of | MCP |
|
||||
|---|---|---|---|---|
|
||||
| DAG Pipe | `dagpipe` | Deterministic state-transformation DAG engine | hand-rolled orchestration/topological ordering | `:8006` |
|
||||
| Hexa | `hexa` | Declarative DI / composition-root tooling | manual wiring, hand-written factories | *(see note)* |
|
||||
| Mongo Ops | `mongo_ops` | Async MongoDB repositories, cache, populate, transactions | raw Motor CRUD and connection lifecycle | `:8007` |
|
||||
| Omniread | `omniread` | Format-agnostic fetch + parse (HTML, PDF, CSV, XLSX) | custom scrapers/parsers | `:8003` |
|
||||
| OpenAPI First | `openapi_first` | Spec-first FastAPI bootstrap, client, codegen | decorator routers and hand-written clients | `:8004` |
|
||||
| Mail Intake | `mail_intake` | Provider-agnostic read-only email ingestion | Gmail OAuth + MIME parsing | `:8002` |
|
||||
| Py JWT | `jwtlib` | JWT auth primitives (register/login/introspect) | password hashing + token code | `:8005` |
|
||||
| Doc Forge | `docforge` | Docstring compiler → MkDocs + MCP docs | custom doc generators | `:8001` |
|
||||
| Auth Server | *(service)* | Central identity service (register/login/me/introspect) | building auth endpoints yourself | — |
|
||||
|
||||
## How to choose
|
||||
|
||||
| You need to… | Use |
|
||||
|---|---|
|
||||
| persist and query documents | Mongo Ops |
|
||||
| compose dependencies | Hexa |
|
||||
| run a multi-step transformation | DAG Pipe |
|
||||
| read/parse an arbitrary input format | Omniread |
|
||||
| expose a contract-first HTTP API | OpenAPI First |
|
||||
| ingest email | Mail Intake |
|
||||
| authenticate users / verify tokens | Py JWT (+ Auth Server) |
|
||||
| generate documentation | Doc Forge |
|
||||
|
||||
## Discovery
|
||||
|
||||
A single **standards MCP** exposes `find_library(capability)`, which returns the
|
||||
matching library, its import name, its "use instead of" guidance, and its MCP
|
||||
endpoint. Query it before writing any infrastructure.
|
||||
|
||||
## Notes
|
||||
|
||||
- **Hexa** ships an MCP bundle but is not yet assigned a dedicated port in the
|
||||
hub config; it is reachable via the standards registry entry and will be
|
||||
wired to a port.
|
||||
- Library MCP servers expose **documentation resources** (not executable tools):
|
||||
they tell an agent the available API surface so it does not invent code.
|
||||
45
standards_src/wiki/index.md
Normal file
45
standards_src/wiki/index.md
Normal file
@@ -0,0 +1,45 @@
|
||||
# The Aetos Standard App Blueprint
|
||||
|
||||
A single, reusable architecture for backend applications in the Aetos ecosystem.
|
||||
|
||||
Every application built with this blueprint has the same shape:
|
||||
|
||||
- a **spec-first HTTP layer** driven by an OpenAPI document;
|
||||
- a **domain layer** of pure models, ports, and services;
|
||||
- an **infrastructure layer** of adapters (MongoDB, cache, workers, LLM, mail);
|
||||
- a **composition root** that wires everything in exactly one place;
|
||||
- an optional **pipeline layer** for data-driven extraction/enrichment;
|
||||
- an optional **background worker** that runs long-running jobs and reports progress.
|
||||
|
||||
> This blueprints is deliberately **app-agnostic**. It never names a concrete
|
||||
> business domain. Any application — expenses, health metrics, inventory,
|
||||
> analytics — is produced by filling in **two axes only**: the resource model
|
||||
> and (if needed) the pipeline family.
|
||||
|
||||
## How to read this
|
||||
|
||||
| Page | What it covers |
|
||||
|---|---|
|
||||
| Overview | The philosophy and the one-way dependency rule |
|
||||
| Layering Rules | Which package may import which |
|
||||
| Composition Root | The single wiring point |
|
||||
| Spec-First Routing | OpenAPI as the source of truth |
|
||||
| CRUD Template | The canonical resource slice |
|
||||
| Pipeline Scaffold | Data-driven processing families |
|
||||
| Worker & Progress | Async jobs, status machine, streaming progress |
|
||||
| Human-in-the-Loop | Pausing a job to ask a question |
|
||||
| Infra Adapters | MongoDB, cache, stores, LLM, mail |
|
||||
| Testing | Integration + unit conventions |
|
||||
| Build Recipe | The ordered checklist to generate a new app |
|
||||
| Genericity Matrix | What is reusable vs what you must write |
|
||||
|
||||
## For AI agents
|
||||
|
||||
This content is served as MCP resources under `docs://index`, `docs://nav`,
|
||||
and `docs://modules/{module}`. To build a new app:
|
||||
|
||||
1. Read `build_recipe` first.
|
||||
2. Read `genericity_matrix` to know what you must author.
|
||||
3. Query the **library registry** (`find_library`) before writing any
|
||||
infrastructure yourself.
|
||||
4. Copy the `templates/` skeleton and fill the two app-specific axes.
|
||||
Reference in New Issue
Block a user