refactor: conform standards docs to doc-forge wiki/lib layout; generic build.py; static kind links
This commit is contained in:
53
standards/src/lib/02_layering_rules.md
Normal file
53
standards/src/lib/02_layering_rules.md
Normal file
@@ -0,0 +1,53 @@
|
||||
# Layering Rules
|
||||
|
||||
## Summary
|
||||
|
||||
The dependency direction is fixed. Violating it is a design defect.
|
||||
|
||||
---
|
||||
|
||||
## Allowed imports
|
||||
|
||||
```
|
||||
interfaces/* -> domain/* (services, ports, models)
|
||||
interfaces/* -> infra/adapters/* (only read-only value objects)
|
||||
domain/* -> domain/* (same layer)
|
||||
domain/* -> pipelines/* (call a pipeline via a port)
|
||||
infra/adapters/* -> domain/ports/* (implement the interface)
|
||||
main.py/init.py -> everything (composition root only)
|
||||
```
|
||||
|
||||
## Forbidden imports
|
||||
|
||||
```
|
||||
domain/* -> interfaces/* (never)
|
||||
domain/* -> fastapi/starlette (never)
|
||||
infra/adapters/* -> interfaces/* (never)
|
||||
interfaces/* -> infra/db/* (state lives in resources, not imports)
|
||||
```
|
||||
|
||||
## Why
|
||||
|
||||
- A pure domain is testable without a server, a database, or an event loop.
|
||||
- Adapters can be swapped (in-memory fake, real database) without touching
|
||||
business logic.
|
||||
- The HTTP edge stays thin and replaceable.
|
||||
|
||||
## The port/adapter contract
|
||||
|
||||
A **port** is an abstract class or protocol declared in `domain/<x>/ports/`.
|
||||
An **adapter** implements it in `infra/adapters/`. The composition root binds
|
||||
the two.
|
||||
|
||||
```
|
||||
domain/<resource>/ports/db.py -> repository port + document model
|
||||
domain/<resource>/ports/service.py -> service + factory
|
||||
infra/db/repos/<vendor>.py -> repository adapter factory
|
||||
infra/adapters/service.py -> generic CRUD service base
|
||||
```
|
||||
|
||||
## Enforcement
|
||||
|
||||
- Keep every domain module importable with only the standard library and
|
||||
Pydantic. If a domain import pulls in `fastapi`, the layering is broken.
|
||||
- Reviewers (and CI) reject `import fastapi` inside `domain/` or `infra/`.
|
||||
89
standards/src/lib/03_composition_root.md
Normal file
89
standards/src/lib/03_composition_root.md
Normal file
@@ -0,0 +1,89 @@
|
||||
# Composition Root
|
||||
|
||||
## Summary
|
||||
|
||||
All wiring happens in exactly one module. It builds a **flat resource
|
||||
registry** and exposes it to the HTTP layer and to scripts.
|
||||
|
||||
---
|
||||
|
||||
## The resource registry
|
||||
|
||||
`init.py` returns a plain `dict[str, Any]`. Keys follow two conventions:
|
||||
|
||||
- `<resource>_repo` — a repository adapter for a resource.
|
||||
- `<resource>_svc` — a service for a resource.
|
||||
|
||||
Plus named infrastructure adaptors:
|
||||
|
||||
- `http_cache_store` — response cache backend.
|
||||
- `progress_bus` — progress/event bus.
|
||||
- `<vendor>_<kind>` — fetchers, clients, stores.
|
||||
|
||||
The API stores the dict on `app.state.resources` during lifespan startup.
|
||||
Handlers read from it; nothing imports concrete adapters.
|
||||
|
||||
## Shape
|
||||
|
||||
```python
|
||||
_resources: dict = {}
|
||||
_initialized = False
|
||||
|
||||
async def init(settings=None) -> dict:
|
||||
global _initialized
|
||||
if _initialized:
|
||||
return _resources
|
||||
|
||||
await setup_database(uri=settings.mongo_uri, db_name=settings.mongo_db)
|
||||
|
||||
# 1. leaf repositories
|
||||
user_repo = create_user_repo()
|
||||
|
||||
# 2. dependent repositories (dependencies passed by keyword)
|
||||
wallet_repo = create_wallet_repo(user_repo=user_repo)
|
||||
|
||||
# 3. services (build via the DI library, injecting the repo)
|
||||
_resources["user_repo"] = user_repo
|
||||
_resources["wallet_repo"] = wallet_repo
|
||||
_resources["user_svc"] = get_user_service(user_repo)
|
||||
_resources["wallet_svc"] = get_wallet_service(wallet_repo)
|
||||
|
||||
# 4. infrastructure adaptors (optional)
|
||||
_resources["http_cache_store"] = HttpCacheStore(...)
|
||||
_resources["progress_bus"] = ProgressEventBus()
|
||||
|
||||
_initialized = True
|
||||
return _resources
|
||||
|
||||
async def shutdown() -> None:
|
||||
global _initialized
|
||||
await close_stores(_resources)
|
||||
await shutdown_database()
|
||||
_resources.clear()
|
||||
_initialized = False
|
||||
```
|
||||
|
||||
## Typed accessors
|
||||
|
||||
Expose a `get_svc(name)` helper with one `@overload` per key so scripts get
|
||||
typing without importing concrete classes:
|
||||
|
||||
```python
|
||||
@overload
|
||||
def get_svc(name: Literal["user_svc"]) -> UserService: ...
|
||||
@overload
|
||||
def get_svc(name: Literal["wallet_svc"]) -> WalletService: ...
|
||||
|
||||
def get_svc(name: str):
|
||||
return _resources[name]
|
||||
```
|
||||
|
||||
Also expose `get_db()` (raw database handle) and pure helpers such as
|
||||
`compute_fingerprint(payload)` (canonical hash used for deduplication).
|
||||
|
||||
## Rules
|
||||
|
||||
- `init()` is **idempotent**.
|
||||
- The composition root is the only place that imports concrete adapters.
|
||||
- Handlers never call `init()`; they read `request.app.state.resources`.
|
||||
- Scripts call `init()` / `shutdown()` around their work.
|
||||
77
standards/src/lib/04_spec_first_routing.md
Normal file
77
standards/src/lib/04_spec_first_routing.md
Normal file
@@ -0,0 +1,77 @@
|
||||
# Spec-First Routing
|
||||
|
||||
## Summary
|
||||
|
||||
The OpenAPI document is the single source of truth for the HTTP contract.
|
||||
Handlers are bound to it by name; there is no decorator router.
|
||||
|
||||
---
|
||||
|
||||
## The contract
|
||||
|
||||
- `openapi.yaml` (or `.json`) declares paths, schemas, parameters, responses,
|
||||
and security.
|
||||
- A custom `operationId` names the handler function.
|
||||
- The framework (`openapi_first`) validates requests/responses against the
|
||||
spec and dispatches to the handler whose name equals `operationId`.
|
||||
|
||||
## App bootstrap
|
||||
|
||||
```python
|
||||
# main.py
|
||||
from openapi_first.app import OpenAPIFirstApp
|
||||
|
||||
app = OpenAPIFirstApp(
|
||||
openapi_path="openapi.yaml",
|
||||
routes_module=interfaces.openapi, # module whose namespace holds handlers
|
||||
title="<App> API",
|
||||
version="0.1.0",
|
||||
lifespan=lifespan,
|
||||
)
|
||||
```
|
||||
|
||||
`lifespan` calls the composition root:
|
||||
|
||||
```python
|
||||
@asynccontextmanager
|
||||
async def lifespan(app):
|
||||
app.state.resources = await init()
|
||||
yield
|
||||
await shutdown()
|
||||
```
|
||||
|
||||
## Handler registry
|
||||
|
||||
`interfaces/openapi.py` is a flat re-export of every handler, so the framework
|
||||
can resolve `operationId` across the whole app:
|
||||
|
||||
```python
|
||||
from interfaces.item.routes import *
|
||||
from interfaces.wallet.routes import *
|
||||
from interfaces.progress.sse import *
|
||||
```
|
||||
|
||||
## Handler shape
|
||||
|
||||
Handlers are thin and synchronous with respect to business logic:
|
||||
|
||||
```python
|
||||
async def create_item(request: Request, body: dict):
|
||||
svc = request.app.state.resources["item_svc"]
|
||||
return await svc.create(Item.model_validate(body))
|
||||
```
|
||||
|
||||
## Conventions
|
||||
|
||||
- `operationId` naming: `list_<resource>`, `create_<resource>`,
|
||||
`get_<resource>`, `update_<resource>`, `delete_<resource>`.
|
||||
- Read-only endpoints may use a cache decorator keyed on path + query + user.
|
||||
- Validation errors -> `400`; not found -> `404`; delete -> `204`.
|
||||
- Security is declared once in the spec and enforced by the framework; handlers
|
||||
contain no auth code.
|
||||
|
||||
## Codegen
|
||||
|
||||
The `openapi_first` library can scaffold models and route stubs from the spec,
|
||||
and ships app templates (CRUD app, model app, service). Prefer codegen over
|
||||
hand-writing boilerplate.
|
||||
79
standards/src/lib/05_crud_template.md
Normal file
79
standards/src/lib/05_crud_template.md
Normal file
@@ -0,0 +1,79 @@
|
||||
# CRUD Template
|
||||
|
||||
## Summary
|
||||
|
||||
Every resource follows the same four-file slice. Repetition is intentional: it
|
||||
makes the codebase predictable and generatable.
|
||||
|
||||
---
|
||||
|
||||
## The slice
|
||||
|
||||
```
|
||||
domain/<resource>/
|
||||
models.py # pure Pydantic model (no DB fields)
|
||||
ports/
|
||||
db.py # document model + repository factory
|
||||
service.py # service class + factory
|
||||
converters.py # (optional) transforms
|
||||
```
|
||||
|
||||
## models.py
|
||||
|
||||
A framework-free Pydantic model describing the business shape.
|
||||
|
||||
```python
|
||||
class Item(BaseModel):
|
||||
name: str
|
||||
quantity: int = 0
|
||||
```
|
||||
|
||||
## ports/db.py
|
||||
|
||||
The persistence model extends the business model and adds database concerns.
|
||||
The repository factory selects the right repository flavour.
|
||||
|
||||
```python
|
||||
class ItemDocument(Item, BaseDocument): # BaseDocument adds id/created_at/...
|
||||
pass
|
||||
|
||||
def create_item_repo(*, owner_repo=None) -> ItemRepository:
|
||||
return repo_factory(
|
||||
collection="items",
|
||||
document=ItemDocument,
|
||||
populate_rules=[PopulateRule(field="owner", collection="users")],
|
||||
cache=None, # or a CachedBaseRepository config
|
||||
)
|
||||
```
|
||||
|
||||
## ports/service.py
|
||||
|
||||
The service is built through the DI library, injecting the repository. The
|
||||
generic CRUD base provides `create/read/update/patch/delete/search/...`.
|
||||
|
||||
```python
|
||||
class ItemService(CrudsServiceAdaptor[Item, ItemDocument]):
|
||||
model = Item
|
||||
schema = ItemDocument
|
||||
|
||||
def get_item_service(repo) -> ItemService:
|
||||
return build(ItemService, instances={"repo": repo})
|
||||
```
|
||||
|
||||
## Naming conventions
|
||||
|
||||
| Concept | Convention |
|
||||
|---|---|
|
||||
| collection | plural snake case (`items`) |
|
||||
| document | `<Resource>Document` |
|
||||
| repository factory | `create_<resource>_repo` |
|
||||
| service | `<Resource>Service` |
|
||||
| service factory | `get_<resource>_service` |
|
||||
| resource key | `<resource>_svc` |
|
||||
| operationId | `list_items`, `create_item`, ... |
|
||||
|
||||
## Optional files
|
||||
|
||||
- `converters.py` — pure functions mapping between domain types.
|
||||
- `ports/enrichment.py` — adapts a pipeline into an async callable the service
|
||||
can invoke (used when a resource is populated by a pipeline).
|
||||
84
standards/src/lib/06_pipeline_scaffold.md
Normal file
84
standards/src/lib/06_pipeline_scaffold.md
Normal file
@@ -0,0 +1,84 @@
|
||||
# Pipeline Scaffold
|
||||
|
||||
## Summary
|
||||
|
||||
Data-driven processing is expressed as a **composition graph**. The graph is
|
||||
built by a DI library; execution is driven by a DAG engine. Stages are pure and
|
||||
individually testable.
|
||||
|
||||
---
|
||||
|
||||
## When to use a pipeline
|
||||
|
||||
Use a pipeline when a resource is produced by a **multi-step transformation**
|
||||
of an input (a document, a file, a message) rather than by a simple CRUD write.
|
||||
|
||||
## Structure of a family
|
||||
|
||||
A **family** is one input shape (e.g. a file format or a message kind). Each
|
||||
family has the same skeleton:
|
||||
|
||||
```
|
||||
pipelines/<family>/
|
||||
abc.py # port contracts (stage interfaces + root interface)
|
||||
impl.py # concrete stages + root implementation
|
||||
pipeline.yaml # graph spec: nodes, edges, roots, schema
|
||||
resources/
|
||||
pipeline.yaml # runtime spec consumed by the loader
|
||||
patterns/*.yaml # rule fragments (pattern registry)
|
||||
banks/<variant>.py # optional variants that re-pin a few slots
|
||||
```
|
||||
|
||||
## Shared scaffolding
|
||||
|
||||
```
|
||||
pipelines/shared/
|
||||
ports.py # RegistryPort, FinalizePort, etc.
|
||||
loader.py # build(root_cls) -> wires the graph + engine
|
||||
registry.py # resolve_root(...) -> root class for a (kind, variant)
|
||||
```
|
||||
|
||||
## The loader
|
||||
|
||||
```python
|
||||
def build(root_cls, *, instances=None, config=None):
|
||||
# merge resources/patterns/*.yaml into a pattern registry
|
||||
# bind registry/config into each graph node by id
|
||||
# build the schema (state) class and the engine
|
||||
root = root_cls()
|
||||
root._engine = Engine(graph, state_cls, payload_cls)
|
||||
return root
|
||||
```
|
||||
|
||||
## The registry
|
||||
|
||||
A single lookup replaces hand-rolled `if/else` selection:
|
||||
|
||||
```python
|
||||
_FAMILIES = {
|
||||
"csv": {"acme": {"heuristic": AcmeCsvPipeline}},
|
||||
"json": {"globex": {"heuristic": GlobexJsonPipeline}},
|
||||
}
|
||||
|
||||
def resolve_root(kind, variant, mode="heuristic"):
|
||||
return _FAMILIES[kind][variant][mode]
|
||||
```
|
||||
|
||||
Adding a new variant means adding one class and one registry entry.
|
||||
|
||||
## Driving a pipeline
|
||||
|
||||
Stages are driven explicitly so intermediate state can be persisted:
|
||||
|
||||
```python
|
||||
pipeline = build(resolve_root(kind, variant))
|
||||
content = pipeline.ingest.materialize(raw_bytes)
|
||||
records = pipeline.reader.read(content)
|
||||
items = pipeline.normalize.build(records)
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
Because stages are pure, test each stage in isolation and test the assembled
|
||||
root against **fixture inputs** whose expected outputs are stored as golden
|
||||
files. A parity test replays fixtures headlessly.
|
||||
95
standards/src/lib/07_worker_and_progress.md
Normal file
95
standards/src/lib/07_worker_and_progress.md
Normal file
@@ -0,0 +1,95 @@
|
||||
# Worker & Progress
|
||||
|
||||
## Summary
|
||||
|
||||
Long-running work runs in a background worker. Jobs are durable documents that
|
||||
move through an explicit status machine. Progress is streamed to clients.
|
||||
|
||||
---
|
||||
|
||||
## The status machine
|
||||
|
||||
```
|
||||
pending -> processing -> completed
|
||||
\-> failed (retried up to N times)
|
||||
\-> paused (waiting for human input)
|
||||
```
|
||||
|
||||
- `pending` — queued, eligible for pickup.
|
||||
- `processing` — claimed by a worker.
|
||||
- `paused` — blocked on a human-in-the-loop question.
|
||||
- `failed` — error; `retry_count` increments until a max, then skipped.
|
||||
- `completed` — terminal success.
|
||||
|
||||
## The poll loop
|
||||
|
||||
```python
|
||||
async def poll(resources):
|
||||
backoff = MIN_POLL
|
||||
while True:
|
||||
jobs = await job_svc.search(filter={"status": "pending"}, limit=100)
|
||||
backoff = MIN_POLL if jobs else min(backoff * 2, MAX_POLL)
|
||||
for job in jobs:
|
||||
if (job.retry_count or 0) >= MAX_RETRIES:
|
||||
continue
|
||||
await job_svc.update_status(job.id, "processing")
|
||||
try:
|
||||
await process(job, resources)
|
||||
except PauseForResolution:
|
||||
await job_svc.update_status(job.id, "paused")
|
||||
except Exception as e:
|
||||
await job_svc.update_status(
|
||||
job.id, "failed", error=str(e),
|
||||
retry_count=(job.retry_count or 0) + 1,
|
||||
)
|
||||
await asyncio.sleep(backoff)
|
||||
```
|
||||
|
||||
Tunables live in environment variables: `WORKER_POLL_MIN`, `WORKER_POLL_MAX`,
|
||||
`WORKER_MAX_RETRIES`.
|
||||
|
||||
## Progress events
|
||||
|
||||
A worker publishes structured progress events to a **bus** (a port). The HTTP
|
||||
edge exposes a streaming endpoint that replays events for a given job id.
|
||||
|
||||
```python
|
||||
class ProgressEvent(BaseModel):
|
||||
job_id: str
|
||||
step: str
|
||||
status: str # started | progress | completed | skipped | paused
|
||||
message: dict | None = None
|
||||
```
|
||||
|
||||
The bus is an in-memory adapter by default; because it is a port, it can be
|
||||
swapped for a durable broker. Emit from the orchestrator at each stage:
|
||||
|
||||
```python
|
||||
await self.bus.publish(ProgressEvent(job_id=jid, step="normalize",
|
||||
status="completed", message={"count": n}))
|
||||
```
|
||||
|
||||
## Streaming endpoint
|
||||
|
||||
```python
|
||||
async def stream_job_events(request, id):
|
||||
bus = request.app.state.resources.get("progress_bus")
|
||||
if bus is None:
|
||||
return {"events": []}
|
||||
|
||||
async def gen():
|
||||
sent = 0
|
||||
while True:
|
||||
events = bus.get_events(id)
|
||||
for e in events[sent:]:
|
||||
yield f"data: {e.model_dump_json(exclude_none=True)}\n\n"
|
||||
sent += 1
|
||||
await asyncio.sleep(0.5)
|
||||
|
||||
return StreamingResponse(gen(), media_type="text/event-stream")
|
||||
```
|
||||
|
||||
> **Known limitation.** The default in-memory bus is per-process. If the worker
|
||||
> and the API run as separate processes, events published by the worker are not
|
||||
> visible to the API. For production, back the bus with a shared broker. The
|
||||
> port boundary makes this a one-adapter change.
|
||||
87
standards/src/lib/08_human_in_loop.md
Normal file
87
standards/src/lib/08_human_in_loop.md
Normal file
@@ -0,0 +1,87 @@
|
||||
# Human-in-the-Loop
|
||||
|
||||
## Summary
|
||||
|
||||
Some inputs are ambiguous and must be resolved by a human before processing can
|
||||
continue. The blueprint models this as a **durable pause**: an exception is
|
||||
raised, a question is recorded, the job pauses, and later resumes.
|
||||
|
||||
---
|
||||
|
||||
## The mechanism
|
||||
|
||||
1. During processing, a stage raises a domain exception carrying the context
|
||||
needed to ask the question:
|
||||
|
||||
```python
|
||||
raise PauseForResolution(
|
||||
job_id=jid,
|
||||
reference="<stable key for this decision>",
|
||||
options=[...],
|
||||
context={...},
|
||||
)
|
||||
```
|
||||
|
||||
2. The orchestrator catches it, records a **question document**, and re-raises
|
||||
so the worker marks the job `paused`.
|
||||
3. A client lists pending questions and answers one:
|
||||
|
||||
```
|
||||
GET /jobs/{id}/questions -> list pending questions
|
||||
POST /questions/{qid}/resolve {choice} -> record the choice
|
||||
```
|
||||
|
||||
4. When the **last** pending question is resolved, the resolve endpoint flips
|
||||
the parent job back to `pending`.
|
||||
5. The worker picks it up again, rebuilds a **resolution map** from previously
|
||||
answered questions, and seeds it into the pipeline so the same decision is
|
||||
not asked twice.
|
||||
|
||||
## The question document
|
||||
|
||||
```python
|
||||
class Question(BaseDocument):
|
||||
job_id: str
|
||||
reference: str # stable key: stage|field|value
|
||||
status: str # pending | resolved
|
||||
options: list[dict]
|
||||
context: dict
|
||||
chosen: dict | None = None
|
||||
resolved_at: str | None = None
|
||||
```
|
||||
|
||||
## Resume map
|
||||
|
||||
The worker turns answered questions into a lookup keyed by `reference`, then
|
||||
injects it into the pipeline so that stage can skip re-raising:
|
||||
|
||||
```python
|
||||
resolved = {q.reference: q.chosen for q in await svc.search(
|
||||
filter={"job_id": jid, "status": "resolved"})}
|
||||
|
||||
pipeline = build(root, instances={"pause": handler}, config={"resolved": resolved})
|
||||
```
|
||||
|
||||
## Resolve endpoint
|
||||
|
||||
```python
|
||||
async def resolve_question(request, id, body):
|
||||
svc = request.app.state.resources["question_svc"]
|
||||
q = await svc.read(id)
|
||||
await svc.update(id, {"chosen": body["chosen"], "status": "resolved",
|
||||
"resolved_at": now()})
|
||||
|
||||
pending = await svc.search(filter={"job_id": q.job_id, "status": "pending"},
|
||||
limit=1)
|
||||
if not pending:
|
||||
jobs = request.app.state.resources["job_svc"]
|
||||
job = await jobs.read(q.job_id)
|
||||
if job and job.status == "paused":
|
||||
await jobs.update_status(q.job_id, "pending")
|
||||
return q
|
||||
```
|
||||
|
||||
## CLI parity
|
||||
|
||||
The same flow is available through scripts/CLI: `list`, `get`, `create`,
|
||||
`questions`, `resolve`, `retry`.
|
||||
84
standards/src/lib/09_infra_adaptors.md
Normal file
84
standards/src/lib/09_infra_adaptors.md
Normal file
@@ -0,0 +1,84 @@
|
||||
# Infra Adapters
|
||||
|
||||
## Summary
|
||||
|
||||
All external systems are reached through adapters in `infra/`. Prefer an
|
||||
existing internal library over writing a new adapter.
|
||||
|
||||
---
|
||||
|
||||
## Database (MongoDB)
|
||||
|
||||
- A single connection manager owns the client lifecycle.
|
||||
`setup_database(uri, db_name)` / `shutdown_database()`.
|
||||
- A repository **factory** selects the repository flavour:
|
||||
|
||||
```python
|
||||
repo_factory(collection, document, engine=None,
|
||||
populate_rules=None, cache=None)
|
||||
```
|
||||
|
||||
- base repository — plain CRUD;
|
||||
- populating repository — resolves referenced documents (foreign keys);
|
||||
- cached repository — an LRU/TTL cache in front of reads.
|
||||
|
||||
- Referenced documents are described declaratively:
|
||||
|
||||
```python
|
||||
PopulateRule(field="owner", collection="users",
|
||||
nested_rules=[], max_depth=2, projection=None)
|
||||
```
|
||||
|
||||
- Transactions use a session context manager provided by the library.
|
||||
|
||||
**Library:** the Mongo operations library (see registry). Do not write raw
|
||||
Motor calls in the domain.
|
||||
|
||||
## HTTP response cache
|
||||
|
||||
A generic GET cache sits behind a decorator.
|
||||
|
||||
```python
|
||||
@cache_response(ttl=60)
|
||||
async def list_items(request): ...
|
||||
```
|
||||
|
||||
- Keyed on path + query + user.
|
||||
- No-op when no store is wired.
|
||||
- `invalidate(*resources)` bumps a generation counter for write-triggered
|
||||
invalidation.
|
||||
|
||||
## Event / progress bus
|
||||
|
||||
A port with an in-memory default adapter (`publish`, `get_events`). Swap for a
|
||||
durable broker in production (see Worker & Progress).
|
||||
|
||||
## External clients (ingestion)
|
||||
|
||||
External fetchers (files, mail, HTTP) are adapters that wrap ingestion
|
||||
libraries. Construct them in the composition root and inject into
|
||||
orchestrators.
|
||||
|
||||
**Libraries:** the ingestion library (formats) and the mail library (email).
|
||||
|
||||
## LLM clients
|
||||
|
||||
When a pipeline calls a model provider, route through a pool adaptor that
|
||||
handles provider selection, retries, and client lifecycle:
|
||||
|
||||
```python
|
||||
pool = setup_async_pool(provider=settings.llm_provider)
|
||||
result = await pool.complete(model="...", messages=[...])
|
||||
```
|
||||
|
||||
Providers are selected by `LLM_PROVIDER` and configured by environment.
|
||||
|
||||
## Key-value store
|
||||
|
||||
For report/snapshot storage, provide a pluggable store port with a Redis
|
||||
adapter and an in-memory fallback.
|
||||
|
||||
## Rule
|
||||
|
||||
If a capability already exists as a library (DB, DI, ingestion, mail, auth,
|
||||
docs), **use it**. Writing a private copy of a library is a defect.
|
||||
57
standards/src/lib/12_genericity_matrix.md
Normal file
57
standards/src/lib/12_genericity_matrix.md
Normal file
@@ -0,0 +1,57 @@
|
||||
# Genericity Matrix
|
||||
|
||||
## Summary
|
||||
|
||||
This page states precisely what is **reusable as-is**, what is a **template to
|
||||
adapt**, and what is **app-specific and must be authored**. Use it to scope a
|
||||
new build.
|
||||
|
||||
---
|
||||
|
||||
## Reusable as-is (do not rewrite)
|
||||
|
||||
| Area | Artifact |
|
||||
|---|---|
|
||||
| Layering & dependency rule | the package layout and import rules |
|
||||
| Composition root | the `init.py` shape, idempotency, `get_svc` overloading |
|
||||
| Web edge | `OpenAPIFirstApp` bootstrap, handler-name = `operationId`, thin handlers |
|
||||
| CRUD slice | the four-file pattern (model / ports-db / ports-service / converters) |
|
||||
| Generic CRUD service | the base service adaptor |
|
||||
| Repository factory | base / populating / cached selection |
|
||||
| Worker | the poll loop, backoff, retry, status machine |
|
||||
| Progress bus | the bus port + in-memory adapter + streaming endpoint |
|
||||
| Human-in-the-loop | the durable pause + question document + resolve endpoint + resume map |
|
||||
| Infra adaptors | HTTP cache decorator, KV store port, LLM pool scaffold |
|
||||
| Testing | integration harness with fakes, fixture-parity pattern |
|
||||
| Docs | GSDFC + generated stubs + MCP bundle pipeline |
|
||||
|
||||
## Template to adapt (copy, then change names)
|
||||
|
||||
| Artifact | Change |
|
||||
|---|---|
|
||||
| `pyproject.toml` | project name, dependencies |
|
||||
| `Dockerfile` / `docker-compose.yaml` | service name, ports, services |
|
||||
| `.env.example` | connection strings, provider keys |
|
||||
| `openapi.yaml` | every path/schema/operationId |
|
||||
| `scripts/` | seed and operator tasks |
|
||||
|
||||
## App-specific (must be authored)
|
||||
|
||||
| Artifact | Why |
|
||||
|---|---|
|
||||
| Resource models | the business shape |
|
||||
| Repository populate/cache rules | the relationships |
|
||||
| Service business logic | domain behaviour |
|
||||
| Pipeline families (if any) | the transformation graph |
|
||||
| Pattern/rule resources | data-driven policy |
|
||||
| Fixtures & golden outputs | test data |
|
||||
|
||||
## The two axes
|
||||
|
||||
Everything app-specific reduces to **two axes**:
|
||||
|
||||
1. **Resource model** — the set of resources, their fields, relationships, and
|
||||
the OpenAPI contract.
|
||||
2. **Pipeline family** — only if inputs require multi-step transformation.
|
||||
|
||||
If you can specify those two axes, the rest of the app follows mechanically.
|
||||
54
standards/src/lib/13_library_registry.md
Normal file
54
standards/src/lib/13_library_registry.md
Normal file
@@ -0,0 +1,54 @@
|
||||
# Library Registry
|
||||
|
||||
## Summary
|
||||
|
||||
The Aetos ecosystem provides internal libraries for the shared concerns of
|
||||
every application. **Use them.** Writing a private copy of any capability below
|
||||
is a defect.
|
||||
|
||||
Each library also exposes its own documentation as an MCP server. An AI agent
|
||||
can read a library's full API by connecting to its MCP and reading
|
||||
`docs://index`, `docs://nav`, and `docs://modules/{module}`.
|
||||
|
||||
---
|
||||
|
||||
## Catalog
|
||||
|
||||
| Library | Import | Purpose | Use instead of | MCP |
|
||||
|---|---|---|---|---|
|
||||
| DAG Pipe | `dagpipe` | Deterministic state-transformation DAG engine | hand-rolled orchestration/topological ordering | `:8006` |
|
||||
| Hexa | `hexa` | Declarative DI / composition-root tooling | manual wiring, hand-written factories | *(see note)* |
|
||||
| Mongo Ops | `mongo_ops` | Async MongoDB repositories, cache, populate, transactions | raw Motor CRUD and connection lifecycle | `:8007` |
|
||||
| Omniread | `omniread` | Format-agnostic fetch + parse (HTML, PDF, CSV, XLSX) | custom scrapers/parsers | `:8003` |
|
||||
| OpenAPI First | `openapi_first` | Spec-first FastAPI bootstrap, client, codegen | decorator routers and hand-written clients | `:8004` |
|
||||
| Mail Intake | `mail_intake` | Provider-agnostic read-only email ingestion | Gmail OAuth + MIME parsing | `:8002` |
|
||||
| Py JWT | `jwtlib` | JWT auth primitives (register/login/introspect) | password hashing + token code | `:8005` |
|
||||
| Doc Forge | `docforge` | Docstring compiler → MkDocs + MCP docs | custom doc generators | `:8001` |
|
||||
| Auth Server | *(service)* | Central identity service (register/login/me/introspect) | building auth endpoints yourself | — |
|
||||
|
||||
## How to choose
|
||||
|
||||
| You need to… | Use |
|
||||
|---|---|
|
||||
| persist and query documents | Mongo Ops |
|
||||
| compose dependencies | Hexa |
|
||||
| run a multi-step transformation | DAG Pipe |
|
||||
| read/parse an arbitrary input format | Omniread |
|
||||
| expose a contract-first HTTP API | OpenAPI First |
|
||||
| ingest email | Mail Intake |
|
||||
| authenticate users / verify tokens | Py JWT (+ Auth Server) |
|
||||
| generate documentation | Doc Forge |
|
||||
|
||||
## Discovery
|
||||
|
||||
A single **standards MCP** exposes `find_library(capability)`, which returns the
|
||||
matching library, its import name, its "use instead of" guidance, and its MCP
|
||||
endpoint. Query it before writing any infrastructure.
|
||||
|
||||
## Notes
|
||||
|
||||
- **Hexa** ships an MCP bundle but is not yet assigned a dedicated port in the
|
||||
hub config; it is reachable via the standards registry entry and will be
|
||||
wired to a port.
|
||||
- Library MCP servers expose **documentation resources** (not executable tools):
|
||||
they tell an agent the available API surface so it does not invent code.
|
||||
20
standards/src/lib/index.md
Normal file
20
standards/src/lib/index.md
Normal file
@@ -0,0 +1,20 @@
|
||||
# Standards Reference
|
||||
|
||||
The reference half of the Standards description: the enforceable rules,
|
||||
patterns, and the internal library catalog that agents consult while coding.
|
||||
|
||||
- **Layering Rules** — the strict request → domain → repository layering and
|
||||
the rules that enforce it.
|
||||
- **Composition Root** — a single composition point wiring the whole app.
|
||||
- **Spec-First Routing** — the OpenAPI contract as the routing source of truth.
|
||||
- **CRUD Template** — the standardized generic resource endpoint scaffold.
|
||||
- **Pipeline Scaffold** — the deterministic pipeline skeleton for multi-stage
|
||||
jobs.
|
||||
- **Worker & Progress** — queued work, progress reporting, and visibility.
|
||||
- **Human-in-the-Loop** — guardrails and approvals inside automations.
|
||||
- **Infra Adapters** — how infrastructure connects without leaking into domain.
|
||||
- **Genericity Matrix** — what is generic vs app-specific.
|
||||
- **Library Registry** — the catalog of internal libraries behind all of it.
|
||||
|
||||
Read the **Wiki** for the philosophy and rationale; these pages are the
|
||||
reference that mirrors the `standards` MCP bundle.
|
||||
Reference in New Issue
Block a user