1 line
48 KiB
JSON
1 line
48 KiB
JSON
{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"\ud83e\udde9 Khata App \u2014 Expenses & Cashflow Service","text":"<p><code>khata-app</code> is the OpenAPI-first backend for tracking expenses and cashflow. It ingests bank statements (PDF, CSV, XLSX, email, LlamaParse), runs rule-driven enrichment to identify payees and tags, and exposes everything through a spec-driven FastAPI service on MongoDB.</p> <p>Doc model: this wiki is written for humans \u2014 how-to guides, examples, and testing recipes. The authoritative API contracts live in the code (docstrings) and the machine-readable bundle under <code>docs/mcp/</code>.</p>"},{"location":"#key-features","title":"\ud83d\ude80 Key Features","text":"<ul> <li>\ud83d\uddc2 OpenAPI-first routing \u2014 the <code>openapi.yaml</code> spec drives every route via <code>opendapi-first</code>'s <code>OpenAPIFirstApp</code>; handlers are thin and matched by <code>operationId</code></li> <li>\ud83c\udfdb Strict three-layer architecture \u2014 interface \u2192 domain \u2192 infra, one-way data flow, zero backflow</li> <li>\ud83d\udce5 Multi-source extraction \u2014 PDF heuristics, CSV/XLSX tabular parsing, email ingestion, and llama-cloud OCR via the hexa pipeline families</li> <li>\ud83e\udde0 Rule-driven enrichment \u2014 entity resolution and tag assignment from YAML pattern registries, wired through <code>dagpipe</code></li> <li>\ud83d\udd04 Fetch request pipeline \u2014 poll \u2192 extract \u2192 enrich \u2192 persist, coordurally by the worker with SSE progress events and ambiguity pauses</li> <li>\ud83d\uddc3 MongoDB via mongo-ops \u2014 generic repos with in-memory caching and foreign-key population (<code>x-fk</code>)</li> <li>\u26a1 Redis-backed caching \u2014 HTTP cache and repo reads with TTL control</li> </ul>"},{"location":"#installation","title":"\ud83d\udce6 Installation","text":"<pre><code># Local development\npip install -e \".[dev]\"\n\n# App dependencies (from pyproject.toml)\npip install -e .\n</code></pre> <p>Requires Python \u2265 3.9 (development venv runs 3.13) and a running MongoDB + optional Redis.</p>"},{"location":"#documentation-structure","title":"\ud83d\udcc1 Documentation Structure","text":"Section Description Overview What Khata is and how it fits the stack Architecture Layers, data flow, and lifecycle rules Components The domain, interfaces, and infra building blocks Domain Resources Users, entities, accounts, tags, expenses, uploads Enrichment Entity resolution and tag assignment pipelines Extraction Pipelines PDF/CSV/XLSX/email/llama extraction families Fetch Request Workflow Worker, SSE, CLI, orchestration Reports & Caching Reports service, HTTP cache, Redis Running The Service Boot, config, seed scripts Development Environment, testing, docstrings, docs builds"},{"location":"#related-resources","title":"\ud83d\udd17 Related Resources","text":"<ul> <li>Source repo: <code>git@git.aetoskia.com:apps/khata-app.git</code></li> <li>Library reference (lib): built into <code>site/lib/</code> from GSDFC docstrings</li> <li>MCP bundles: <code>docs/mcp/{domain,infra,interfaces,pipelines}/</code> \u2014 machine-readable API reference</li> <li>API reference (OpenAPI): built into <code>site/api/</code> from <code>openapi.yaml</code></li> </ul> <p>\u00a9 Aetoskia Internal \u2014 <code>khata-app</code> 0.1.0</p>"},{"location":"01_overview/","title":"\ud83e\udded Overview","text":"<p>Khata is an internal expenses and cashflow service. It manages users, entities, accounts, tags, expenses, uploads, and fetch requests on MongoDB, ingesting bank statements from several file formats and turning them into normalized expense records enriched with payee entities and tags.</p>"},{"location":"01_overview/#what-it-does","title":"\ud83c\udfaf What It Does","text":"<ul> <li>Statement ingestion \u2014 a fetch request declares a source (PDF file, email, CSV/XLSX upload, or llama-cloud document); a background worker executes the extraction, enrichment, and persistence pipeline.</li> <li>Normalized expenses \u2014 expenses carry <code>amount</code>, <code>balance</code>, a <code>kind</code> (debit/credit/transfer), <code>mcc</code>, <code>memo</code>, <code>date</code>, an optional <code>tag</code>, and an optional <code>entity</code> (the payee).</li> <li>Enrichment \u2014 rule registries map raw payee/memo patterns to canonical entities and tags; pipelines run depth-first through <code>dagpipe</code>.</li> <li>Tag & entity reconciliation \u2014 entities, tags, and accounts are created on demand and linked by name-based resolution.</li> <li>Progress visibility \u2014 the pipeline emits <code>ProgressEvent</code>s over SSE; ambiguity pauses are stored for later resolution.</li> </ul>"},{"location":"01_overview/#where-it-lives","title":"\ud83e\udde9 Where It Lives","text":"<p>Khata is the read/write service in the wider stack: other services query expenses and tags downstream, while Khata itself pulls documents and external OCR. The domain knows only its own models and ports; all adapters (Mongo, Redis, HTTP, OpenAI-compatible LLM, email, LlamaParse) live in <code>infra</code>.</p>"},{"location":"01_overview/#notable-trade-offs","title":"\ud83d\udccc Notable Trade-offs","text":"Decision Rationale OpenAPI-first routing Single source of truth for the HTTP surface; handlers are thin no plain <code>__init__</code> factories Entrypoints wire the object graph explicitly in <code>init.py</code> In-memory LRU + Redis Hot entity/account lookups avoid Mongo round-trips during enrichment <code>x-fk</code> cross-collection population Repos resolve referenced ids with explicit filters instead of Mongo joins YAML pattern registries Rule changes require no code deploy"},{"location":"01_overview/#roadmap-notes","title":"\ud83d\uddfa Roadmap Notes","text":"<ul> <li>The ambiguity resolver currently returns <code>None</code> in <code>worker._maybe_create_resolver</code> \u2014 LLM-assisted resolution is stubbed behind <code>LLM_API_KEY</code>.</li> <li><code>infra/llm</code> (OpenAI / OpenRouter pools, prompt policies) is the substrate for future resolver work.</li> <li>The XLSX/CSV tabular families cover Axis and ICICI layouts first.</li> </ul>"},{"location":"01_overview/#related","title":"\ud83d\udd17 Related","text":"<p>Architecture \u00b7 Components \u00b7 Running The Service \u00b7 Development</p>"},{"location":"02_architecture/","title":"\ud83c\udfdb Architecture","text":"<p>Khata follows a strict three-layer architecture with one-way data flow and no backflow between layers.</p>"},{"location":"02_architecture/#layers","title":"\ud83e\uddf1 Layers","text":"Layer Package(s) Responsibility interface <code>interfaces/</code> HTTP routes (OpenAPI handlers), CLI, SSE bus wiring, DTO mapping domain <code>domain/</code> Models, ports (abstract contract), services, orchestration, enrichment pipelines infra <code>infra/</code> Mongo repos, Redis cache, HTTP/email/Llama fetchers, OpenAI-compatible LLM pools <p>The <code>pipelines/</code> package holds the hexa extraction runtime \u2014 the stateful, stage-driven extractors for PDF, CSV, XLSX, email, and llama sources. It sits outside the three-layer packages in <code>pyproject.toml</code> (<code>packages = domain*, infra*, interfaces*</code>) but is first-class code.</p>"},{"location":"02_architecture/#data-flow","title":"\ud83d\udd00 Data Flow","text":"<pre><code> \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 INTERFACES \u2502\n \u2502 openapi handler \u00b7 sse bus \u00b7 cli \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 DTOs / commands\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 DOMAIN \u2502\n \u2502 models \u00b7 ports \u00b7 services (rules) \u2502\n \u2502 fetch_pipeline orchestrator \u2502\n \u2502 enrichment (entity / tags) \u2190 dagpipe \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 ports (abstract)\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 INFRA \u2502\n \u2502 mongo repos \u00b7 redis \u00b7 http \u00b7 llm pools \u2502\n \u2502 pdf/email/xlsx fetchers \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u25bc \u25bc\n MongoDB Redis / external\n</code></pre> <p>Interfaces speak DTOs/commands; the domain owns rules and orchestration; infra implements ports and talks to the outside world. No infra import may appear in the domain, and no domain import may appear in infra.</p>"},{"location":"02_architecture/#wiring","title":"\ud83e\udde9 Wiring","text":"<p>Entrypoints (<code>main.py</code>, <code>worker.py</code>, <code>init.py</code>) build the object graph explicitly:</p> <ul> <li><code>init()</code> constructs the Mongo engine, repos, services, fetchers, cache store, the SSE bus, and a <code>PipelineOrchestrator</code> wiring, then returns a dict of named services.</li> <li>Handler lookups go through <code>get_svc(name)</code> (e.g. <code>\"fetch_request_svc\"</code>, <code>\"expense_svc\"</code>, <code>\"ambiguity_svc\"</code>).</li> <li><code>main.py</code> registers the OpenAPI app from <code>openapi.yaml</code>; <code>worker.py</code> polls for pending fetch requests; <code>interfaces/fetch_request/cli</code> drives the same services from the terminal.</li> </ul>"},{"location":"02_architecture/#lifecycle-rules","title":"\ud83d\udd01 Lifecycle Rules","text":"<ul> <li>One-way ports \u2014 repos implement (<code>implements</code>-style) the port interfaces declared in the domain; the domain never knows a concrete repo.</li> <li>Canonical ids \u2014 documents are keyed by MongoDB <code>ObjectId</code>; cross references are populated with explicit filter-based <code>x-fk</code> resolution, never Mongo joins.</li> <li>Caching discipline \u2014 repo reads go through an in-memory LRU (and an optional Redis mirror for HTTP); invalidation happens on writes via <code>_invalidate_cache</code>.</li> <li>No god init \u2014 the domain has no plain <code>__init__</code> modules with factories; <code>facades.py</code> and <code>init.py</code> hold the composition.</li> </ul>"},{"location":"02_architecture/#related","title":"\ud83d\udd17 Related","text":"<p>Components \u00b7 Domain Resources \u00b7 Extraction Pipelines \u00b7 Running The Service \u00b7 Development</p>"},{"location":"03_components/","title":"\ud83e\udde9 Components","text":"<p>A map of the codebase's major pieces and where they live.</p>"},{"location":"03_components/#packages","title":"\ud83e\udded Packages","text":"Path Contents <code>interfaces/openapi/</code> OpenAPI-first routing: app bootstrap, handlers, DTO schemas, middlewares <code>interfaces/fetch_request/cli/</code> Argparse CLI over the fetch-request service (<code>list</code>, <code>get</code>, <code>create-file</code>, <code>create-email</code>, <code>ambiguities</code>, <code>resolve</code>, <code>retry</code>) <code>interfaces/sse/</code> SSE <code>ProgressEventBus</code> and subscription endpoints <code>interfaces/report/</code> Report generation endpoints and query models <code>domain/</code> The three resource groups + enrichment + fetch pipeline (see below) <code>domain/enrichment/</code> Entity identity/resolution + tag assignment; <code>dagpipe</code>-based pipelines <code>domain/fetch_pipeline/</code> Fetch-request orchestration: pipeline, ambiguity, orchestrator, progress events <code>domain/fetch_request/</code> FetchRequest model, file/email sources, bank enum, service <code>infra/repos/</code> Mongo repositories (mongo-ops) for every domain model <code>infra/adaptors/</code> Cache store, gitignore-safe secrets, fetch helpers <code>infra/cache/</code> LRU maps, <code>CacheableDict</code>, Redis cache, TTL decorators <code>infra/http_cache/</code> HTTP client cache layer with TTL + max-entries eviction <code>infra/email/</code> IMAP email fetcher and parsed-mailbox traversal <code>infra/llm/</code> OpenAI / OpenRouter provider pools, policy prompts, DTOs <code>infra/adapters/sse.py</code> SSE response adapter for progress event streaming <code>pipelines/extraction/</code> Hexa runtime: pdf, csv, xlsx, email, llama extraction families <code>pipelines/enrichment/</code> Pipeline node construction for entity/tag enrichment stages"},{"location":"03_components/#domain-resource-groups","title":"\ud83c\udfd7 Domain Resource Groups","text":"Group Models include Purpose users <code>User</code>, <code>UserActivity</code> Actors; ownership of expenses/fetch requests entities <code>Entity</code>, <code>EntityAccount</code> Canonical payees with brands/branches accounts <code>Account</code>, <code>AccountType</code> Bank accounts owning expenses and statements expenses <code>Expense</code>, <code>ExpenseKind</code>, <code>ExpenseSource</code>, <code>ExpenseTag</code> The core normalized record tags <code>Tag</code>, <code>TagColor</code> Categorization labels with display colors uploads <code>Upload</code> Raw file/email ingest records fetch_request <code>FetchRequest</code>, <code>FileSource</code>, <code>EmailSource</code>, <code>Bank</code> Declares a statement ingestion job report <code>Report</code>, report models Aggregated outputs"},{"location":"03_components/#extraction-families-pipelinesextraction","title":"\u2699\ufe0f Extraction Families (<code>pipelines/extraction/</code>)","text":"Family Inputs Notes <code>pdf</code> Bank statement PDFs Heuristic, balance-driven reconciliation <code>csv</code> CSV exports Axis/ICICI layouts first <code>xlsx</code> Excel exports Tabular header+row parsing via <code>openpyxl</code> <code>email</code> IMAP mailboxes + attachments <code>EmailFetchingParadigm</code> traversal <code>llama</code> Documents via LlamaParse OCR + markdown distribution over pages <p>All families share the hexa runtime: <code>BaseExtractionPipeline</code>, <code>Fetcher</code>, <code>ExtractionStage</code>, and the <code>ambiguous</code> flag \u2014 <code>get_families_to_enrich</code> returns families that should proceed to enrichment.</p>"},{"location":"03_components/#related","title":"\ud83d\udd17 Related","text":"<p>Architecture \u00b7 Domain Resources \u00b7 Extraction Pipelines \u00b7 Reports & Caching</p>"},{"location":"04_domain_resources/","title":"\ud83d\udce6 Domain Resources","text":"<p>Each resource group is a domain package with <code>models</code>, <code>ports</code> (abstract repositories/services), and a <code>service</code> implementation \u2014 a facade over the repo. Everything is defined by GSDFC docstrings and mirrored as <code>.pyi</code> stubs.</p>"},{"location":"04_domain_resources/#users","title":"\ud83d\udc64 Users","text":"<ul> <li><code>User</code> \u2014 id, name, email, phone, password hash, avatar.</li> <li><code>UserActivity</code> \u2014 audit trail of user-driven actions.</li> <li>Service: <code>UserService</code> (create, read, update, delete, activity log). Routes: <code>GET/POST /users</code>, <code>GET/PATCH/DELETE /users/{id}</code>, activity listing under <code>/users/{id}/activities</code>.</li> </ul>"},{"location":"04_domain_resources/#entities","title":"\ud83c\udfe2 Entities","text":"<ul> <li><code>Entity</code> \u2014 canonical payee: name, brands, branch, aliases, classification, type.</li> <li><code>EntityAccount</code> \u2014 an account known to belong to an entity.</li> <li>Service: <code>EntityService</code> \u2014 entity creation, name-based lookup, account linking. Routes: <code>GET/POST /entities</code>, <code>GET/PATCH /entities/{id}</code>, <code>POST /entities/{id}/accounts</code>.</li> </ul>"},{"location":"04_domain_resources/#accounts","title":"\ud83c\udfe6 Accounts","text":"<ul> <li><code>Account</code> \u2014 bank account: name, account number, bank, currency, account type, iban, linked entity.</li> <li><code>AccountType</code> \u2014 savings / current / credit etc.</li> <li>Service: <code>AccountService</code> \u2014 create, read, update, delete, balance helpers. Routes: <code>GET/POST /accounts</code>, <code>GET/PATCH/DELETE /accounts/{id}</code>.</li> </ul>"},{"location":"04_domain_resources/#expenses","title":"\ud83e\uddfe Expenses","text":"<ul> <li><code>Expense</code> \u2014 the core record: amount, balance, kind (debit/credit/transfer), mcc, memo, date, tag, entity, source.</li> <li><code>ExpenseTag</code> \u2014 tag membership on an expense; <code>ExpenseSource</code> \u2014 provenance (upload/fetch-request).</li> <li>Service: <code>ExpenseService</code> \u2014 CRUD plus tag assignment and entity resolution. Routes: <code>GET/POST /expenses</code>, <code>GET/PATCH/DELETE /expenses/{id}</code>, tag/entity mutation sub-routes.</li> </ul>"},{"location":"04_domain_resources/#tags","title":"\ud83c\udff7 Tags","text":"<ul> <li><code>Tag</code> \u2014 name, color, optional description.</li> <li><code>TagColor</code> \u2014 palette enum for display.</li> <li>Service: <code>TagService</code> \u2014 CRUD. Routes: <code>GET/POST /tags</code>, <code>GET/PATCH/DELETE /tags/{id}</code>.</li> </ul>"},{"location":"04_domain_resources/#uploads","title":"\ud83d\udce4 Uploads","text":"<ul> <li><code>Upload</code> \u2014 records a raw statement upload (file or email) with status.</li> <li>Service: <code>UploadService</code> \u2014 create, read, update status, list. Routes under <code>/uploads</code>.</li> </ul>"},{"location":"04_domain_resources/#fetch-request","title":"\ud83d\udce5 Fetch Request","text":"<ul> <li><code>FetchRequest</code> \u2014 a statement ingestion job: user, account, source (file/email), date range, status, retry count.</li> <li><code>FileSource</code> \u2014 path + optional bank; <code>EmailSource</code> \u2014 mailbox + filters; <code>Bank</code> \u2014 enum of supported banks.</li> <li>Service: <code>FetchRequestService</code> \u2014 CRUD + status transitions (<code>update_status</code>). Routes: <code>GET/POST /fetch-requests</code>, <code>GET/PATCH /fetch-requests/{id}</code>, <code>POST /fetch-requests/{id}/retry</code>.</li> </ul>"},{"location":"04_domain_resources/#repositories-ports","title":"\ud83d\udd01 Repositories (Ports)","text":"<p>Every group declares abstract repo ports (<code>Ports</code>: <code>db.py</code>, <code>service.py</code>) in the domain and implements them in <code>infra/repos/</code> with mongo-ops. Repos support:</p> <ul> <li>filter-based queries \u2014 <code>search(filter={...}, limit=...)</code></li> <li>explicit FK population \u2014 <code>x-fk</code> resolves referenced ids via exact filters</li> <li>cache invalidation \u2014 writes clear the in-memory LRU</li> </ul> <p>Shared wiring lives in <code>facades.py</code> (domain) and <code>repositories.py</code> (infra). The service registry keys used in <code>init.py</code> are the string names passed to <code>get_svc(...)</code>.</p>"},{"location":"04_domain_resources/#related","title":"\ud83d\udd17 Related","text":"<p>Components \u00b7 Enrichment \u00b7 Reports & Caching</p>"},{"location":"05_enrichment/","title":"\ud83e\udde0 Enrichment","text":"<p>Enrichment turns raw payee/memo text into canonical entities and tags using YAML pattern registries, executed through <code>dagpipe</code> pipelines.</p>"},{"location":"05_enrichment/#entity-resolution","title":"\ud83c\udfe2 Entity Resolution","text":"<p>Lives under <code>domain/enrichment/entity/</code>:</p> <ul> <li><code>identity.py</code> \u2014 normalizes and fingerprints payee strings for matching.</li> <li><code>resolver.py</code> \u2014 <code>EntityResolver</code>: resolves a raw payee to an <code>entity_id</code> via name lookup + fingerprint + account linkage.</li> <li><code>pipeline.py</code> \u2014 the enrichment pipeline node that uses the resolver on expense records.</li> <li><code>utils.py</code> \u2014 helpers for entity display names and brand collapsing.</li> </ul> <p>The pipeline is expressed as a <code>dagpipe.Pipeline</code>; the stub <code>pipeline.pyi</code> type-checks against <code>dagpipe</code> directly (see <code>domain/enrichment/entity/pipeline.pyi</code>).</p>"},{"location":"05_enrichment/#tag-assignment","title":"\ud83c\udff7 Tag Assignment","text":"<p>Lives under <code>domain/enrichment/tags/</code>:</p> <ul> <li><code>flags.py</code> \u2014 tag candidate flags (priority/confidence hints) computed from memo and MCC.</li> <li><code>pipeline.py</code> \u2014 a <code>dagpipe.Pipeline</code> that assigns a <code>tag_id</code> to expenses based on matched rules.</li> <li>Both are seeded with YAML pattern registries (regex + literal patterns per tag) under the enrichment rules data.</li> </ul>"},{"location":"05_enrichment/#how-rules-are-wired","title":"\ud83d\udd04 How Rules Are Wired","text":"<ol> <li>The extraction stage yields expenses with raw <code>memo</code>/payee fields.</li> <li>The orchestrator (<code>PipelineOrchestrator</code> in <code>domain/fetch_pipeline/</code>) chains extraction \u2192 enrichment.</li> <li>Enrichment nodes receive entity/tag services and the http cache store for external lookups.</li> <li>Resolved <code>entity_id</code> / <code>tag_id</code> are written back onto the expense and persisted.</li> </ol>"},{"location":"05_enrichment/#testing","title":"\ud83e\uddea Testing","text":"<ul> <li>Unit tests in <code>tests/domain/enrichment/</code> cover fingerprint/identity edge cases, tag flag computation, and pipeline wiring.</li> <li>Golden fixtures for memo\u2192entity and memo\u2192tag mappings keep the registries honest.</li> <li>Run them: <code>.venv\\Scripts\\pytest.exe tests\\domain\\enrichment</code></li> </ul>"},{"location":"05_enrichment/#related","title":"\ud83d\udd17 Related","text":"<p>Domain Resources \u00b7 Fetch Request Workflow \u00b7 Development</p>"},{"location":"06_extraction_pipelines/","title":"\ud83d\udce5 Extraction Pipelines","text":"<p><code>pipelines/extraction/</code> implements the hexa runtime: an extensible, stateful pipeline that fetches a source document and reduces it to normalized expense records, flagging anything ambiguous.</p>"},{"location":"06_extraction_pipelines/#common-runtime","title":"\ud83c\udfd7 Common Runtime","text":"Concept Where Purpose <code>BaseExtractionPipeline</code> shared Template method: fetch \u2192 stage \u2192 emit; tracks ambiguous state <code>Fetcher</code> shared Loads raw content (file bytes, emails, llama docs) <code>ExtractionStage</code> shared A single reduction step (headers, rows, reconciliation) <code>PauseForResolution</code> shared Raised when numbers can't be reconciled; workflow pauses <code>get_families_to_enrich</code> shared Families that should continue to enrichment <p>Every family registers a factory so <code>pipelines/enrichment/</code> can build nodes from config at runtime.</p>"},{"location":"06_extraction_pipelines/#extraction-families","title":"\ud83d\udcc2 Extraction Families","text":"Family Payload Stages (typical) <code>pdf</code> Bank statement PDFs page split \u2192 text extraction \u2192 balance reconciliation <code>csv</code> CSV exports header detect \u2192 row mapping \u2192 dtype normalization <code>xlsx</code> Excel exports (<code>openpyxl</code>) sheet select \u2192 header detect \u2192 row mapping (Axis/ICICI first) <code>email</code> IMAP mailbox + attachments traversal paradigm \u2192 attachment dispatch to family <code>llama</code> LlamaParse documents page distribution \u2192 markdown parse \u2192 amount extraction"},{"location":"06_extraction_pipelines/#cross-cutting-utilities-pipelinesextractionshared","title":"Cross-cutting utilities (<code>pipelines/extraction/shared/</code>)","text":"<ul> <li><code>tabular.py</code> \u2014 generic tabular parsing helpers (used by CSV/XLSX families)</li> <li><code>files.py</code>, <code>utils.py</code> \u2014 content sniffing, date/amount normalization</li> <li><code>ambiguity.py</code> \u2014 <code>PauseForResolution</code> + <code>AmbiguityCandidate</code> helpers</li> <li><code>extractor.py</code> \u2014 the <code>BaseExtractionPipeline</code> template and registry</li> </ul>"},{"location":"06_extraction_pipelines/#ambiguity-reconciliation","title":"\u26a0\ufe0f Ambiguity & Reconciliation","text":"<p>Amount/balance reconciliation is the safety net: if extracted amounts don't balance against the statement, the pipeline raises <code>PauseForResolution</code>, the fetch request flips to paused, and the candidate is stored. A human (or later the LLM resolver) chooses the correct tuple; the worker resets the retry counter on pause.</p>"},{"location":"06_extraction_pipelines/#lifecycle-inside-a-run","title":"\ud83d\udd04 Lifecycle Inside a Run","text":"<ol> <li>Worker polls pending fetch requests (<code>status == \"pending\"</code>).</li> <li><code>PipelineOrchestrator.run_pipeline(fr_id)</code>:</li> <li>picks the family from <code>fr.source.type</code> + bank,</li> <li>runs extraction stages to raw <code>Expense</code> candidates,</li> <li>keeps unresolved candidates as ambiguities,</li> <li>forwards the clean set to enrichment,</li> <li>Enrichment assigns entity/tag; expenses are persisted; SSE progress events stream the whole way.</li> </ol>"},{"location":"06_extraction_pipelines/#related","title":"\ud83d\udd17 Related","text":"<p>Components \u00b7 Fetch Request Workflow \u00b7 Development</p>"},{"location":"07_fetch_request_workflow/","title":"\ud83d\udd04 Fetch Request Workflow","text":"<p>A fetch request is a statement ingestion job. This is the end-to-end journey and how to drive it.</p>"},{"location":"07_fetch_request_workflow/#creating-a-fetch-request","title":"\ud83d\udce5 Creating a Fetch Request","text":"<p>Via API:</p> <pre><code># File source\ncurl -X POST http://localhost:8000/fetch-requests \\\n -H \"Content-Type: application/json\" \\\n -d '{\"user_id\":\"...\",\"account_id\":\"...\",\"source\":{\"type\":\"file\",\"path\":\"/var/statements/axis.pdf\",\"bank\":\"axis\"},\"start_date\":\"2025-04-01\",\"end_date\":\"2025-04-30\"}'\n</code></pre> <p>Via CLI:</p> <pre><code># from repo root, with the khata venv active\n.venv\\Scripts\\python.exe -m interfaces.fetch_request.cli create-file --account \"Axis Savings\" --path /var/statements/axis.pdf --bank axis\n.venv\\Scripts\\python.exe -m interfaces.fetch_request.cli list --status pending\n.venv\\Scripts\\python.exe -m interfaces.fetch_request.cli get <id>\n</code></pre> <p>CLI subcommands: <code>list</code>, <code>get</code>, <code>create-file</code>, <code>create-email</code>, <code>ambiguities</code>, <code>resolve</code>, <code>retry</code>.</p>"},{"location":"07_fetch_request_workflow/#the-worker","title":"\u2699\ufe0f The Worker","text":"<p><code>worker.py</code> polls for pending fetch requests and runs the pipeline.</p> <pre><code>.venv\\Scripts\\python.exe worker.py\n</code></pre> <p>Knobs (env vars):</p> Env var Default Meaning <code>WORKER_POLL_MIN</code> <code>5</code> Min poll interval (seconds); backs off \u00d72 up to MAX when idle <code>WORKER_POLL_MAX</code> <code>60</code> Max poll interval <code>WORKER_MAX_RETRIES</code> <code>3</code> Failed requests stop being retried beyond this count <p>Per cycle: fetch <code>status=\"pending\"</code> (limit 100), mark each <code>processing</code>, run the orchestrator, then:</p> <ul> <li>success \u2192 completed,</li> <li><code>PauseForResolution</code> \u2192 paused (retry count reset),</li> <li>exception \u2192 failed with <code>retry_count + 1</code>.</li> </ul> <p>Ambiguity resolution is enabled only when an <code>LLM_API_KEY</code> is present; today <code>worker._maybe_create_resolver()</code> returns <code>None</code>.</p>"},{"location":"07_fetch_request_workflow/#progress-sse","title":"\ud83d\udce1 Progress & SSE","text":"<p><code>ProgressEventBus</code> (<code>interfaces/sse/</code>) emits typed progress events during a run; clients subscribe to the SSE endpoint per fetch request to watch stage transitions. The domain declares the event model in <code>domain/fetch_pipeline/messaging/</code> (see <code>ProgressEventBus</code> port).</p>"},{"location":"07_fetch_request_workflow/#orchestration","title":"\ud83e\udde0 Orchestration","text":"<p><code>PipelineOrchestrator</code> (<code>domain/fetch_pipeline/utils/pipeline.py</code>) wires the worker's services:</p> <pre><code>fr_svc \u00b7 ambiguity_svc \u00b7 expense_svc \u00b7 entity_svc \u00b7 tag_svc\npdf/email/xlsx/csv fetchers \u00b7 sse bus \u00b7 http cache store \u00b7 resolved_choices\n</code></pre> <p>with <code>pause_on_ambiguity=True</code> and a pre-loaded map of previously resolved ambiguity choices so repeats don't re-pause.</p>"},{"location":"07_fetch_request_workflow/#related","title":"\ud83d\udd17 Related","text":"<p>Extraction Pipelines \u00b7 Domain Resources \u00b7 Running The Service</p>"},{"location":"08_reports_and_caching/","title":"\ud83d\udcc8 Reports & Caching","text":"<p>Reports aggregate expense data; caching keeps the hot paths fast without Mongo round-trips.</p>"},{"location":"08_reports_and_caching/#reports","title":"\ud83d\udcca Reports","text":"<p>Report endpoints live under <code>interfaces/report/</code> and run over the expenses via the report domain models in <code>domain/report/</code>.</p> <ul> <li>Aggregate by tag, entity, category, or date range.</li> <li>Output shapes are defined in domain report models and serialized by the interface DTOs.</li> <li>Shared with the CLI report helpers when driven locally.</li> </ul> <p>Typical flow: <code>GET /reports/{kind}?from=...&to=...&tag=...</code> returns a grouped breakdown (count + total per bucket).</p>"},{"location":"08_reports_and_caching/#repositories-fk-population","title":"\ud83d\uddc3 Repositories & FK Population","text":"<p>Repos in <code>infra/repos/</code> are generic mongo-ops repos wrapped per model. Key behaviors:</p> <ul> <li><code>search(filter={...}, limit=...)</code> \u2014 filter-first reads, no magic.</li> <li>In-memory LRU cache \u2014 the default read path; writes call <code>_invalidate_cache</code>.</li> <li>Optional Redis mirror \u2014 <code>infra/cache/redis.py</code> with TTL-aware caching.</li> <li><code>x-fk</code> cross-collection population \u2014 referenced ids (e.g. <code>account_id</code> on expenses) are populated by explicit filter lookups against the target repo, not Mongo joins.</li> </ul>"},{"location":"08_reports_and_caching/#http-cache","title":"\ud83c\udf10 HTTP Cache","text":"<p><code>infra/http_cache/</code> caches external HTTP responses:</p> Config Meaning <code>HTTP_CACHE_TTL</code> Cache time-to-live for responses <code>HTTP_CACHE_MAX_ENTRIES</code> Eviction cap for the cache store <code>REDIS_URL</code> When set, cache lives in Redis; otherwise in-memory <p>Used by fetchers and enrichment lookups to avoid hammering upstreams during a full-statement import.</p>"},{"location":"08_reports_and_caching/#config-summary","title":"\u2699\ufe0f Config Summary","text":"Env var Purpose <code>MONGO_URI</code> / <code>MONGO_DB</code> MongoDB connection + database <code>REDIS_URL</code> Redis (cache mirror, optional) <code>HTTP_CACHE_TTL</code> / <code>HTTP_CACHE_MAX_ENTRIES</code> HTTP cache behavior <code>LLAMA_CLOUD_API_KEY</code> LlamaParse (llama family) <code>EMAIL_HOST</code> / <code>EMAIL_USER</code> / <code>EMAIL_PASS</code> IMAP ingestion <code>AUTH_SERVER</code> External auth/service integration <code>WORKER_*</code> Worker polling knobs (see fetch-request workflow) <code>DEBUG</code> Verbose logging"},{"location":"08_reports_and_caching/#related","title":"\ud83d\udd17 Related","text":"<p>Domain Resources \u00b7 Running The Service \u00b7 Development</p>"},{"location":"09_running_the_service/","title":"\ud83d\ude80 Running The Service","text":"<p>Boot sequence, configuration, and operational runbooks.</p>"},{"location":"09_running_the_service/#boot","title":"\ud83c\udfc3 Boot","text":"<pre><code># API (OpenAPI-first app from openapi.yaml)\n.venv\\Scripts\\python.exe main.py\n# \u2192 http://localhost:8000, /openapi.json, /docs (per spec config)\n\n# Worker (poll loop)\n.venv\\Scripts\\python.exe worker.py\n</code></pre> <p><code>main.py</code> calls <code>init()</code> (wiring from <code>init.py</code>), registers handlers, and serves. <code>worker.py</code> runs <code>poll_fetch_requests(resources)</code> in a loop.</p>"},{"location":"09_running_the_service/#configuration","title":"\ud83d\udd0c Configuration","text":"<p>Everything is env-var driven (see Reports & Caching for the full table). Minimal <code>.env</code>:</p> <pre><code>MONGO_URI=mongodb://localhost:27017\nMONGO_DB=khata\nREDIS_URL=redis://localhost:6379/0\nWORKER_POLL_MIN=5\nWORKER_POLL_MAX=60\nWORKER_MAX_RETRIES=3\nDEBUG=1\n</code></pre> <p>Secret-ish values (LLM keys, IMAP creds) go through the secrets adapter in <code>infra/adaptors/</code> \u2014 never commit real keys.</p>"},{"location":"09_running_the_service/#local-workflow","title":"\ud83e\uddd1\ud83d\udcbb Local Workflow","text":"<pre><code>pip install -e \".[dev]\"\n\n# seed / smoke\n.venv\\Scripts\\pytest.exe tests\\unit -q\n\n# CLI against the live service\n.venv\\Scripts\\python.exe -m interfaces.fetch_request.cli list\n</code></pre> <p>Requires MongoDB (and Redis for the cache mirror). A local Mongo on <code>mongodb://localhost:27017</code> is the default dev target.</p>"},{"location":"09_running_the_service/#production-notes","title":"\u2699\ufe0f Production Notes","text":"<ul> <li>Run API and worker as separate processes/service units.</li> <li>Keep <code>WORKER_MAX_RETRIES</code> bounded so poisoned fetch requests fail fast.</li> <li>Point <code>REDIS_URL</code> at a real Redis before enabling the HTTP cache.</li> <li>Fetch requests flip <code>paused</code> on ambiguity; a resolver (LLM) must be wired to auto-advance them.</li> </ul>"},{"location":"09_running_the_service/#related","title":"\ud83d\udd17 Related","text":"<p>Fetch Request Workflow \u00b7 Development \u00b7 Development \u2014 Environment</p>"},{"location":"10_development/","title":"\ud83d\udee0 Development","text":"<p>How to work on Khata safely: environments, testing, documentation conventions, and doc builds.</p>"},{"location":"10_development/#sections","title":"\ud83d\udcda Sections","text":"Page What it covers Environment Venv setup, deps, package layout Testing Test layout, running, quality gates GSDFC Docstrings The docstring & stub rules for this repo Building Docs doc-forge lib/api/wiki/MCP builds"},{"location":"10_development/#contributing-checklist","title":"\u2795 Contributing Checklist","text":"<ol> <li>Branch off <code>main</code>, keep the three-layer boundary (no backflow).</li> <li>Every public symbol gets a GSDFC docstring + updated <code>.pyi</code> stub.</li> <li>Wire services through <code>init.py</code> \u2014 do not by-pass the service registry.</li> <li>Run the gates: <code>ruff check</code>, <code>pytest</code>, <code>py_compile</code> on changed files.</li> <li>Regenerate docs (see Building Docs) when API/docstrings change.</li> </ol>"},{"location":"10_development/#related","title":"\ud83d\udd17 Related","text":"<p>Architecture \u00b7 Running The Service</p>"},{"location":"10_development/01_environment/","title":"\u2699 Environment","text":"<p>Local setup for developing <code>khata-app</code>.</p>"},{"location":"10_development/01_environment/#create-the-venv","title":"\u25b6 Create The Venv","text":"<pre><code># PowerShell, from repo root\npython -m venv .venv\n.\\.venv\\Scripts\\Activate.ps1\npip install -e \".[dev,doc]\"\n</code></pre> <p>The repo is Python 3.13 compatible (dev venv runs 3.13.5). <code>pyproject.toml</code> declares the app dist as packages <code>domain*</code>, <code>infra*</code>, <code>interfaces*</code>.</p>"},{"location":"10_development/01_environment/#key-tools","title":"\ud83d\udce6 Key Tools","text":"Tool Where Notes <code>pytest</code> <code>.venv\\Scripts\\pytest.exe</code> Unit + integration tests <code>ruff</code> <code>.venv\\Scripts\\ruff.exe</code> Lint/format checks <code>doc-forge</code> <code>.venv\\Scripts\\doc-forge.exe</code> Docs engine (lib/api/wiki/MCP) <code>mypy</code> (not installed) Optional type gate; stubs currently lint-clean instead <p><code>doc-forge</code> is installed from the local aetos source (<code>pip install -e \"C:\\Users\\vishe\\WorkSpace\\code\\aetos\\doc-forge[mkdocs,mcp]\"</code>); it isn't on PyPI.</p>"},{"location":"10_development/01_environment/#pin-note-python-313","title":"\ud83d\udca1 Pin Note (Python 3.13)","text":"<p>MkDocs highlighter crashes on <code>pygments 2.20</code> under 3.13 (<code>filename=None</code> bug). Keep <code>pygments==2.19.2</code> in the khata venv:</p> <pre><code>.\\.venv\\Scripts\\python.exe -m pip install \"pygments==2.19.2\"\n</code></pre>"},{"location":"10_development/01_environment/#package-map","title":"\ud83e\udded Package Map","text":"<pre><code>khata-app/\n\u251c\u2500\u2500 main.py, worker.py, init.py # entrypoints + wiring\n\u251c\u2500\u2500 openapi.yaml # API contract (drives routing)\n\u251c\u2500\u2500 interfaces/ # openapi handlers, sse, report, cli\n\u251c\u2500\u2500 domain/ # models \u00b7 ports \u00b7 services \u00b7 enrichment\n\u251c\u2500\u2500 infra/ # mongo repos \u00b7 cache \u00b7 fetchers \u00b7 llm pools\n\u251c\u2500\u2500 pipelines/ # extraction families + enrichment nodes\n\u251c\u2500\u2500 tests/ # unit + integration suites\n\u251c\u2500\u2500 docs/ # lib sources, api, wiki, mcp bundles (generated + authored)\n\u251c\u2500\u2500 docforge.nav.yml # lib navigation spec\n\u2514\u2500\u2500 py.typed (in each package) # typed marker for stub distribution\n</code></pre>"},{"location":"10_development/01_environment/#related","title":"\ud83d\udd17 Related","text":"<p>Building Docs \u00b7 Testing \u00b7 Running The Service</p>"},{"location":"10_development/02_testing/","title":"\ud83e\uddea Testing","text":"<p>What's covered and how to run the gates.</p>"},{"location":"10_development/02_testing/#layout","title":"\ud83d\udcc1 Layout","text":"Path Content <code>tests/unit/</code> Pure unit tests (domain, extraction stages, enrichment helpers) <code>tests/domain/</code> Domain sub-suites: resources, enrichment, fetch pipeline <code>tests/</code> (integration) Mongo-backed repo/service integration tests"},{"location":"10_development/02_testing/#running","title":"\ud83d\ude80 Running","text":"<pre><code># Full unit suite\n.\\.venv\\Scripts\\pytest.exe tests\\unit -q\n\n# A single area\n.\\.venv\\Scripts\\pytest.exe tests\\domain\\enrichment -q\n\n# Lint gate (baseline ~79 pre-existing errors on main; don't add new ones)\n.\\.venv\\Scripts\\ruff.exe check domain infra interfaces pipelines\n\n# Syntax gate on everything including new stubs\n.\\.venv\\Scripts\\python.exe -m py_compile .\\main.py .\\worker.py\nGet-ChildItem -Recurse -Filter *.py -Path domain,infra,interfaces,pipelines | `\n ForEach-Object { .\\.venv\\Scripts\\python.exe -m py_compile $_.FullName }\n</code></pre> <p>Current health: 1190 passed, 1 skipped; all <code>*.pyi</code> lint-clean after the F821 fixes in the enrichment pipeline stubs.</p>"},{"location":"10_development/02_testing/#whats-heavily-tested","title":"\ud83e\udde9 What's Heavily Tested","text":"<ul> <li>Enrichment \u2014 identity/fingerprint edge cases, tag flag computation, pipeline wiring.</li> <li>Extraction \u2014 CSV/XLSX/PDF tabular parsing against golden fixtures, ambiguity detection.</li> <li>Repos \u2014 cache invalidation and <code>x-fk</code> population against an in-memory Mongo.</li> </ul>"},{"location":"10_development/02_testing/#stub-checks","title":"\u26a0\ufe0f Stub Checks","text":"<p>Stubs mirror public signatures exactly. When a signature changes, the <code>.pyi</code> must change with it or the type gate breaks. Two stubs import <code>dagpipe.Pipeline</code> instead of a rogue string forward-ref:</p> <pre><code># domain/enrichment/entity/pipeline.pyi\n# domain/enrichment/tags/pipeline.pyi\nfrom dagpipe import Pipeline\ndef load() -> Pipeline: ...\n</code></pre>"},{"location":"10_development/02_testing/#related","title":"\ud83d\udd17 Related","text":"<p>Environment \u00b7 GSDFC Docstrings \u00b7 Building Docs</p>"},{"location":"10_development/03_gsdfc_docstrings/","title":"\ud83d\udcdd GSDFC Docstrings","text":"<p><code>khata-app</code> adheres to the GSDFC docstring conventions (see the doc-forge GSDFC guide). The rules below are the ones enforced in this repo.</p>"},{"location":"10_development/03_gsdfc_docstrings/#core-rules","title":"\ud83c\udfd7 Core Rules","text":"<ul> <li>Modules \u2014 start with a Markdown <code># Summary</code> heading, then a <code>---</code> separator, then prose. Long modules follow with <code>Notes:</code>/<code>Example:</code> sections.</li> <li>Classes \u2014 use <code>Attributes:</code> for fields, <code>Notes:</code> for invariants, <code>Example:</code> when a snippet clarifies usage.</li> <li>Functions/methods \u2014 imperative summary line, then <code>Args:</code> / <code>Returns:</code> / <code>Raises:</code> / <code>Yields:</code> in that conventional order, then optional <code>Notes:</code> / <code>Example:</code>.</li> <li>Prose types must match the signature verbatim \u2014 no modernization. If the source says <code>str | None</code>, the docstring and the stub say <code>str | None</code>; do not write <code>Optional[str]</code>.</li> </ul>"},{"location":"10_development/03_gsdfc_docstrings/#format-cheat-sheet","title":"\ud83d\udcc4 Format Cheat-Sheet","text":"<pre><code>\"\"\"# Summary\n\nImperative one-liner of what this does.\n\n---\n\nArgs:\n name (str):\n Description on the next line.\n\nReturns:\n bool:\n Description.\n\"\"\"\n</code></pre> <p>A typical module docstring:</p> <pre><code>\"\"\"# Summary\n\n...one-liner...\n\n---\n\nNotes:\n- point one.\n- point two.\n\"\"\"\n</code></pre>"},{"location":"10_development/03_gsdfc_docstrings/#stub-files","title":"\ud83d\uddc3 Stub Files","text":"<p>Every module gets a sibling <code>.pyi</code> stub mirroring the public surface only:</p> <ul> <li>exact signatures and return annotations,</li> <li><code>...</code> bodies,</li> <li>no docstrings (the source carries them),</li> <li>no private helpers, no imports of implementation-only symbols.</li> </ul> <p>Generate/validate conveniently with doc-forge's stub tooling, and always run <code>py_compile</code> + <code>ruff</code> on new stubs.</p>"},{"location":"10_development/03_gsdfc_docstrings/#anti-patterns","title":"\u274c Anti-Patterns","text":"<ul> <li>Docstrings that contradict annotations (typed vs untyped mismatch).</li> <li>Forward-refs as strings in stubs (use a real <code>import</code> instead \u2014 e.g. <code>from dagpipe import Pipeline</code>).</li> <li>Documenting private / underscored members in stubs (public surface only).</li> </ul>"},{"location":"10_development/03_gsdfc_docstrings/#related","title":"\ud83d\udd17 Related","text":"<p>Building Docs \u00b7 Testing \u00b7 Development</p>"},{"location":"10_development/04_building_docs/","title":"\ud83d\udee0 Building Docs","text":"<p>All documentation for <code>khata-app</code> is produced with doc-forge (installed into the khata venv from the local aetos source).</p>"},{"location":"10_development/04_building_docs/#what-gets-built","title":"\ud83d\udccb What Gets Built","text":"Target Command (from repo root) Output Lib (mkdocs) <code>doc-forge build --mkdocs --module <pkg> [--refresh]</code> <code>docs/lib/\u2026</code> + <code>docs/mkdocs.lib.yml</code>, <code>site/lib/</code> MCP bundles <code>doc-forge build --mcp --module <pkg> --out-dir docs\\mcp\\<pkg></code> <code>docs/mcp/<pkg>/</code> (index.json, nav.json, modules/) API (OpenAPI) <code>doc-forge build --api --openapi-spec <spec.json></code> <code>docs/api/</code> + <code>docs/mkdocs.api.yml</code> Wiki <code>doc-forge build --wiki --site-name \"Khata App \u2014 Wiki\"</code> nav from <code>docs/wiki/</code> + <code>docs/mkdocs.wiki.yml</code> Serve <code>doc-forge serve</code> local preview <p>Builds must run from the repo root so Griffe discovers the packages.</p>"},{"location":"10_development/04_building_docs/#lib-builds","title":"\ud83d\udcda Lib Builds","text":"<pre><code># first time per package\n.\\.venv\\Scripts\\doc-forge.exe build --mkdocs --module domain\n.\\.venv\\Scripts\\doc-forge.exe build --mkdocs --module infra\n.\\.venv\\Scripts\\doc-forge.exe build --mkdocs --module interfaces\n.\\.venv\\Scripts\\doc-forge.exe build --mkdocs --module pipelines\n\n# after docstring/nav changes, refresh and set the nice site name\n.\\.venv\\Scripts\\doc-forge.exe build --mkdocs --module domain --refresh --site-name \"Khata App \u2014 Library Reference\" --project-name khata-app\n</code></pre> <p><code>docforge.nav.yml</code> drives the nav groups:</p> <pre><code>home: lib/index.md\ngroups:\n Domain: [lib/domain/index.md, lib/domain/enrichment/index.md, lib/domain/fetch_pipeline/index.md, lib/domain/fetch_request/index.md]\n Infra: [lib/infra/index.md]\n Interfaces: [lib/interfaces/index.md]\n Pipelines: [lib/pipelines/index.md]\n</code></pre>"},{"location":"10_development/04_building_docs/#mcp-bundles","title":"\ud83e\udde0 MCP Bundles","text":"<p>Serve the API as machine-readable resources an MCP client can consume:</p> <pre><code>.\\.venv\\Scripts\\doc-forge.exe build --mcp --module domain --out-dir docs\\mcp\\domain\n.\\.venv\\Scripts\\doc-forge.exe build --mcp --module infra --out-dir docs\\mcp\\infra\n.\\.venv\\Scripts\\doc-forge.exe build --mcp --module interfaces --out-dir docs\\mcp\\interfaces\n.\\.venv\\Scripts\\doc-forge.exe build --mcp --module pipelines --out-dir docs\\mcp\\pipelines\n</code></pre>"},{"location":"10_development/04_building_docs/#api-build","title":"\ud83c\udf10 API Build","text":"<p><code>build --api</code> reads JSON (not YAML). Convert <code>openapi.yaml</code> first:</p> <pre><code>.\\.venv\\Scripts\\python.exe -c \"import yaml,json,pathlib; p=pathlib.Path('docs/openapi.json'); p.write_text(json.dumps(yaml.safe_load(pathlib.Path('openapi.yaml').read_text(encoding='utf-8')),indent=2),encoding='utf-8')\"\n.\\.venv\\Scripts\\doc-forge.exe build --api --openapi-spec docs\\openapi.json\n</code></pre>"},{"location":"10_development/04_building_docs/#wiki-build","title":"\ud83d\udcd6 Wiki Build","text":"<p>The wiki is hand-authored under <code>docs/wiki/</code>; doc-forge only derives the nav:</p> <pre><code>.\\.venv\\Scripts\\doc-forge.exe build --wiki --site-name \"Khata App \u2014 Wiki\" --project-name khata-app\n</code></pre> <p>Nav labels come from filenames (<code>01_overview.md</code> \u2192 <code>Overview</code>); tune <code>docs/mkdocs.wiki.yml</code> after generation for nicer titles.</p>"},{"location":"10_development/04_building_docs/#known-gotchas","title":"\u26a0\ufe0f Known Gotchas","text":"<ul> <li>pygments \u2014 3.13 + pygments 2.20 crashes the highlighter; keep <code>pygments==2.19.2</code>.</li> <li>nav refresh \u2014 after changing <code>docforge.nav.yml</code>, rebuild with <code>--refresh</code> or the nav won't regenerate.</li> <li>unregistered links \u2014 MkDocs may warn about <code>ports/</code> style relative links on the auto-generated index pages; cosmetic only.</li> </ul>"},{"location":"10_development/04_building_docs/#related","title":"\ud83d\udd17 Related","text":"<p>Environment \u00b7 GSDFC Docstrings \u00b7 Testing</p>"}]} |