Vishesh 'ironeagle' Bangotra
b1447fb72b
Adds the mail-intake wiki to the hub alongside lib and mcp and picks up the regenerated flat lib reference and standardized MCP modules.
1 line
18 KiB
JSON
1 line
18 KiB
JSON
{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"\ud83d\udcec Mail Intake \u2014 Provider-Agnostic Email Ingestion","text":"<p><code>Mail Intake</code> is a contract-first, read-only email ingestion framework. It pulls mail from external providers (such as Gmail), parses and normalizes it into clean, provider-agnostic domain models \u2014 ready to persist, index, or analyze downstream.</p> <p>Doc model: this wiki is written for humans \u2014 how\u2011to guides and extension recipes. The authoritative API contracts live in the code (GSDFC docstrings) and the machine\u2011readable bundle under <code>docs/mcp/</code>.</p>"},{"location":"#key-features","title":"\ud83d\ude80 Key Features","text":"<ul> <li>\ud83d\udcec Read-only ingestion \u2014 never mutates provider state</li> <li>\ud83e\udde9 Contract-first layers \u2014 adapters, parsers, and readers separated</li> <li>\u2709\ufe0f Provider-agnostic models \u2014 <code>MailIntakeMessage</code> / <code>MailIntakeThread</code> have no provider internals</li> <li>\ud83d\udd10 Extensible auth \u2014 pluggable auth providers and credential stores (pickle for dev, Redis for production)</li> <li>\ud83e\uddea Deterministic & testable \u2014 no implicit global state or env reads</li> <li>\ud83d\udcca Gmail support \u2014 reference adapter built on the official Google APIs</li> </ul>"},{"location":"#quick-start","title":"\u26a1 Quick Start","text":"<pre><code>from mail_intake.ingestion import MailIntakeReader\nfrom mail_intake.adapters import MailIntakeGmailAdapter\nfrom mail_intake.auth import MailIntakeGoogleAuth\nfrom mail_intake.credentials import PickleCredentialStore\n\nstore = PickleCredentialStore(path=\"token.pickle\")\n\nauth = MailIntakeGoogleAuth(\n credentials_path=\"credentials.json\",\n store=store,\n scopes=[\"https://www.googleapis.com/auth/gmail.readonly\"],\n)\n\nadapter = MailIntakeGmailAdapter(auth_provider=auth)\nreader = MailIntakeReader(adapter)\n\nfor message in reader.iter_messages(\"from:recruiter@example.com\"):\n print(message.subject, message.from_email)\n</code></pre>"},{"location":"#documentation-structure","title":"\ud83d\udcc1 Documentation Structure","text":"Section What you'll find Overview Layers, domain models, and design guarantees How to Use Gmail ingestion, parsing, and credential stores Extending Mail Intake Custom adapters, auth providers, stores Development Setup, tests, and regenerating docs"},{"location":"#related-resources","title":"\ud83d\udd17 Related Resources","text":"<ul> <li>Source Code: Gitea Repository</li> <li>Internal PyPI: pip.aetoskia.com/simple/mail-intake</li> <li>CI: Builds and publishes tagged releases, gated on black / ruff / mypy / pytest.</li> </ul> <p>\u00a9 Aetoskia Internal \u2014 <code>mail-intake</code> 0.0.2</p>"},{"location":"01_overview/","title":"\ud83e\uddf1 Overview","text":"<p>Mail Intake is a contract-first ingestion pipeline. Adapters handle transport to a provider, parsers normalize provider payloads, and the reader orchestrates the whole flow into canonical domain models.</p>"},{"location":"01_overview/#architecture","title":"\ud83c\udfd7\ufe0f Architecture","text":"<pre><code> \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 External Provider (e.g. Gmail API) \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 MailIntakeAdapter (transport) \u2502 provider API calls\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 provider-native payloads\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Parsers (normalization) \u2502 headers, body, subject\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 composed\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 MailIntakeReader (orchestration) \u2502 iter_messages / iter_threads\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 MailIntakeMessage / Thread \u2502 canonical domain models\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n</code></pre> <p>Layers:</p> <ol> <li>Adapters (<code>mail_intake.adapters</code>) \u2014 provider-specific, read-only transport. Return provider-native payloads; never interpret them.</li> <li>Auth (<code>mail_intake.auth</code>) \u2014 credential acquisition and lifecycle management, decoupled from adapters.</li> <li>Credentials (<code>mail_intake.credentials</code>) \u2014 persistence of auth tokens; <code>PickleCredentialStore</code> locally, <code>RedisCredentialStore</code> for production.</li> <li>Parsers (<code>mail_intake.parsers</code>) \u2014 extract headers, body text, sender, and normalized subjects from provider payloads.</li> <li>Ingestion (<code>mail_intake.ingestion</code>) \u2014 <code>MailIntakeReader</code> wires an adapter + parsers into iterators over canonical models.</li> <li>Models (<code>mail_intake.models</code>) \u2014 provider-agnostic <code>MailIntakeMessage</code> and <code>MailIntakeThread</code>.</li> </ol>"},{"location":"01_overview/#domain-models","title":"\ud83d\udce6 Domain models","text":"<p><code>MailIntakeMessage</code>:</p> Field Type Meaning <code>message_id</code> <code>str</code> Provider message id <code>thread_id</code> <code>str</code> Conversation thread id <code>timestamp</code> <code>datetime</code> Message timestamp <code>from_email</code> <code>str</code> Sender email <code>from_name</code> <code>str \\| None</code> Sender display name <code>subject</code> <code>str</code> Message subject <code>body_text</code> <code>str</code> Extracted plain-text body <code>snippet</code> <code>str</code> Provider snippet <code>raw_headers</code> <code>dict[str, str]</code> Unmodified headers <p><code>MailIntakeThread</code>:</p> Field Type Meaning <code>thread_id</code> <code>str</code> Conversation id <code>normalized_subject</code> <code>str</code> Normalized subject (threads share one) <code>participants</code> <code>set[str]</code> Distinct senders <code>messages</code> <code>list[MailIntakeMessage]</code> Ordered messages <code>last_activity_at</code> <code>datetime \\| None</code> Latest message time"},{"location":"01_overview/#design-guarantees","title":"\ud83d\udd12 Design guarantees","text":"<ul> <li>Read-only access \u2014 no mutation of provider state.</li> <li>Provider-agnostic domain models.</li> <li>Explicit configuration and dependency injection (no implicit env reads).</li> <li>Extensible via public contracts; built-in adapters are reference implementations and may change internally.</li> </ul>"},{"location":"01_overview/#read-next","title":"\ud83d\udcda Read Next","text":"<ul> <li>How to Use \u2014 the Gmail ingestion flow.</li> <li>Extending Mail Intake \u2014 custom adapters and stores.</li> </ul>"},{"location":"02_how_to_use/","title":"\ud83d\udda5\ufe0f How to Use","text":"<p>This page walks through authenticating, ingesting, and parsing mail with the built-in Gmail support.</p>"},{"location":"02_how_to_use/#authentication","title":"\ud83d\udd10 Authentication","text":"<p>Create a credential store and an auth provider:</p> <pre><code>from mail_intake.auth import MailIntakeGoogleAuth\nfrom mail_intake.credentials import PickleCredentialStore, RedisCredentialStore\n\nstore = PickleCredentialStore(path=\"token.pickle\") # local dev\n\nauth = MailIntakeGoogleAuth(\n credentials_path=\"credentials.json\", # your OAuth client file\n store=store,\n scopes=[\"https://www.googleapis.com/auth/gmail.readonly\"],\n)\n</code></pre> <p>OAuth credentials come from the Google Cloud Console and are provided via file paths \u2014 never hard-code tokens or secrets in source. Once authorized, the token is persisted by the credential store and refreshed automatically.</p> <p>For distributed deployments, use Redis instead of pickle:</p> <pre><code>store = RedisCredentialStore(redis_client=redis_client) # production\n</code></pre>"},{"location":"02_how_to_use/#ingesting-messages","title":"\ud83d\udce5 Ingesting messages","text":"<p>Build an adapter and a reader, then iterate:</p> <pre><code>from mail_intake.ingestion import MailIntakeReader\nfrom mail_intake.adapters import MailIntakeGmailAdapter\n\nadapter = MailIntakeGmailAdapter(auth_provider=auth)\nreader = MailIntakeReader(adapter)\n\nfor message in reader.iter_messages(\"from:recruiter@example.com\"):\n print(message.subject, message.from_email, message.timestamp)\n</code></pre> <p>Access the full model:</p> <pre><code>message.message_id # provider id\nmessage.thread_id\nmessage.body_text # plain-text body\nmessage.raw_headers # unmodified headers dict\n</code></pre>"},{"location":"02_how_to_use/#ingesting-threads","title":"\ud83d\udd16 Ingesting threads","text":"<pre><code>for thread in reader.iter_threads(\"subject:Interview\"):\n print(thread.normalized_subject)\n print(thread.participants)\n print(len(thread.messages))\n print(thread.last_activity_at)\n</code></pre> <p><code>MailIntakeThread</code> aggregates its messages, participants, and last activity.</p>"},{"location":"02_how_to_use/#parsers","title":"\ud83e\uddea Parsers","text":"<p>Parsers normalize provider payloads and are used by the reader internally. They are also importable on their own:</p> <pre><code>from mail_intake.parsers import extract_body, parse_headers, normalize_subject\n</code></pre> <ul> <li><code>extract_body(...)</code> \u2014 plain-text body extraction.</li> <li><code>parse_headers(...)</code> \u2014 structured header parsing.</li> <li><code>extract_sender(...)</code> \u2014 sender email/name extraction.</li> <li><code>normalize_subject(...)</code> \u2014 subject normalization for threading.</li> </ul>"},{"location":"02_how_to_use/#checklist","title":"\u2705 Checklist","text":"<ol> <li>Provide OAuth credentials via a file path and a credential store.</li> <li>Build the <code>MailIntakeGoogleAuth</code> provider with the Gmail read-only scope.</li> <li>Wrap the adapter in a <code>MailIntakeReader</code>.</li> <li>Consume <code>iter_messages</code> / <code>iter_threads</code>; never call provider APIs directly.</li> </ol>"},{"location":"02_how_to_use/#read-next","title":"\ud83d\udcda Read Next","text":"<ul> <li>Overview \u2014 layers and domain models.</li> <li>Extending Mail Intake \u2014 custom providers and stores.</li> </ul>"},{"location":"03_extending/","title":"\ud83e\udde9 Extending Mail Intake","text":"<p>Mail Intake is designed to be extended through its public contracts. Implement your own adapter, auth provider, or credential store to support a new provider or storage backend.</p>"},{"location":"03_extending/#custom-adapters","title":"\ud83d\udcec Custom adapters","text":"<p>Subclass <code>MailIntakeAdapter</code> to integrate a new provider. Adapters perform read-only transport and return provider-native payloads:</p> <pre><code>from mail_intake.adapters import MailIntakeAdapter\n\nclass ExchangeAdapter(MailIntakeAdapter):\n def __init__(self, auth_provider):\n self._auth = auth_provider\n\n def fetch_messages(self, query):\n # call the provider API, return native payloads\n ...\n\n def fetch_threads(self, query):\n ...\n</code></pre> <p>Do not subclass built-in adapters like <code>MailIntakeGmailAdapter</code> \u2014 they are reference implementations and may change internally without notice.</p>"},{"location":"03_extending/#custom-auth-providers","title":"\ud83d\udd10 Custom auth providers","text":"<p>Subclass <code>MailIntakeAuthProvider[T]</code> to own a different credential flow:</p> <pre><code>from mail_intake.auth import MailIntakeAuthProvider\n\nclass ExchangeAuth(MailIntakeAuthProvider[exchange_credentials]):\n def get_credentials(self):\n return self._store.load()\n\n def refresh(self):\n ...\n</code></pre> <p>Auth providers remain decoupled from adapter logic \u2014 they only manage credentials.</p>"},{"location":"03_extending/#custom-credential-stores","title":"\ud83d\uddc4\ufe0f Custom credential stores","text":"<p>Implement the <code>CredentialStore[T]</code> contract for a new persistence backend:</p> <pre><code>from mail_intake.credentials import CredentialStore\n\nclass S3CredentialStore(CredentialStore):\n def save(self, credentials):\n ...\n\n def load(self):\n ...\n\n def delete(self):\n ...\n</code></pre> <p>The store abstraction keeps tokens out of config and rotates safely.</p>"},{"location":"03_extending/#wiring-a-custom-backend","title":"\ud83e\uddea Wiring a custom backend","text":"<pre><code>from mail_intake.ingestion import MailIntakeReader\n\nauth = ExchangeAuth(store=S3CredentialStore(...))\nadapter = ExchangeAdapter(auth_provider=auth)\nreader = MailIntakeReader(adapter)\n\nfor message in reader.iter_messages(\"query\"):\n print(message.subject)\n</code></pre> <p>The reader only depends on the adapter contract, so the rest of the pipeline keeps working unchanged.</p>"},{"location":"03_extending/#extension-checklist","title":"\u2705 Extension checklist","text":"<ol> <li>Implement the public contract \u2014 never subclass built-in adapters.</li> <li>Keep transport in the adapter, parsing in parsers, auth in the provider.</li> <li>Return the provider-native payload and let parsers normalize it.</li> <li>Inject dependencies explicitly \u2014 no global state or env reads.</li> </ol>"},{"location":"03_extending/#read-next","title":"\ud83d\udcda Read Next","text":"<ul> <li>How to Use \u2014 the built-in Gmail flow.</li> <li>Development \u2014 running tests and docs.</li> </ul>"},{"location":"04_development/","title":"\ud83d\udee0\ufe0f Development","text":"<p>Working on <code>mail-intake</code> itself.</p>"},{"location":"04_development/#repository-layout","title":"\ud83d\udcc2 Repository layout","text":"Path Purpose <code>mail_intake/</code> The library package (adapters, auth, credentials, parsers, ingestion, models) <code>mail_intake/*.pyi</code> Type stubs kept in sync with implementations <code>tests/</code> Unit and integration tests (mock transports, no live mail) <code>docs/lib/</code> Generated library reference (docforge, flat layout) <code>docs/mcp/</code> Machine-readable bundle served by the MCP server <code>docs/wiki/</code> This hand-written wiki"},{"location":"04_development/#setup","title":"\ud83d\udd27 Setup","text":"<pre><code>python -m venv .venv\n.venv/Scripts/pip install -e \".[dev]\"\n</code></pre> <p>OAuth credential files (<code>credentials*.json</code>, <code>token.pickle</code>, <code>client_secret_*.json</code>) are gitignored \u2014 keep them out of the repository.</p>"},{"location":"04_development/#tests","title":"\ud83e\uddea Tests","text":"<p>Run the suite (no network or live Gmail required):</p> <pre><code>.venv/Scripts/pytest\n</code></pre> <p>Coverage spans ingestion flows, credential stores, parsers, and auth against mock providers.</p>"},{"location":"04_development/#quality-gates","title":"\u2705 Quality gates","text":"<p>The CI quality gate runs, matching the Drone pipeline:</p> <pre><code>.venv/Scripts/black --check .\n.venv/Scripts/ruff check .\n.venv/Scripts/mypy\n.venv/Scripts/pytest\n</code></pre>"},{"location":"04_development/#building-documentation-docforge","title":"\ud83d\udcdd Building documentation (docforge)","text":"<p>The site is generated by <code>docforge</code> and served per kind under <code>site/{kind}</code>:</p> <pre><code>doc-forge build \\\n --mkdocs --mcp --wiki \\\n --module-is-source --module mail_intake \\\n --site-name \"Mail Intake\"\n</code></pre> <ul> <li><code>--module-is-source</code> renders the flat <code>docs/lib/</code> layout (no nesting under <code>mail_intake/</code>), matching <code>docforge.nav.yml</code> and <code>docs/mkdocs.lib.yml</code>.</li> <li><code>--mcp</code> regenerates the structured bundle in <code>docs/mcp/</code>.</li> <li><code>--wiki</code> builds this wiki.</li> </ul> <p>Preview locally:</p> <pre><code>doc-forge serve --lib\ndoc-forge serve --wiki\ndoc-forge serve --mcp\n</code></pre>"},{"location":"04_development/#read-next","title":"\ud83d\udcda Read Next","text":"<ul> <li>Extending Mail Intake \u2014 custom adapters and stores.</li> <li>Overview \u2014 the core architecture.</li> </ul>"}]} |