{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"\ud83d\udcec Mail Intake \u2014 Provider-Agnostic Email Ingestion","text":"

Mail Intake is a contract-first, read-only email ingestion framework. It pulls mail from external providers (such as Gmail), parses and normalizes it into clean, provider-agnostic domain models \u2014 ready to persist, index, or analyze downstream.

Doc model: this wiki is written for humans \u2014 how\u2011to guides and extension recipes. The authoritative API contracts live in the code (GSDFC docstrings) and the machine\u2011readable bundle under docs/mcp/.

"},{"location":"#key-features","title":"\ud83d\ude80 Key Features","text":""},{"location":"#quick-start","title":"\u26a1 Quick Start","text":"
from mail_intake.ingestion import MailIntakeReader\nfrom mail_intake.adapters import MailIntakeGmailAdapter\nfrom mail_intake.auth import MailIntakeGoogleAuth\nfrom mail_intake.credentials import PickleCredentialStore\n\nstore = PickleCredentialStore(path=\"token.pickle\")\n\nauth = MailIntakeGoogleAuth(\n    credentials_path=\"credentials.json\",\n    store=store,\n    scopes=[\"https://www.googleapis.com/auth/gmail.readonly\"],\n)\n\nadapter = MailIntakeGmailAdapter(auth_provider=auth)\nreader = MailIntakeReader(adapter)\n\nfor message in reader.iter_messages(\"from:recruiter@example.com\"):\n    print(message.subject, message.from_email)\n
"},{"location":"#documentation-structure","title":"\ud83d\udcc1 Documentation Structure","text":"Section What you'll find Overview Layers, domain models, and design guarantees How to Use Gmail ingestion, parsing, and credential stores Extending Mail Intake Custom adapters, auth providers, stores Development Setup, tests, and regenerating docs"},{"location":"#related-resources","title":"\ud83d\udd17 Related Resources","text":"

\u00a9 Aetoskia Internal \u2014 mail-intake 0.0.2

"},{"location":"01_overview/","title":"\ud83e\uddf1 Overview","text":"

Mail Intake is a contract-first ingestion pipeline. Adapters handle transport to a provider, parsers normalize provider payloads, and the reader orchestrates the whole flow into canonical domain models.

"},{"location":"01_overview/#architecture","title":"\ud83c\udfd7\ufe0f Architecture","text":"
                 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n                 \u2502  External Provider (e.g. Gmail API)     \u2502\n                 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n                                      \u2502\n                    \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n                    \u2502  MailIntakeAdapter (transport)      \u2502   provider API calls\n                    \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n                                      \u2502  provider-native payloads\n                    \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n                    \u2502  Parsers (normalization)           \u2502   headers, body, subject\n                    \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n                                      \u2502  composed\n                    \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n                    \u2502  MailIntakeReader (orchestration)  \u2502   iter_messages / iter_threads\n                    \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n                                      \u2502\n                    \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n                    \u2502  MailIntakeMessage / Thread        \u2502   canonical domain models\n                    \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n

Layers:

  1. Adapters (mail_intake.adapters) \u2014 provider-specific, read-only transport. Return provider-native payloads; never interpret them.
  2. Auth (mail_intake.auth) \u2014 credential acquisition and lifecycle management, decoupled from adapters.
  3. Credentials (mail_intake.credentials) \u2014 persistence of auth tokens; PickleCredentialStore locally, RedisCredentialStore for production.
  4. Parsers (mail_intake.parsers) \u2014 extract headers, body text, sender, and normalized subjects from provider payloads.
  5. Ingestion (mail_intake.ingestion) \u2014 MailIntakeReader wires an adapter + parsers into iterators over canonical models.
  6. Models (mail_intake.models) \u2014 provider-agnostic MailIntakeMessage and MailIntakeThread.
"},{"location":"01_overview/#domain-models","title":"\ud83d\udce6 Domain models","text":"

MailIntakeMessage:

Field Type Meaning message_id str Provider message id thread_id str Conversation thread id timestamp datetime Message timestamp from_email str Sender email from_name str \\| None Sender display name subject str Message subject body_text str Extracted plain-text body snippet str Provider snippet raw_headers dict[str, str] Unmodified headers

MailIntakeThread:

Field Type Meaning thread_id str Conversation id normalized_subject str Normalized subject (threads share one) participants set[str] Distinct senders messages list[MailIntakeMessage] Ordered messages last_activity_at datetime \\| None Latest message time"},{"location":"01_overview/#design-guarantees","title":"\ud83d\udd12 Design guarantees","text":""},{"location":"01_overview/#read-next","title":"\ud83d\udcda Read Next","text":""},{"location":"02_how_to_use/","title":"\ud83d\udda5\ufe0f How to Use","text":"

This page walks through authenticating, ingesting, and parsing mail with the built-in Gmail support.

"},{"location":"02_how_to_use/#authentication","title":"\ud83d\udd10 Authentication","text":"

Create a credential store and an auth provider:

from mail_intake.auth import MailIntakeGoogleAuth\nfrom mail_intake.credentials import PickleCredentialStore, RedisCredentialStore\n\nstore = PickleCredentialStore(path=\"token.pickle\")          # local dev\n\nauth = MailIntakeGoogleAuth(\n    credentials_path=\"credentials.json\",                    # your OAuth client file\n    store=store,\n    scopes=[\"https://www.googleapis.com/auth/gmail.readonly\"],\n)\n

OAuth credentials come from the Google Cloud Console and are provided via file paths \u2014 never hard-code tokens or secrets in source. Once authorized, the token is persisted by the credential store and refreshed automatically.

For distributed deployments, use Redis instead of pickle:

store = RedisCredentialStore(redis_client=redis_client)     # production\n
"},{"location":"02_how_to_use/#ingesting-messages","title":"\ud83d\udce5 Ingesting messages","text":"

Build an adapter and a reader, then iterate:

from mail_intake.ingestion import MailIntakeReader\nfrom mail_intake.adapters import MailIntakeGmailAdapter\n\nadapter = MailIntakeGmailAdapter(auth_provider=auth)\nreader = MailIntakeReader(adapter)\n\nfor message in reader.iter_messages(\"from:recruiter@example.com\"):\n    print(message.subject, message.from_email, message.timestamp)\n

Access the full model:

message.message_id    # provider id\nmessage.thread_id\nmessage.body_text     # plain-text body\nmessage.raw_headers   # unmodified headers dict\n
"},{"location":"02_how_to_use/#ingesting-threads","title":"\ud83d\udd16 Ingesting threads","text":"
for thread in reader.iter_threads(\"subject:Interview\"):\n    print(thread.normalized_subject)\n    print(thread.participants)\n    print(len(thread.messages))\n    print(thread.last_activity_at)\n

MailIntakeThread aggregates its messages, participants, and last activity.

"},{"location":"02_how_to_use/#parsers","title":"\ud83e\uddea Parsers","text":"

Parsers normalize provider payloads and are used by the reader internally. They are also importable on their own:

from mail_intake.parsers import extract_body, parse_headers, normalize_subject\n
"},{"location":"02_how_to_use/#checklist","title":"\u2705 Checklist","text":"
  1. Provide OAuth credentials via a file path and a credential store.
  2. Build the MailIntakeGoogleAuth provider with the Gmail read-only scope.
  3. Wrap the adapter in a MailIntakeReader.
  4. Consume iter_messages / iter_threads; never call provider APIs directly.
"},{"location":"02_how_to_use/#read-next","title":"\ud83d\udcda Read Next","text":""},{"location":"03_extending/","title":"\ud83e\udde9 Extending Mail Intake","text":"

Mail Intake is designed to be extended through its public contracts. Implement your own adapter, auth provider, or credential store to support a new provider or storage backend.

"},{"location":"03_extending/#custom-adapters","title":"\ud83d\udcec Custom adapters","text":"

Subclass MailIntakeAdapter to integrate a new provider. Adapters perform read-only transport and return provider-native payloads:

from mail_intake.adapters import MailIntakeAdapter\n\nclass ExchangeAdapter(MailIntakeAdapter):\n    def __init__(self, auth_provider):\n        self._auth = auth_provider\n\n    def fetch_messages(self, query):\n        # call the provider API, return native payloads\n        ...\n\n    def fetch_threads(self, query):\n        ...\n

Do not subclass built-in adapters like MailIntakeGmailAdapter \u2014 they are reference implementations and may change internally without notice.

"},{"location":"03_extending/#custom-auth-providers","title":"\ud83d\udd10 Custom auth providers","text":"

Subclass MailIntakeAuthProvider[T] to own a different credential flow:

from mail_intake.auth import MailIntakeAuthProvider\n\nclass ExchangeAuth(MailIntakeAuthProvider[exchange_credentials]):\n    def get_credentials(self):\n        return self._store.load()\n\n    def refresh(self):\n        ...\n

Auth providers remain decoupled from adapter logic \u2014 they only manage credentials.

"},{"location":"03_extending/#custom-credential-stores","title":"\ud83d\uddc4\ufe0f Custom credential stores","text":"

Implement the CredentialStore[T] contract for a new persistence backend:

from mail_intake.credentials import CredentialStore\n\nclass S3CredentialStore(CredentialStore):\n    def save(self, credentials):\n        ...\n\n    def load(self):\n        ...\n\n    def delete(self):\n        ...\n

The store abstraction keeps tokens out of config and rotates safely.

"},{"location":"03_extending/#wiring-a-custom-backend","title":"\ud83e\uddea Wiring a custom backend","text":"
from mail_intake.ingestion import MailIntakeReader\n\nauth = ExchangeAuth(store=S3CredentialStore(...))\nadapter = ExchangeAdapter(auth_provider=auth)\nreader = MailIntakeReader(adapter)\n\nfor message in reader.iter_messages(\"query\"):\n    print(message.subject)\n

The reader only depends on the adapter contract, so the rest of the pipeline keeps working unchanged.

"},{"location":"03_extending/#extension-checklist","title":"\u2705 Extension checklist","text":"
  1. Implement the public contract \u2014 never subclass built-in adapters.
  2. Keep transport in the adapter, parsing in parsers, auth in the provider.
  3. Return the provider-native payload and let parsers normalize it.
  4. Inject dependencies explicitly \u2014 no global state or env reads.
"},{"location":"03_extending/#read-next","title":"\ud83d\udcda Read Next","text":""},{"location":"04_development/","title":"\ud83d\udee0\ufe0f Development","text":"

Working on mail-intake itself.

"},{"location":"04_development/#repository-layout","title":"\ud83d\udcc2 Repository layout","text":"Path Purpose mail_intake/ The library package (adapters, auth, credentials, parsers, ingestion, models) mail_intake/*.pyi Type stubs kept in sync with implementations tests/ Unit and integration tests (mock transports, no live mail) docs/lib/ Generated library reference (docforge, flat layout) docs/mcp/ Machine-readable bundle served by the MCP server docs/wiki/ This hand-written wiki"},{"location":"04_development/#setup","title":"\ud83d\udd27 Setup","text":"
python -m venv .venv\n.venv/Scripts/pip install -e \".[dev]\"\n

OAuth credential files (credentials*.json, token.pickle, client_secret_*.json) are gitignored \u2014 keep them out of the repository.

"},{"location":"04_development/#tests","title":"\ud83e\uddea Tests","text":"

Run the suite (no network or live Gmail required):

.venv/Scripts/pytest\n

Coverage spans ingestion flows, credential stores, parsers, and auth against mock providers.

"},{"location":"04_development/#quality-gates","title":"\u2705 Quality gates","text":"

The CI quality gate runs, matching the Drone pipeline:

.venv/Scripts/black --check .\n.venv/Scripts/ruff check .\n.venv/Scripts/mypy\n.venv/Scripts/pytest\n
"},{"location":"04_development/#building-documentation-docforge","title":"\ud83d\udcdd Building documentation (docforge)","text":"

The site is generated by docforge and served per kind under site/{kind}:

doc-forge build \\\n  --mkdocs --mcp --wiki \\\n  --module-is-source --module mail_intake \\\n  --site-name \"Mail Intake\"\n

Preview locally:

doc-forge serve --lib\ndoc-forge serve --wiki\ndoc-forge serve --mcp\n
"},{"location":"04_development/#read-next","title":"\ud83d\udcda Read Next","text":""}]}