docs: add wiki and refresh flat lib with mcp artifacts
- Add hand-written wiki (index, overview, how-to, extending, dev) following the platform anatomy - Remove stale nested docs/lib/mail_intake and regenerate the flat lib reference to match docforge.nav.yml - Regenerate MCP bundle with standardized docstrings
This commit is contained in:
91
docs/wiki/01_overview.md
Normal file
91
docs/wiki/01_overview.md
Normal file
@@ -0,0 +1,91 @@
|
||||
# 🧱 Overview
|
||||
|
||||
Mail Intake is a **contract-first ingestion pipeline**. Adapters handle
|
||||
transport to a provider, parsers normalize provider payloads, and the reader
|
||||
orchestrates the whole flow into canonical domain models.
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Architecture
|
||||
|
||||
```text
|
||||
┌─────────────────────────────────────────┐
|
||||
│ External Provider (e.g. Gmail API) │
|
||||
└─────────────────────┬───────────────────┘
|
||||
│
|
||||
┌─────────────────▼──────────────────┐
|
||||
│ MailIntakeAdapter (transport) │ provider API calls
|
||||
└─────────────────┬──────────────────┘
|
||||
│ provider-native payloads
|
||||
┌─────────────────▼──────────────────┐
|
||||
│ Parsers (normalization) │ headers, body, subject
|
||||
└─────────────────┬──────────────────┘
|
||||
│ composed
|
||||
┌─────────────────▼──────────────────┐
|
||||
│ MailIntakeReader (orchestration) │ iter_messages / iter_threads
|
||||
└─────────────────┬──────────────────┘
|
||||
│
|
||||
┌─────────────────▼──────────────────┐
|
||||
│ MailIntakeMessage / Thread │ canonical domain models
|
||||
└────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Layers:
|
||||
|
||||
1. **Adapters** (`mail_intake.adapters`) — provider-specific, read-only
|
||||
transport. Return provider-native payloads; never interpret them.
|
||||
2. **Auth** (`mail_intake.auth`) — credential acquisition and lifecycle
|
||||
management, decoupled from adapters.
|
||||
3. **Credentials** (`mail_intake.credentials`) — persistence of auth tokens;
|
||||
`PickleCredentialStore` locally, `RedisCredentialStore` for production.
|
||||
4. **Parsers** (`mail_intake.parsers`) — extract headers, body text, sender,
|
||||
and normalized subjects from provider payloads.
|
||||
5. **Ingestion** (`mail_intake.ingestion`) — `MailIntakeReader` wires an
|
||||
adapter + parsers into iterators over canonical models.
|
||||
6. **Models** (`mail_intake.models`) — provider-agnostic `MailIntakeMessage`
|
||||
and `MailIntakeThread`.
|
||||
|
||||
---
|
||||
|
||||
## 📦 Domain models
|
||||
|
||||
`MailIntakeMessage`:
|
||||
|
||||
| Field | Type | Meaning |
|
||||
|---|---|---|
|
||||
| `message_id` | `str` | Provider message id |
|
||||
| `thread_id` | `str` | Conversation thread id |
|
||||
| `timestamp` | `datetime` | Message timestamp |
|
||||
| `from_email` | `str` | Sender email |
|
||||
| `from_name` | `str \| None` | Sender display name |
|
||||
| `subject` | `str` | Message subject |
|
||||
| `body_text` | `str` | Extracted plain-text body |
|
||||
| `snippet` | `str` | Provider snippet |
|
||||
| `raw_headers` | `dict[str, str]` | Unmodified headers |
|
||||
|
||||
`MailIntakeThread`:
|
||||
|
||||
| Field | Type | Meaning |
|
||||
|---|---|---|
|
||||
| `thread_id` | `str` | Conversation id |
|
||||
| `normalized_subject` | `str` | Normalized subject (threads share one) |
|
||||
| `participants` | `set[str]` | Distinct senders |
|
||||
| `messages` | `list[MailIntakeMessage]` | Ordered messages |
|
||||
| `last_activity_at` | `datetime \| None` | Latest message time |
|
||||
|
||||
---
|
||||
|
||||
## 🔒 Design guarantees
|
||||
|
||||
- Read-only access — no mutation of provider state.
|
||||
- Provider-agnostic domain models.
|
||||
- Explicit configuration and dependency injection (no implicit env reads).
|
||||
- Extensible via public contracts; built-in adapters are reference
|
||||
implementations and may change internally.
|
||||
|
||||
---
|
||||
|
||||
## 📚 Read Next
|
||||
|
||||
- [How to Use](02_how_to_use.md) — the Gmail ingestion flow.
|
||||
- [Extending Mail Intake](03_extending.md) — custom adapters and stores.
|
||||
105
docs/wiki/02_how_to_use.md
Normal file
105
docs/wiki/02_how_to_use.md
Normal file
@@ -0,0 +1,105 @@
|
||||
# 🖥️ How to Use
|
||||
|
||||
This page walks through authenticating, ingesting, and parsing mail with the
|
||||
built-in Gmail support.
|
||||
|
||||
---
|
||||
|
||||
## 🔐 Authentication
|
||||
|
||||
Create a credential store and an auth provider:
|
||||
|
||||
```python
|
||||
from mail_intake.auth import MailIntakeGoogleAuth
|
||||
from mail_intake.credentials import PickleCredentialStore, RedisCredentialStore
|
||||
|
||||
store = PickleCredentialStore(path="token.pickle") # local dev
|
||||
|
||||
auth = MailIntakeGoogleAuth(
|
||||
credentials_path="credentials.json", # your OAuth client file
|
||||
store=store,
|
||||
scopes=["https://www.googleapis.com/auth/gmail.readonly"],
|
||||
)
|
||||
```
|
||||
|
||||
> OAuth credentials come from the Google Cloud Console and are provided via
|
||||
> file paths — never hard-code tokens or secrets in source. Once authorized,
|
||||
> the token is persisted by the credential store and refreshed automatically.
|
||||
|
||||
For distributed deployments, use Redis instead of pickle:
|
||||
|
||||
```python
|
||||
store = RedisCredentialStore(redis_client=redis_client) # production
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📥 Ingesting messages
|
||||
|
||||
Build an adapter and a reader, then iterate:
|
||||
|
||||
```python
|
||||
from mail_intake.ingestion import MailIntakeReader
|
||||
from mail_intake.adapters import MailIntakeGmailAdapter
|
||||
|
||||
adapter = MailIntakeGmailAdapter(auth_provider=auth)
|
||||
reader = MailIntakeReader(adapter)
|
||||
|
||||
for message in reader.iter_messages("from:recruiter@example.com"):
|
||||
print(message.subject, message.from_email, message.timestamp)
|
||||
```
|
||||
|
||||
Access the full model:
|
||||
|
||||
```python
|
||||
message.message_id # provider id
|
||||
message.thread_id
|
||||
message.body_text # plain-text body
|
||||
message.raw_headers # unmodified headers dict
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔖 Ingesting threads
|
||||
|
||||
```python
|
||||
for thread in reader.iter_threads("subject:Interview"):
|
||||
print(thread.normalized_subject)
|
||||
print(thread.participants)
|
||||
print(len(thread.messages))
|
||||
print(thread.last_activity_at)
|
||||
```
|
||||
|
||||
`MailIntakeThread` aggregates its messages, participants, and last activity.
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Parsers
|
||||
|
||||
Parsers normalize provider payloads and are used by the reader internally.
|
||||
They are also importable on their own:
|
||||
|
||||
```python
|
||||
from mail_intake.parsers import extract_body, parse_headers, normalize_subject
|
||||
```
|
||||
|
||||
- `extract_body(...)` — plain-text body extraction.
|
||||
- `parse_headers(...)` — structured header parsing.
|
||||
- `extract_sender(...)` — sender email/name extraction.
|
||||
- `normalize_subject(...)` — subject normalization for threading.
|
||||
|
||||
---
|
||||
|
||||
## ✅ Checklist
|
||||
|
||||
1. Provide OAuth credentials via a file path and a credential store.
|
||||
2. Build the `MailIntakeGoogleAuth` provider with the Gmail read-only scope.
|
||||
3. Wrap the adapter in a `MailIntakeReader`.
|
||||
4. Consume `iter_messages` / `iter_threads`; never call provider APIs directly.
|
||||
|
||||
---
|
||||
|
||||
## 📚 Read Next
|
||||
|
||||
- [Overview](01_overview.md) — layers and domain models.
|
||||
- [Extending Mail Intake](03_extending.md) — custom providers and stores.
|
||||
106
docs/wiki/03_extending.md
Normal file
106
docs/wiki/03_extending.md
Normal file
@@ -0,0 +1,106 @@
|
||||
# 🧩 Extending Mail Intake
|
||||
|
||||
Mail Intake is designed to be extended through its **public contracts**.
|
||||
Implement your own adapter, auth provider, or credential store to support a
|
||||
new provider or storage backend.
|
||||
|
||||
---
|
||||
|
||||
## 📬 Custom adapters
|
||||
|
||||
Subclass `MailIntakeAdapter` to integrate a new provider. Adapters perform
|
||||
read-only transport and return provider-native payloads:
|
||||
|
||||
```python
|
||||
from mail_intake.adapters import MailIntakeAdapter
|
||||
|
||||
class ExchangeAdapter(MailIntakeAdapter):
|
||||
def __init__(self, auth_provider):
|
||||
self._auth = auth_provider
|
||||
|
||||
def fetch_messages(self, query):
|
||||
# call the provider API, return native payloads
|
||||
...
|
||||
|
||||
def fetch_threads(self, query):
|
||||
...
|
||||
```
|
||||
|
||||
> **Do not** subclass built-in adapters like `MailIntakeGmailAdapter` — they
|
||||
> are reference implementations and may change internally without notice.
|
||||
|
||||
---
|
||||
|
||||
## 🔐 Custom auth providers
|
||||
|
||||
Subclass `MailIntakeAuthProvider[T]` to own a different credential flow:
|
||||
|
||||
```python
|
||||
from mail_intake.auth import MailIntakeAuthProvider
|
||||
|
||||
class ExchangeAuth(MailIntakeAuthProvider[exchange_credentials]):
|
||||
def get_credentials(self):
|
||||
return self._store.load()
|
||||
|
||||
def refresh(self):
|
||||
...
|
||||
```
|
||||
|
||||
Auth providers remain decoupled from adapter logic — they only manage
|
||||
credentials.
|
||||
|
||||
---
|
||||
|
||||
## 🗄️ Custom credential stores
|
||||
|
||||
Implement the `CredentialStore[T]` contract for a new persistence backend:
|
||||
|
||||
```python
|
||||
from mail_intake.credentials import CredentialStore
|
||||
|
||||
class S3CredentialStore(CredentialStore):
|
||||
def save(self, credentials):
|
||||
...
|
||||
|
||||
def load(self):
|
||||
...
|
||||
|
||||
def delete(self):
|
||||
...
|
||||
```
|
||||
|
||||
The store abstraction keeps tokens out of config and rotates safely.
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Wiring a custom backend
|
||||
|
||||
```python
|
||||
from mail_intake.ingestion import MailIntakeReader
|
||||
|
||||
auth = ExchangeAuth(store=S3CredentialStore(...))
|
||||
adapter = ExchangeAdapter(auth_provider=auth)
|
||||
reader = MailIntakeReader(adapter)
|
||||
|
||||
for message in reader.iter_messages("query"):
|
||||
print(message.subject)
|
||||
```
|
||||
|
||||
The reader only depends on the adapter contract, so the rest of the pipeline
|
||||
keeps working unchanged.
|
||||
|
||||
---
|
||||
|
||||
## ✅ Extension checklist
|
||||
|
||||
1. Implement the **public contract** — never subclass built-in adapters.
|
||||
2. Keep transport in the adapter, parsing in parsers, auth in the provider.
|
||||
3. Return the provider-native payload and let parsers normalize it.
|
||||
4. Inject dependencies explicitly — no global state or env reads.
|
||||
|
||||
---
|
||||
|
||||
## 📚 Read Next
|
||||
|
||||
- [How to Use](02_how_to_use.md) — the built-in Gmail flow.
|
||||
- [Development](04_development.md) — running tests and docs.
|
||||
88
docs/wiki/04_development.md
Normal file
88
docs/wiki/04_development.md
Normal file
@@ -0,0 +1,88 @@
|
||||
# 🛠️ Development
|
||||
|
||||
Working on `mail-intake` itself.
|
||||
|
||||
---
|
||||
|
||||
## 📂 Repository layout
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `mail_intake/` | The library package (adapters, auth, credentials, parsers, ingestion, models) |
|
||||
| `mail_intake/*.pyi` | Type stubs kept in sync with implementations |
|
||||
| `tests/` | Unit and integration tests (mock transports, no live mail) |
|
||||
| `docs/lib/` | Generated library reference (docforge, flat layout) |
|
||||
| `docs/mcp/` | Machine-readable bundle served by the MCP server |
|
||||
| `docs/wiki/` | This hand-written wiki |
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Setup
|
||||
|
||||
```bash
|
||||
python -m venv .venv
|
||||
.venv/Scripts/pip install -e ".[dev]"
|
||||
```
|
||||
|
||||
> OAuth credential files (`credentials*.json`, `token.pickle`,
|
||||
> `client_secret_*.json`) are gitignored — keep them out of the repository.
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Tests
|
||||
|
||||
Run the suite (no network or live Gmail required):
|
||||
|
||||
```bash
|
||||
.venv/Scripts/pytest
|
||||
```
|
||||
|
||||
Coverage spans ingestion flows, credential stores, parsers, and auth against
|
||||
mock providers.
|
||||
|
||||
---
|
||||
|
||||
## ✅ Quality gates
|
||||
|
||||
The CI quality gate runs, matching the Drone pipeline:
|
||||
|
||||
```bash
|
||||
.venv/Scripts/black --check .
|
||||
.venv/Scripts/ruff check .
|
||||
.venv/Scripts/mypy
|
||||
.venv/Scripts/pytest
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📝 Building documentation (docforge)
|
||||
|
||||
The site is generated by [`docforge`](https://git.aetoskia.com/aetos/doc-forge)
|
||||
and served per kind under `site/{kind}`:
|
||||
|
||||
```bash
|
||||
doc-forge build \
|
||||
--mkdocs --mcp --wiki \
|
||||
--module-is-source --module mail_intake \
|
||||
--site-name "Mail Intake"
|
||||
```
|
||||
|
||||
- `--module-is-source` renders the flat `docs/lib/` layout (no nesting under
|
||||
`mail_intake/`), matching `docforge.nav.yml` and `docs/mkdocs.lib.yml`.
|
||||
- `--mcp` regenerates the structured bundle in `docs/mcp/`.
|
||||
- `--wiki` builds this wiki.
|
||||
|
||||
Preview locally:
|
||||
|
||||
```bash
|
||||
doc-forge serve --lib
|
||||
doc-forge serve --wiki
|
||||
doc-forge serve --mcp
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📚 Read Next
|
||||
|
||||
- [Extending Mail Intake](03_extending.md) — custom adapters and stores.
|
||||
- [Overview](01_overview.md) — the core architecture.
|
||||
71
docs/wiki/index.md
Normal file
71
docs/wiki/index.md
Normal file
@@ -0,0 +1,71 @@
|
||||
# 📬 Mail Intake — Provider-Agnostic Email Ingestion
|
||||
|
||||
`Mail Intake` is a contract-first, read-only email ingestion framework. It
|
||||
pulls mail from external providers (such as Gmail), parses and normalizes it
|
||||
into clean, provider-agnostic domain models — ready to persist, index, or
|
||||
analyze downstream.
|
||||
|
||||
> **Doc model:** this wiki is written for humans — how‑to guides and extension
|
||||
> recipes. The authoritative API contracts live in the code (GSDFC docstrings)
|
||||
> and the machine‑readable bundle under `docs/mcp/`.
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Key Features
|
||||
|
||||
* 📬 **Read-only ingestion** — never mutates provider state
|
||||
* 🧩 **Contract-first layers** — adapters, parsers, and readers separated
|
||||
* ✉️ **Provider-agnostic models** — `MailIntakeMessage` / `MailIntakeThread`
|
||||
have no provider internals
|
||||
* 🔐 **Extensible auth** — pluggable auth providers and credential stores
|
||||
(pickle for dev, Redis for production)
|
||||
* 🧪 **Deterministic & testable** — no implicit global state or env reads
|
||||
* 📊 **Gmail support** — reference adapter built on the official Google APIs
|
||||
|
||||
---
|
||||
|
||||
## ⚡ Quick Start
|
||||
|
||||
```python
|
||||
from mail_intake.ingestion import MailIntakeReader
|
||||
from mail_intake.adapters import MailIntakeGmailAdapter
|
||||
from mail_intake.auth import MailIntakeGoogleAuth
|
||||
from mail_intake.credentials import PickleCredentialStore
|
||||
|
||||
store = PickleCredentialStore(path="token.pickle")
|
||||
|
||||
auth = MailIntakeGoogleAuth(
|
||||
credentials_path="credentials.json",
|
||||
store=store,
|
||||
scopes=["https://www.googleapis.com/auth/gmail.readonly"],
|
||||
)
|
||||
|
||||
adapter = MailIntakeGmailAdapter(auth_provider=auth)
|
||||
reader = MailIntakeReader(adapter)
|
||||
|
||||
for message in reader.iter_messages("from:recruiter@example.com"):
|
||||
print(message.subject, message.from_email)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📁 Documentation Structure
|
||||
|
||||
| Section | What you'll find |
|
||||
|---|---|
|
||||
| [Overview](01_overview.md) | Layers, domain models, and design guarantees |
|
||||
| [How to Use](02_how_to_use.md) | Gmail ingestion, parsing, and credential stores |
|
||||
| [Extending Mail Intake](03_extending.md) | Custom adapters, auth providers, stores |
|
||||
| [Development](04_development.md) | Setup, tests, and regenerating docs |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 Related Resources
|
||||
|
||||
* **Source Code:** [Gitea Repository](https://git.aetoskia.com/aetos/mail-intake)
|
||||
* **Internal PyPI:** [pip.aetoskia.com/simple/mail-intake](https://pip.aetoskia.com/simple/mail-intake)
|
||||
* **CI:** Builds and publishes tagged releases, gated on black / ruff / mypy / pytest.
|
||||
|
||||
---
|
||||
|
||||
© Aetoskia Internal — `mail-intake` 0.0.2
|
||||
Reference in New Issue
Block a user