docs: add wiki, complete lib nav, and rebuild mcp artifacts
- Add hand-written wiki (index, overview, how-to, extending, dev) with MkDocs config following the platform anatomy - Complete the library nav by registering the csv and xlsx groups in docforge.nav.yml and docs/mkdocs.lib.yml - Regenerate lib/MCP outputs with the rebuilt nav
This commit is contained in:
86
docs/wiki/01_overview.md
Normal file
86
docs/wiki/01_overview.md
Normal file
@@ -0,0 +1,86 @@
|
||||
# 🧱 Overview
|
||||
|
||||
OmniRead is designed as a **decoupled content engine** with three distinct
|
||||
layers. Understanding them is the key to using and extending the library.
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Architecture
|
||||
|
||||
```text
|
||||
┌─────────────────────────────┐
|
||||
│ Source (URL, file, storage) │
|
||||
└────────────┬────────────────┘
|
||||
│
|
||||
┌─────────▼──────────┐
|
||||
│ Scraper / Client │ fetches raw bytes
|
||||
└─────────┬──────────┘
|
||||
│ returns
|
||||
┌─────────▼──────────┐
|
||||
│ Content │ raw + source + type
|
||||
└─────────┬──────────┘
|
||||
│
|
||||
┌─────────▼──────────┐
|
||||
│ Parser │ parse() → structured T
|
||||
└────────────────────┘
|
||||
```
|
||||
|
||||
1. **Scraper** (`BaseScraper`) — fetches raw bytes from a source (HTTP URL,
|
||||
filesystem path, object storage). Returns a `Content` instance.
|
||||
Scrapers never interpret content.
|
||||
2. **Content** (`Content`) — the canonical exchange model: raw bytes,
|
||||
source identifier, and optional `ContentType` enum.
|
||||
3. **Parser** (`BaseParser[T]`) — receives a `Content` and returns a
|
||||
structured result of type `T` via `parse()`.
|
||||
|
||||
---
|
||||
|
||||
## 📦 The `Content` model
|
||||
|
||||
Defined in `omniread.core.content`:
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass
|
||||
from omniread import Content, ContentType
|
||||
|
||||
@dataclass(slots=True)
|
||||
class Content:
|
||||
raw: bytes
|
||||
source: str
|
||||
content_type: ContentType | None = None
|
||||
```
|
||||
|
||||
- `raw` — the raw bytes exactly as retrieved.
|
||||
- `source` — URL, file path, or logical name identifying the origin.
|
||||
- `content_type` — optional `ContentType` enum value.
|
||||
|
||||
---
|
||||
|
||||
## 🎭 The `ContentType` enum
|
||||
|
||||
| Value | MIME | Used by |
|
||||
|---|---|---|
|
||||
| `HTML` | `text/html` | HTMLScraper |
|
||||
| `PDF` | `application/pdf` | PDFScraper |
|
||||
| `XLSX` | `application/vnd.openxmlformats-...` | XlsxScraper |
|
||||
| `CSV` | `text/csv` | CsvScraper |
|
||||
| `JSON` | `application/json` | — |
|
||||
| `XML` | `application/xml` | — |
|
||||
|
||||
---
|
||||
|
||||
## 🧩 Format modules at a glance
|
||||
|
||||
| Module | Scraper | Parser | Client | Notes |
|
||||
|---|---|---|---|---|
|
||||
| `omniread.html` | `HTMLScraper` | `HTMLParser` | — | httpx + BeautifulSoup |
|
||||
| `omniread.pdf` | `PDFScraper` | `PDFParser` | `FileSystemPDFClient` | explicit client layer |
|
||||
| `omniread.csv` | `CsvScraper` | `CsvParser` | `FileSystemCsvClient` | stdlib csv module |
|
||||
| `omniread.xlsx` | `XlsxScraper` | `XlsxParser` | `FileSystemXlsxClient` | openpyxl-backed |
|
||||
|
||||
---
|
||||
|
||||
## 📚 Read Next
|
||||
|
||||
- [How to Use](02_how_to_use.md) — working examples per format.
|
||||
- [Extending OmniRead](03_extending.md) — subclassing scrapers and parsers.
|
||||
Reference in New Issue
Block a user