Vishesh 'ironeagle' Bangotra
b984dd5f42
- Add hand-written wiki (index, overview, how-to, extending, dev) with MkDocs config following the platform anatomy - Complete the library nav by registering the csv and xlsx groups in docforge.nav.yml and docs/mkdocs.lib.yml - Regenerate lib/MCP outputs with the rebuilt nav
86 lines
3.1 KiB
Markdown
86 lines
3.1 KiB
Markdown
# 🧱 Overview
|
|
|
|
OmniRead is designed as a **decoupled content engine** with three distinct
|
|
layers. Understanding them is the key to using and extending the library.
|
|
|
|
---
|
|
|
|
## 🏗️ Architecture
|
|
|
|
```text
|
|
┌─────────────────────────────┐
|
|
│ Source (URL, file, storage) │
|
|
└────────────┬────────────────┘
|
|
│
|
|
┌─────────▼──────────┐
|
|
│ Scraper / Client │ fetches raw bytes
|
|
└─────────┬──────────┘
|
|
│ returns
|
|
┌─────────▼──────────┐
|
|
│ Content │ raw + source + type
|
|
└─────────┬──────────┘
|
|
│
|
|
┌─────────▼──────────┐
|
|
│ Parser │ parse() → structured T
|
|
└────────────────────┘
|
|
```
|
|
|
|
1. **Scraper** (`BaseScraper`) — fetches raw bytes from a source (HTTP URL,
|
|
filesystem path, object storage). Returns a `Content` instance.
|
|
Scrapers never interpret content.
|
|
2. **Content** (`Content`) — the canonical exchange model: raw bytes,
|
|
source identifier, and optional `ContentType` enum.
|
|
3. **Parser** (`BaseParser[T]`) — receives a `Content` and returns a
|
|
structured result of type `T` via `parse()`.
|
|
|
|
---
|
|
|
|
## 📦 The `Content` model
|
|
|
|
Defined in `omniread.core.content`:
|
|
|
|
```python
|
|
from dataclasses import dataclass
|
|
from omniread import Content, ContentType
|
|
|
|
@dataclass(slots=True)
|
|
class Content:
|
|
raw: bytes
|
|
source: str
|
|
content_type: ContentType | None = None
|
|
```
|
|
|
|
- `raw` — the raw bytes exactly as retrieved.
|
|
- `source` — URL, file path, or logical name identifying the origin.
|
|
- `content_type` — optional `ContentType` enum value.
|
|
|
|
---
|
|
|
|
## 🎭 The `ContentType` enum
|
|
|
|
| Value | MIME | Used by |
|
|
|---|---|---|
|
|
| `HTML` | `text/html` | HTMLScraper |
|
|
| `PDF` | `application/pdf` | PDFScraper |
|
|
| `XLSX` | `application/vnd.openxmlformats-...` | XlsxScraper |
|
|
| `CSV` | `text/csv` | CsvScraper |
|
|
| `JSON` | `application/json` | — |
|
|
| `XML` | `application/xml` | — |
|
|
|
|
---
|
|
|
|
## 🧩 Format modules at a glance
|
|
|
|
| Module | Scraper | Parser | Client | Notes |
|
|
|---|---|---|---|---|
|
|
| `omniread.html` | `HTMLScraper` | `HTMLParser` | — | httpx + BeautifulSoup |
|
|
| `omniread.pdf` | `PDFScraper` | `PDFParser` | `FileSystemPDFClient` | explicit client layer |
|
|
| `omniread.csv` | `CsvScraper` | `CsvParser` | `FileSystemCsvClient` | stdlib csv module |
|
|
| `omniread.xlsx` | `XlsxScraper` | `XlsxParser` | `FileSystemXlsxClient` | openpyxl-backed |
|
|
|
|
---
|
|
|
|
## 📚 Read Next
|
|
|
|
- [How to Use](02_how_to_use.md) — working examples per format.
|
|
- [Extending OmniRead](03_extending.md) — subclassing scrapers and parsers. |