omniread/docs/wiki/01_overview.md

🧱 Overview

OmniRead is designed as a decoupled content engine with three distinct layers. Understanding them is the key to using and extending the library.

---

🏗️ Architecture

                 ┌─────────────────────────────┐
                 │  Source (URL, file, storage) │
                 └────────────┬────────────────┘
                              │
                    ┌─────────▼──────────┐
                    │   Scraper / Client  │  fetches raw bytes
                    └─────────┬──────────┘
                              │  returns
                    ┌─────────▼──────────┐
                    │     Content         │  raw + source + type
                    └─────────┬──────────┘
                              │
                    ┌─────────▼──────────┐
                    │      Parser         │  parse() → structured T
                    └────────────────────┘

1. Scraper (BaseScraper) — fetches raw bytes from a source (HTTP URL, filesystem path, object storage). Returns a Content instance. Scrapers never interpret content.
2. Content (Content) — the canonical exchange model: raw bytes, source identifier, and optional ContentType enum.
3. Parser (BaseParser[T]) — receives a Content and returns a structured result of type T via parse().

---

📦 The Content model

Defined in omniread.core.content:

from dataclasses import dataclass
from omniread import Content, ContentType

@dataclass(slots=True)
class Content:
    raw: bytes
    source: str
    content_type: ContentType | None = None

• raw — the raw bytes exactly as retrieved.
• source — URL, file path, or logical name identifying the origin.
• content_type — optional ContentType enum value.

---

🎭 The ContentType enum

Value	MIME	Used by
HTML	text/html	HTMLScraper
PDF	application/pdf	PDFScraper
XLSX	application/vnd.openxmlformats-...	XlsxScraper
CSV	text/csv	CsvScraper
JSON	application/json	—
XML	application/xml	—

---

🧩 Format modules at a glance

Module	Scraper	Parser	Client	Notes
omniread.html	HTMLScraper	HTMLParser	—	httpx + BeautifulSoup
omniread.pdf	PDFScraper	PDFParser	FileSystemPDFClient	explicit client layer
omniread.csv	CsvScraper	CsvParser	FileSystemCsvClient	stdlib csv module
omniread.xlsx	XlsxScraper	XlsxParser	FileSystemXlsxClient	openpyxl-backed

---

📚 Read Next

• How to Use — working examples per format.
• Extending OmniRead — subclassing scrapers and parsers.