Skip to content

🧱 Overview

OmniRead is designed as a decoupled content engine with three distinct layers. Understanding them is the key to using and extending the library.

🏗️ Architecture

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
                 ┌─────────────────────────────┐
                 │  Source (URL, file, storage) │
                 └────────────┬────────────────┘
                              │
                    ┌─────────▼──────────┐
                    │   Scraper / Client  │  fetches raw bytes
                    └─────────┬──────────┘
                              │  returns
                    ┌─────────▼──────────┐
                    │     Content         │  raw + source + type
                    └─────────┬──────────┘
                              │
                    ┌─────────▼──────────┐
                    │      Parser         │  parse() → structured T
                    └────────────────────┘
  1. Scraper (BaseScraper) — fetches raw bytes from a source (HTTP URL, filesystem path, object storage). Returns a Content instance. Scrapers never interpret content.
  2. Content (Content) — the canonical exchange model: raw bytes, source identifier, and optional ContentType enum.
  3. Parser (BaseParser[T]) — receives a Content and returns a structured result of type T via parse().

📦 The Content model

Defined in omniread.core.content:

1
2
3
4
5
6
7
8
from dataclasses import dataclass
from omniread import Content, ContentType

@dataclass(slots=True)
class Content:
    raw: bytes
    source: str
    content_type: ContentType | None = None
  • raw — the raw bytes exactly as retrieved.
  • source — URL, file path, or logical name identifying the origin.
  • content_type — optional ContentType enum value.

🎭 The ContentType enum

Value MIME Used by
HTML text/html HTMLScraper
PDF application/pdf PDFScraper
XLSX application/vnd.openxmlformats-... XlsxScraper
CSV text/csv CsvScraper
JSON application/json —
XML application/xml —

🧩 Format modules at a glance

Module Scraper Parser Client Notes
omniread.html HTMLScraper HTMLParser — httpx + BeautifulSoup
omniread.pdf PDFScraper PDFParser FileSystemPDFClient explicit client layer
omniread.csv CsvScraper CsvParser FileSystemCsvClient stdlib csv module
omniread.xlsx XlsxScraper XlsxParser FileSystemXlsxClient openpyxl-backed