aetos/omniread
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updated docs strings and added README.md

Browse Source
This commit is contained in:
Vishesh 'ironeagle' Bangotra
2026-03-08 17:59:56 +05:30
parent 0fbf0ca0f0
commit de7d04eb1a
26 changed files with 546 additions and 406 deletions
Download Patch File Download Diff File Expand all files Collapse all files

123
omniread/__init__.py
View File

@@ -1,99 +1,108 @@
"""
OmniRead — format-agnostic content acquisition and parsing framework.
# Summary
---
`OmniRead` — format-agnostic content acquisition and parsing framework.
## Summary
OmniRead provides a **cleanly layered architecture** for fetching, parsing,
`OmniRead` provides a **cleanly layered architecture** for fetching, parsing,
and normalizing content from heterogeneous sources such as HTML documents
and PDF files.
The library is structured around three core concepts:
1. **Content**: A canonical, format-agnostic container representing raw content bytes and minimal contextual metadata.
2. **Scrapers**: Components responsible for *acquiring* raw content from a source (HTTP, filesystem, object storage, etc.). Scrapers never interpret content.
3. **Parsers**: Components responsible for *interpreting* acquired content and converting it into structured, typed representations.
1. **`Content`**: A canonical, format-agnostic container representing raw content
bytes and minimal contextual metadata.
2. **`Scrapers`**: Components responsible for *acquiring* raw content from a
source (HTTP, filesystem, object storage, etc.). `Scrapers` never interpret
content.
3. **`Parsers`**: Components responsible for *interpreting* acquired content and
converting it into structured, typed representations.
OmniRead deliberately separates these responsibilities to ensure:
- Clear boundaries between IO and interpretation
- Replaceable implementations per format
- Predictable, testable behavior
`OmniRead` deliberately separates these responsibilities to ensure:
---
- Clear boundaries between IO and interpretation.
- Replaceable implementations per format.
- Predictable, testable behavior.
## Installation
# Installation
Install OmniRead using pip:
Install `OmniRead` using pip:
pip install omniread
```bash
pip install omniread
```
Or with Poetry:
poetry add omniread
Install OmniRead using Poetry:
```bash
poetry add omniread
```
---
## Quick start
HTML example:
Example:
HTML example:
```python
from omniread import HTMLScraper, HTMLParser
from omniread import HTMLScraper, HTMLParser
scraper = HTMLScraper()
content = scraper.fetch("https://example.com")
scraper = HTMLScraper()
content = scraper.fetch("https://example.com")
class TitleParser(HTMLParser[str]):
def parse(self) -> str:
return self._soup.title.string
class TitleParser(HTMLParser[str]):
def parse(self) -> str:
return self._soup.title.string
parser = TitleParser(content)
title = parser.parse()
```
parser = TitleParser(content)
title = parser.parse()
PDF example:
```python
from omniread import FileSystemPDFClient, PDFScraper, PDFParser
from pathlib import Path
PDF example:
client = FileSystemPDFClient()
scraper = PDFScraper(client=client)
content = scraper.fetch(Path("document.pdf"))
from omniread import FileSystemPDFClient, PDFScraper, PDFParser
from pathlib import Path
class TextPDFParser(PDFParser[str]):
def parse(self) -> str:
# implement PDF text extraction
...
client = FileSystemPDFClient()
scraper = PDFScraper(client=client)
content = scraper.fetch(Path("document.pdf"))
class TextPDFParser(PDFParser[str]):
def parse(self) -> str:
# implement PDF text extraction
...
parser = TextPDFParser(content)
result = parser.parse()
parser = TextPDFParser(content)
result = parser.parse()
```
---
## Public API
# Public API
This module re-exports the **recommended public entry points** of OmniRead.
Consumers are encouraged to import from this namespace rather than from
format-specific submodules directly, unless advanced customization is
required.
**Core:**
- Content
- ContentType
- `Content`: Canonical content model.
- `ContentType`: Supported media types.
- `HTMLScraper`: HTTP-based HTML acquisition.
- `HTMLParser`: Base parser for HTML DOM interpretation.
- `FileSystemPDFClient`: Local filesystem PDF access.
- `PDFScraper`: PDF-specific content acquisition.
- `PDFParser`: Base parser for PDF binary interpretation.
**HTML:**
- HTMLScraper
- HTMLParser
---
**PDF:**
- FileSystemPDFClient
- PDFScraper
- PDFParser
# Core Philosophy
**Core Philosophy:**
`OmniRead` is designed as a **decoupled content engine**:
1. **Separation of Concerns**: Scrapers *fetch*, Parsers *interpret*. Neither knows about the other.
2. **Normalized Exchange**: All components communicate via the `Content` model, ensuring a consistent contract.
3. **Format Agnosticism**: The core logic is independent of whether the input is HTML, PDF, or JSON.
1. **Separation of Concerns**: Scrapers *fetch*, Parsers *interpret*. Neither
knows about the other.
2. **Normalized Exchange**: All components communicate via the `Content` model,
ensuring a consistent contract.
3. **Format Agnosticism**: The core logic is independent of whether the input
is HTML, PDF, or JSON.
---
"""

19
omniread/core/__init__.py
View File

@@ -1,10 +1,8 @@
"""
# Summary
Core domain contracts for OmniRead.
---
## Summary
This package defines the **format-agnostic domain layer** of OmniRead.
It exposes canonical content models and abstract interfaces that are
implemented by format-specific modules (HTML, PDF, etc.).
@@ -13,18 +11,19 @@ Public exports from this package are considered **stable contracts** and
are safe for downstream consumers to depend on.
Submodules:
- content: Canonical content models and enums
- parser: Abstract parsing contracts
- scraper: Abstract scraping contracts
- `content`: Canonical content models and enums.
- `parser`: Abstract parsing contracts.
- `scraper`: Abstract scraping contracts.
Format-specific behavior must not be introduced at this layer.
---
## Public API
# Public API
Content
ContentType
- `Content`
- `ContentType`
---
"""

18
omniread/core/content.py
View File

@@ -1,10 +1,8 @@
"""
# Summary
Canonical content models for OmniRead.
---
## Summary
This module defines the **format-agnostic content representation** used across
all parsers and scrapers in OmniRead.
@@ -25,8 +23,10 @@ class ContentType(str, Enum):
Notes:
**Guarantees:**
- This enum represents the declared or inferred media type of the content source
- It is primarily used for routing content to the appropriate parser or downstream consumer
- This enum represents the declared or inferred media type of the
content source.
- It is primarily used for routing content to the appropriate
parser or downstream consumer.
"""
HTML = "text/html"
@@ -50,8 +50,10 @@ class Content:
Notes:
**Responsibilities:**
- A `Content` instance represents a raw content payload along with minimal contextual metadata describing its origin and type
- This class is the primary exchange format between Scrapers, Parsers, and Downstream consumers
- A `Content` instance represents a raw content payload along with
minimal contextual metadata describing its origin and type.
- This class is the primary exchange format between scrapers,
parsers, and downstream consumers.
"""
raw: bytes

23
omniread/core/parser.py
View File

@@ -1,19 +1,19 @@
"""
# Summary
Abstract parsing contracts for OmniRead.
---
## Summary
This module defines the **format-agnostic parser interface** used to transform
raw content into structured, typed representations.
Parsers are responsible for:
- Interpreting a single `Content` instance
- Validating compatibility with the content type
- Producing a structured output suitable for downstream consumers
Parsers are not responsible for:
- Fetching or acquiring content
- Performing retries or error recovery
- Managing multiple content sources
@@ -34,14 +34,16 @@ class BaseParser(ABC, Generic[T]):
Notes:
**Guarantees:**
- A parser is a self-contained object that owns the Content it is responsible for interpreting
- Consumers may rely on early validation of content compatibility and type-stable return values from `parse()`
- A parser is a self-contained object that owns the `Content` it is
responsible for interpreting.
- Consumers may rely on early validation of content compatibility
and type-stable return values from `parse()`.
**Responsibilities:**
- Implementations must declare supported content types via `supported_types`
- Implementations must raise parsing-specific exceptions from `parse()`
- Implementations must remain deterministic for a given input
- Implementations must declare supported content types via `supported_types`.
- Implementations must raise parsing-specific exceptions from `parse()`.
- Implementations must remain deterministic for a given input.
"""
supported_types: Set[ContentType] = set()
@@ -86,7 +88,8 @@ class BaseParser(ABC, Generic[T]):
Notes:
**Responsibilities:**
- Implementations must fully consume the provided content and return a deterministic, structured output
- Implementations must fully consume the provided content and
return a deterministic, structured output.
"""
raise NotImplementedError

30
omniread/core/scraper.py
View File

@@ -1,19 +1,19 @@
"""
# Summary
Abstract scraping contracts for OmniRead.
---
## Summary
This module defines the **format-agnostic scraper interface** responsible for
acquiring raw content from external sources.
Scrapers are responsible for:
- Locating and retrieving raw content bytes
- Attaching minimal contextual metadata
- Returning normalized `Content` objects
Scrapers are explicitly NOT responsible for:
- Parsing or interpreting content
- Inferring structure or semantics
- Performing content-type specific processing
@@ -34,14 +34,18 @@ class BaseScraper(ABC):
Notes:
**Responsibilities:**
- A scraper is responsible ONLY for fetching raw content (bytes) from a source. It must not interpret or parse it
- A scraper is a stateless acquisition component that retrieves raw content from a source and returns it as a `Content` object
- Scrapers define how content is obtained, not what the content means
- Implementations may vary in transport mechanism, authentication strategy, retry and backoff behavior
- A scraper is responsible ONLY for fetching raw content (bytes)
from a source. It must not interpret or parse it.
- A scraper is a stateless acquisition component that retrieves raw
content from a source and returns it as a `Content` object.
- Scrapers define how content is obtained, not what the content means.
- Implementations may vary in transport mechanism, authentication
strategy, retry and backoff behavior.
**Constraints:**
- Implementations must not parse content, modify content semantics, or couple scraping logic to a specific parser
- Implementations must not parse content, modify content semantics,
or couple scraping logic to a specific parser.
"""
@abstractmethod
@@ -56,9 +60,10 @@ class BaseScraper(ABC):
Args:
source (str):
Location identifier (URL, file path, S3 URI, etc.)
Location identifier (URL, file path, S3 URI, etc.).
metadata (Optional[Mapping[str, Any]], optional):
Optional hints for the scraper (headers, auth, etc.)
Optional hints for the scraper (headers, auth, etc.).
Returns:
Content:
@@ -71,6 +76,7 @@ class BaseScraper(ABC):
Notes:
**Responsibilities:**
- Implementations must retrieve the content referenced by `source` and return it as raw bytes wrapped in a `Content` object
- Implementations must retrieve the content referenced by `source`
and return it as raw bytes wrapped in a `Content` object.
"""
raise NotImplementedError

26
omniread/html/__init__.py
View File

@@ -1,31 +1,31 @@
"""
# Summary
HTML format implementation for OmniRead.
---
## Summary
This package provides **HTML-specific implementations** of the core OmniRead
contracts defined in `omniread.core`.
It includes:
- HTML parsers that interpret HTML content
- HTML scrapers that retrieve HTML documents
This package:
- Implements, but does not redefine, core contracts
- May contain HTML-specific behavior and edge-case handling
- Produces canonical content models defined in `omniread.core.content`
- HTML parsers that interpret HTML content.
- HTML scrapers that retrieve HTML documents.
Key characteristics:
- Implements, but does not redefine, core contracts.
- May contain HTML-specific behavior and edge-case handling.
- Produces canonical content models defined in `omniread.core.content`.
Consumers should depend on `omniread.core` interfaces wherever possible and
use this package only when HTML-specific behavior is required.
---
## Public API
# Public API
HTMLScraper
HTMLParser
- `HTMLScraper`
- `HTMLParser`
---
"""

30
omniread/html/parser.py
View File

@@ -1,14 +1,13 @@
"""
# Summary
HTML parser base implementations for OmniRead.
---
## Summary
This module provides reusable HTML parsing utilities built on top of
the abstract parser contracts defined in `omniread.core.parser`.
It supplies:
- Content-type enforcement for HTML inputs
- BeautifulSoup initialization and lifecycle management
- Common helper methods for extracting structured data from HTML elements
@@ -35,16 +34,21 @@ class HTMLParser(BaseParser[T], Generic[T]):
Notes:
**Responsibilities:**
- This class extends the core `BaseParser` with HTML-specific behavior, including DOM parsing via BeautifulSoup and reusable extraction helpers
- Provides reusable helpers for HTML extraction. Concrete parsers must explicitly define the return type
- This class extends the core `BaseParser` with HTML-specific behavior,
including DOM parsing via BeautifulSoup and reusable extraction helpers.
- Provides reusable helpers for HTML extraction. Concrete parsers must
explicitly define the return type.
**Guarantees:**
- Characteristics: Accepts only HTML content, owns a parsed BeautifulSoup DOM tree, provides pure helper utilities for common HTML structures
- Accepts only HTML content.
- Owns a parsed BeautifulSoup DOM tree.
- Provides pure helper utilities for common HTML structures.
**Constraints:**
- Concrete subclasses must define the output type `T` and implement the `parse()` method
- Concrete subclasses must define the output type `T` and implement
the `parse()` method.
"""
supported_types = {ContentType.HTML}
@@ -86,7 +90,8 @@ class HTMLParser(BaseParser[T], Generic[T]):
Notes:
**Responsibilities:**
- Implementations must fully interpret the HTML DOM and return a deterministic, structured output
- Implementations must fully interpret the HTML DOM and return a
deterministic, structured output.
"""
raise NotImplementedError
@@ -180,8 +185,9 @@ class HTMLParser(BaseParser[T], Generic[T]):
Notes:
**Responsibilities:**
- Extract high-level metadata from the HTML document
- This includes: Document title, `<meta>` tag name/property → content mappings
- Extract high-level metadata from the HTML document.
- This includes: Document title, `<meta>` tag name/property to
content mappings.
"""
soup = self._soup

22
omniread/html/scraper.py
View File

@@ -1,20 +1,20 @@
"""
# Summary
HTML scraping implementation for OmniRead.
---
## Summary
This module provides an HTTP-based scraper for retrieving HTML documents.
It implements the core `BaseScraper` contract using `httpx` as the transport
layer.
This scraper is responsible for:
- Fetching raw HTML bytes over HTTP(S)
- Validating response content type
- Attaching HTTP metadata to the returned content
This scraper is not responsible for:
- Parsing or interpreting HTML
- Retrying failed requests
- Managing crawl policies or rate limiting
@@ -29,17 +29,21 @@ from omniread.core.scraper import BaseScraper
class HTMLScraper(BaseScraper):
"""
Base HTML scraper using httpx.
Base HTML scraper using `httpx`.
Notes:
**Responsibilities:**
- This scraper retrieves HTML documents over HTTP(S) and returns them as raw content wrapped in a `Content` object
- Fetches raw bytes and metadata only. The scraper uses `httpx.Client` for HTTP requests, enforces an HTML content type, preserves HTTP response metadata
- This scraper retrieves HTML documents over HTTP(S) and returns
them as raw content wrapped in a `Content` object.
- Fetches raw bytes and metadata only.
- The scraper uses `httpx.Client` for HTTP requests, enforces an
HTML content type, and preserves HTTP response metadata.
**Constraints:**
- The scraper does not: Parse HTML, perform retries or backoff, handle non-HTML responses
- The scraper does not: Parse HTML, perform retries or backoff,
handle non-HTML responses.
"""
def __init__(

21
omniread/pdf/__init__.py
View File

@@ -1,29 +1,28 @@
"""
# Summary
PDF format implementation for OmniRead.
---
## Summary
This package provides **PDF-specific implementations** of the core OmniRead
contracts defined in `omniread.core`.
Unlike HTML, PDF handling requires an explicit client layer for document
access. This package therefore includes:
- PDF clients for acquiring raw PDF data
- PDF scrapers that coordinate client access
- PDF parsers that extract structured content from PDF binaries
- PDF clients for acquiring raw PDF data.
- PDF scrapers that coordinate client access.
- PDF parsers that extract structured content from PDF binaries.
Public exports from this package represent the supported PDF pipeline
and are safe for consumers to import directly when working with PDFs.
---
## Public API
# Public API
FileSystemPDFClient
PDFScraper
PDFParser
- `FileSystemPDFClient`
- `PDFScraper`
- `PDFParser`
---
"""

20
omniread/pdf/client.py
View File

@@ -1,10 +1,8 @@
"""
# Summary
PDF client abstractions for OmniRead.
---
## Summary
This module defines the **client layer** responsible for retrieving raw PDF
bytes from a concrete backing store.
@@ -13,6 +11,7 @@ decoupled from scraping and parsing logic. They do not perform validation,
interpretation, or content extraction.
Typical backing stores include:
- Local filesystems
- Object storage (S3, GCS, etc.)
- Network file systems
@@ -25,13 +24,17 @@ from pathlib import Path
class BasePDFClient(ABC):
"""
Abstract client responsible for retrieving PDF bytes
from a specific backing store (filesystem, S3, FTP, etc.).
Abstract client responsible for retrieving PDF bytes.
Retrieves bytes from a specific backing store (filesystem, S3, FTP, etc.).
Notes:
**Responsibilities:**
- Implementations must accept a source identifier appropriate to the backing store, return the full PDF binary payload, and raise retrieval-specific errors on failure
- Implementations must accept a source identifier appropriate to
the backing store.
- Return the full PDF binary payload.
- Raise retrieval-specific errors on failure.
"""
@abstractmethod
@@ -61,7 +64,8 @@ class FileSystemPDFClient(BasePDFClient):
Notes:
**Guarantees:**
- This client reads PDF files directly from the disk and returns their raw binary contents
- This client reads PDF files directly from the disk and returns
their raw binary contents.
"""
def fetch(self, path: Path) -> bytes:

15
omniread/pdf/parser.py
View File

@@ -1,10 +1,8 @@
"""
# Summary
PDF parser base implementations for OmniRead.
---
## Summary
This module defines the **PDF-specific parser contract**, extending the
format-agnostic `BaseParser` with constraints appropriate for PDF content.
@@ -28,11 +26,13 @@ class PDFParser(BaseParser[T], Generic[T]):
Notes:
**Responsibilities:**
- This class enforces PDF content-type compatibility and provides the extension point for implementing concrete PDF parsing strategies
- This class enforces PDF content-type compatibility and provides
the extension point for implementing concrete PDF parsing strategies.
**Constraints:**
- Concrete implementations must: Define the output type `T`, implement the `parse()` method
- Concrete implementations must define the output type `T` and
implement the `parse()` method.
"""
supported_types = {ContentType.PDF}
@@ -56,6 +56,7 @@ class PDFParser(BaseParser[T], Generic[T]):
Notes:
**Responsibilities:**
- Implementations must fully interpret the PDF binary payload and return a deterministic, structured output
- Implementations must fully interpret the PDF binary payload and
return a deterministic, structured output.
"""
raise NotImplementedError

16
omniread/pdf/scraper.py
View File

@@ -1,10 +1,8 @@
"""
# Summary
PDF scraping implementation for OmniRead.
---
## Summary
This module provides a PDF-specific scraper that coordinates PDF byte
retrieval via a client and normalizes the result into a `Content` object.
@@ -26,12 +24,14 @@ class PDFScraper(BaseScraper):
Notes:
**Responsibilities:**
- Delegates byte retrieval to a PDF client and normalizes output into Content
- Preserves caller-provided metadata
- Delegates byte retrieval to a PDF client and normalizes output
into `Content`.
- Preserves caller-provided metadata.
**Constraints:**
- The scraper: Does not perform parsing or interpretation, does not assume a specific storage backend
- The scraper does not perform parsing or interpretation.
- Does not assume a specific storage backend.
"""
def __init__(self, *, client: BasePDFClient):