Skip to content

# omniread

## `` omniread

### Summary

`OmniRead` — format-agnostic content acquisition and parsing framework.

`OmniRead` provides a cleanly layered architecture for fetching, parsing, and normalizing content from heterogeneous sources such as HTML documents and PDF files.

The library is structured around three core concepts:

1. `Content`: A canonical, format-agnostic container representing raw content bytes and minimal contextual metadata.
2. `Scrapers`: Components responsible for acquiring raw content from a source (HTTP, filesystem, object storage, etc.). `Scrapers` never interpret content.
3. `Parsers`: Components responsible for interpreting acquired content and converting it into structured, typed representations.

`OmniRead` deliberately separates these responsibilities to ensure:

- Clear boundaries between IO and interpretation.
- Replaceable implementations per format.
- Predictable, testable behavior.

### Installation

Install `OmniRead` using pip:

    1

    pip install omniread

Install OmniRead using Poetry:

    1

    poetry add omniread

#### Quick start

Example

HTML example:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11

    from omniread import HTMLScraper, HTMLParser

    scraper = HTMLScraper()
    content = scraper.fetch("https://example.com")

    class TitleParser(HTMLParser[str]):
        def parse(self) -> str:
            return self._soup.title.string

    parser = TitleParser(content)
    title = parser.parse()

PDF example:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14

    from omniread import FileSystemPDFClient, PDFScraper, PDFParser
    from pathlib import Path

    client = FileSystemPDFClient()
    scraper = PDFScraper(client=client)
    content = scraper.fetch(Path("document.pdf"))

    class TextPDFParser(PDFParser[str]):
        def parse(self) -> str:
            # implement PDF text extraction
            ...

    parser = TextPDFParser(content)
    result = parser.parse()

### Public API

This module re-exports the recommended public entry points of OmniRead. Consumers are encouraged to import from this namespace rather than from format-specific submodules directly, unless advanced customization is required.

- `Content`: Canonical content model.
- `ContentType`: Supported media types.
- `HTMLScraper`: HTTP-based HTML acquisition.
- `HTMLParser`: Base parser for HTML DOM interpretation.
- `FileSystemPDFClient`: Local filesystem PDF access.
- `PDFScraper`: PDF-specific content acquisition.
- `PDFParser`: Base parser for PDF binary interpretation.

### Core Philosophy

`OmniRead` is designed as a decoupled content engine:

1. Separation of Concerns: Scrapers fetch, Parsers interpret. Neither knows about the other.
2. Normalized Exchange: All components communicate via the `Content` model, ensuring a consistent contract.
3. Format Agnosticism: The core logic is independent of whether the input is HTML, PDF, or JSON.

### Classes

#### `` Content `dataclass`

    1

    Content(raw: bytes, source: str, content_type: Optional[ContentType] = ..., metadata: Optional[Mapping[str, Any]] = ...)

Normalized representation of extracted content.

Notes

Responsibilities:

    1
    2
    3
    4

    - A `Content` instance represents a raw content payload along with
      minimal contextual metadata describing its origin and type.
    - This class is the primary exchange format between scrapers,
      parsers, and downstream consumers.

##### Attributes

###### `` content_type `class-attribute` `instance-attribute`

    1

    content_type: Optional[ContentType] = None

Optional MIME type of the content, if known.

###### `` metadata `class-attribute` `instance-attribute`

    1

    metadata: Optional[Mapping[str, Any]] = None

Optional, implementation-defined metadata associated with the content (e.g., headers, encoding hints, extraction notes).

###### `` raw `instance-attribute`

    1

    raw: bytes

Raw content bytes as retrieved from the source.

###### `` source `instance-attribute`

    1

    source: str

Identifier of the content origin (URL, file path, or logical name).

#### `` ContentType

Bases: `str`, `Enum`

Supported MIME types for extracted content.

Notes

Guarantees:

    1
    2
    3
    4

    - This enum represents the declared or inferred media type of the
      content source.
    - It is primarily used for routing content to the appropriate
      parser or downstream consumer.

##### Attributes

###### `` HTML `class-attribute` `instance-attribute`

    1

    HTML = 'text/html'

HTML document content.

###### `` JSON `class-attribute` `instance-attribute`

    1

    JSON = 'application/json'

JSON document content.

###### `` PDF `class-attribute` `instance-attribute`

    1

    PDF = 'application/pdf'

PDF document content.

###### `` XML `class-attribute` `instance-attribute`

    1

    XML = 'application/xml'

XML document content.

#### `` FileSystemPDFClient

Bases: `BasePDFClient`

PDF client that reads from the local filesystem.

Notes

Guarantees:

    1
    2

    - This client reads PDF files directly from the disk and returns
      their raw binary contents.

##### Functions

###### `` fetch

    1

    fetch(path: Path) -> bytes

Read a PDF file from the local filesystem.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`path` | `Path` | Filesystem path to the PDF file. | required

Returns:

Name | Type | Description
--- | --- | ---
`bytes` | `bytes` | Raw PDF bytes.

Raises:

Type | Description
--- | ---
`FileNotFoundError` | If the path does not exist.
`ValueError` | If the path exists but is not a file.

#### `` HTMLParser

    1

    HTMLParser(content: Content, features: str = 'html.parser')

Bases: `BaseParser[T]`, `Generic[T]`

Base HTML parser.

Notes

Responsibilities:

    1
    2
    3
    4

    - This class extends the core `BaseParser` with HTML-specific behavior,
      including DOM parsing via BeautifulSoup and reusable extraction helpers.
    - Provides reusable helpers for HTML extraction. Concrete parsers must
      explicitly define the return type.

Guarantees:

    1
    2
    3

    - Accepts only HTML content.
    - Owns a parsed BeautifulSoup DOM tree.
    - Provides pure helper utilities for common HTML structures.

Constraints:

    1
    2

    - Concrete subclasses must define the output type `T` and implement
      the `parse()` method.

Initialize the HTML parser.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`content` | `Content` | HTML content to be parsed. | required
`features` | `str` | BeautifulSoup parser backend to use (e.g., 'html.parser', 'lxml'). | `'html.parser'`

Raises:

Type | Description
--- | ---
`ValueError` | If the content is empty or not valid HTML.

##### Attributes

###### `` supported_types `class-attribute` `instance-attribute`

    1

    supported_types: set[ContentType] = {HTML}

Set of content types supported by this parser (HTML only).

##### Functions

###### `` parse `abstractmethod`

    1

    parse() -> T

Fully parse the HTML content into structured output.

Returns:

Name | Type | Description
--- | --- | ---
`T` | `T` | Parsed representation of type `T`.

Notes

Responsibilities:

    1
    2

    - Implementations must fully interpret the HTML DOM and return a
      deterministic, structured output.

###### `` parse_div `staticmethod`

    1

    parse_div(div: Tag, *, separator: str = ' ') -> str

Extract normalized text from a `<div>` element.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`div` | `Tag` | BeautifulSoup tag representing a `<div>`. | required
`separator` | `str` | String used to separate text nodes. | `' '`

Returns:

Name | Type | Description
--- | --- | ---
`str` | `str` | Flattened, whitespace-normalized text content.

###### `` parse_link `staticmethod`

    1

    parse_link(a: Tag) -> Optional[str]

Extract the hyperlink reference from an `<a>` element.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`a` | `Tag` | BeautifulSoup tag representing an anchor. | required

Returns:

Type | Description
--- | ---
`Optional[str]` | Optional[str]: The value of the `href` attribute, or None if absent.

###### `` parse_meta

    1

    parse_meta() -> dict[str, Any]

Extract high-level metadata from the HTML document.

Returns:

Type | Description
--- | ---
`dict[str, Any]` | dict[str, Any]: Dictionary containing extracted metadata.

Notes

Responsibilities:

    1
    2
    3

    - Extract high-level metadata from the HTML document.
    - This includes: Document title, `<meta>` tag name/property to
      content mappings.

###### `` parse_table `staticmethod`

    1

    parse_table(table: Tag) -> list[list[str]]

Parse an HTML table into a 2D list of strings.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`table` | `Tag` | BeautifulSoup tag representing a `<table>`. | required

Returns:

Type | Description
--- | ---
`list[list[str]]` | list[list[str]]: A list of rows, where each row is a list of cell text values.

###### `` supports

    1

    supports() -> bool

Check whether this parser supports the content's type.

Returns:

Name | Type | Description
--- | --- | ---
`bool` | `bool` | True if the content type is supported; False otherwise.

#### `` HTMLScraper

    1

    HTMLScraper(*, client: Optional[httpx.Client] = None, timeout: float = 15.0, headers: Optional[Mapping[str, str]] = None, follow_redirects: bool = True)

Bases: `BaseScraper`

Base HTML scraper using `httpx`.

Notes

Responsibilities:

    1
    2
    3
    4
    5

    - This scraper retrieves HTML documents over HTTP(S) and returns
      them as raw content wrapped in a `Content` object.
    - Fetches raw bytes and metadata only.
    - The scraper uses `httpx.Client` for HTTP requests, enforces an
      HTML content type, and preserves HTTP response metadata.

Constraints:

    1
    2

    - The scraper does not: Parse HTML, perform retries or backoff,
      handle non-HTML responses.

Initialize the HTML scraper.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`client` | `Client | None` | Optional pre-configured `httpx.Client`. If omitted, a client is created internally. | `None`
`timeout` | `float` | Request timeout in seconds. | `15.0`
`headers` | `Optional[Mapping[str, str]]` | Optional default HTTP headers. | `None`
`follow_redirects` | `bool` | Whether to follow HTTP redirects. | `True`

##### Functions

###### `` fetch

    1

    fetch(source: str, *, metadata: Optional[Mapping[str, Any]] = None) -> Content

Fetch an HTML document from the given source.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`source` | `str` | URL of the HTML document. | required
`metadata` | `Optional[Mapping[str, Any]]` | Optional metadata to be merged into the returned content. | `None`

Returns:

Name | Type | Description
--- | --- | ---
`Content` | `Content` | A `Content` instance containing raw HTML bytes, source URL, HTML content type, and HTTP response metadata.

Raises:

Type | Description
--- | ---
`HTTPError` | If the HTTP request fails.
`ValueError` | If the response is not valid HTML.

###### `` validate_content_type

    1

    validate_content_type(response: httpx.Response) -> None

Validate that the HTTP response contains HTML content.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`response` | `Response` | HTTP response returned by `httpx`. | required

Raises:

Type | Description
--- | ---
`ValueError` | If the `Content-Type` header is missing or does not indicate HTML content.

#### `` PDFParser

    1

    PDFParser(content: Content)

Bases: `BaseParser[T]`, `Generic[T]`

Base PDF parser.

Notes

Responsibilities:

    1
    2

    - This class enforces PDF content-type compatibility and provides
      the extension point for implementing concrete PDF parsing strategies.

Constraints:

    1
    2

    - Concrete implementations must define the output type `T` and
      implement the `parse()` method.

Initialize the parser with content to be parsed.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`content` | `Content` | Content instance to be parsed. | required

Raises:

Type | Description
--- | ---
`ValueError` | If the content type is not supported by this parser.

##### Attributes

###### `` supported_types `class-attribute` `instance-attribute`

    1

    supported_types: set[ContentType] = {PDF}

Set of content types supported by this parser (PDF only).

##### Functions

###### `` parse `abstractmethod`

    1

    parse() -> T

Parse PDF content into a structured output.

Returns:

Name | Type | Description
--- | --- | ---
`T` | `T` | Parsed representation of type `T`.

Raises:

Type | Description
--- | ---
`Exception` | Parsing-specific errors as defined by the implementation.

Notes

Responsibilities:

    1
    2

    - Implementations must fully interpret the PDF binary payload and
      return a deterministic, structured output.

###### `` supports

    1

    supports() -> bool

Check whether this parser supports the content's type.

Returns:

Name | Type | Description
--- | --- | ---
`bool` | `bool` | True if the content type is supported; False otherwise.

#### `` PDFScraper

    1

    PDFScraper(*, client: BasePDFClient)

Bases: `BaseScraper`

Scraper for PDF sources.

Notes

Responsibilities:

    1
    2
    3

    - Delegates byte retrieval to a PDF client and normalizes output
      into `Content`.
    - Preserves caller-provided metadata.

Constraints:

    1
    2

    - The scraper does not perform parsing or interpretation.
    - Does not assume a specific storage backend.

Initialize the PDF scraper.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`client` | `BasePDFClient` | PDF client responsible for retrieving raw PDF bytes. | required

##### Functions

###### `` fetch

    1

    fetch(source: Any, *, metadata: Optional[Mapping[str, Any]] = None) -> Content

Fetch a PDF document from the given source.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`source` | `Any` | Identifier of the PDF source as understood by the configured PDF client. | required
`metadata` | `Optional[Mapping[str, Any]]` | Optional metadata to attach to the returned content. | `None`

Returns:

Name | Type | Description
--- | --- | ---
`Content` | `Content` | A `Content` instance containing raw PDF bytes, source identifier, PDF content type, and optional metadata.

Raises:

Type | Description
--- | ---
`Exception` | Retrieval-specific errors raised by the PDF client.