Skip to content

🧩 Extending OmniRead

OmniRead is meant to be extended by subclassing. All public extension points are generic over their result type, so your parser returns exactly the shape you need.

🧬 Custom parsers

Subclass BaseParser[T] (or a format parser) and implement parse():

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
from pydantic import BaseModel
from omniread import HTMLParser

class Page(BaseModel):
    title: str
    content: str | None

class PageParser(HTMLParser[Page]):
    def parse(self) -> Page:
        soup = self._soup
        div = soup.find("div", id="content")
        return Page(
            title=soup.title.string,
            content=div.get_text() if div else None,
        )

The parsed page is validated by Pydantic on construction — no manual assertions required.

🧬 Custom PDF parsers

PDF binary layout is format-specific, so parsers return your own model:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
from typing import Literal
from pydantic import BaseModel
from omniread import PDFParser

class ParsedPDF(BaseModel):
    size_bytes: int
    magic: Literal[b"%PDF"]

class SimplePDFParser(PDFParser[ParsedPDF]):
    def parse(self) -> ParsedPDF:
        if not self.content.raw.startswith(b"%PDF"):
            raise ValueError("Not a valid PDF")
        return ParsedPDF(size_bytes=len(self.content.raw), magic=b"%PDF")

🧬 Custom clients

Clients supply raw bytes to a scraper. For PDFs, subclass BasePDFClient (or FileSystemPDFClient) and implement fetch(source) -> bytes:

1
2
3
4
5
from omniread.pdf.client import BasePDFClient

class MockPDFClient(BasePDFClient):
    def fetch(self, source):
        return b"%PDF ..."  # bytes for the logical identifier

The same pattern applies to BaseCsvClient and BaseXlsxClient.

🚀 Custom scrapers

Festch something that a built-in scraper does not cover by extending BaseScraper:

1
2
3
4
5
6
from omniread import BaseScraper, Content, ContentType

class StorageScraper(BaseScraper):
    def fetch(self, source, *, metadata=None):
        raw = my_object_storage.download(source)      # your I/O
        return Content(raw=raw, source=source, content_type=ContentType.JSON)

✅ Extension checklist

  1. Keep scraper and parser separate — never mix I/O into parse().
  2. Return Content from any scraper/client so downstream stays uniform.
  3. Return a typed result from your parser (Pydantic model, dataclass, str).
  4. Test your custom layers with a mock client, not a live network.