omniread/docs/wiki/03_extending.md

🧩 Extending OmniRead

OmniRead is meant to be extended by subclassing. All public extension points are generic over their result type, so your parser returns exactly the shape you need.

---

🧬 Custom parsers

Subclass BaseParser[T] (or a format parser) and implement parse():

from pydantic import BaseModel
from omniread import HTMLParser

class Page(BaseModel):
    title: str
    content: str | None

class PageParser(HTMLParser[Page]):
    def parse(self) -> Page:
        soup = self._soup
        div = soup.find("div", id="content")
        return Page(
            title=soup.title.string,
            content=div.get_text() if div else None,
        )

The parsed page is validated by Pydantic on construction — no manual assertions required.

---

🧬 Custom PDF parsers

PDF binary layout is format-specific, so parsers return your own model:

from typing import Literal
from pydantic import BaseModel
from omniread import PDFParser

class ParsedPDF(BaseModel):
    size_bytes: int
    magic: Literal[b"%PDF"]

class SimplePDFParser(PDFParser[ParsedPDF]):
    def parse(self) -> ParsedPDF:
        if not self.content.raw.startswith(b"%PDF"):
            raise ValueError("Not a valid PDF")
        return ParsedPDF(size_bytes=len(self.content.raw), magic=b"%PDF")

---

🧬 Custom clients

Clients supply raw bytes to a scraper. For PDFs, subclass BasePDFClient (or FileSystemPDFClient) and implement fetch(source) -> bytes:

from omniread.pdf.client import BasePDFClient

class MockPDFClient(BasePDFClient):
    def fetch(self, source):
        return b"%PDF ..."  # bytes for the logical identifier

The same pattern applies to BaseCsvClient and BaseXlsxClient.

---

🚀 Custom scrapers

Festch something that a built-in scraper does not cover by extending BaseScraper:

from omniread import BaseScraper, Content, ContentType

class StorageScraper(BaseScraper):
    def fetch(self, source, *, metadata=None):
        raw = my_object_storage.download(source)      # your I/O
        return Content(raw=raw, source=source, content_type=ContentType.JSON)

---

✅ Extension checklist

1. Keep scraper and parser separate — never mix I/O into parse().
2. Return Content from any scraper/client so downstream stays uniform.
3. Return a typed result from your parser (Pydantic model, dataclass, str).
4. Test your custom layers with a mock client, not a live network.

---

📚 Read Next

• How to Use — built-in example flows.
• Development — running tests and docs.