Vishesh 'ironeagle' Bangotra
b984dd5f42
- Add hand-written wiki (index, overview, how-to, extending, dev) with MkDocs config following the platform anatomy - Complete the library nav by registering the csv and xlsx groups in docforge.nav.yml and docs/mkdocs.lib.yml - Regenerate lib/MCP outputs with the rebuilt nav
104 lines
2.6 KiB
Markdown
104 lines
2.6 KiB
Markdown
# 🧩 Extending OmniRead
|
|
|
|
OmniRead is meant to be extended by subclassing. All public extension points
|
|
are generic over their result type, so your parser returns exactly the shape
|
|
you need.
|
|
|
|
---
|
|
|
|
## 🧬 Custom parsers
|
|
|
|
Subclass `BaseParser[T]` (or a format parser) and implement `parse()`:
|
|
|
|
```python
|
|
from pydantic import BaseModel
|
|
from omniread import HTMLParser
|
|
|
|
class Page(BaseModel):
|
|
title: str
|
|
content: str | None
|
|
|
|
class PageParser(HTMLParser[Page]):
|
|
def parse(self) -> Page:
|
|
soup = self._soup
|
|
div = soup.find("div", id="content")
|
|
return Page(
|
|
title=soup.title.string,
|
|
content=div.get_text() if div else None,
|
|
)
|
|
```
|
|
|
|
The parsed page is validated by Pydantic on construction — no manual
|
|
assertions required.
|
|
|
|
---
|
|
|
|
## 🧬 Custom PDF parsers
|
|
|
|
PDF binary layout is format-specific, so parsers return your own model:
|
|
|
|
```python
|
|
from typing import Literal
|
|
from pydantic import BaseModel
|
|
from omniread import PDFParser
|
|
|
|
class ParsedPDF(BaseModel):
|
|
size_bytes: int
|
|
magic: Literal[b"%PDF"]
|
|
|
|
class SimplePDFParser(PDFParser[ParsedPDF]):
|
|
def parse(self) -> ParsedPDF:
|
|
if not self.content.raw.startswith(b"%PDF"):
|
|
raise ValueError("Not a valid PDF")
|
|
return ParsedPDF(size_bytes=len(self.content.raw), magic=b"%PDF")
|
|
```
|
|
|
|
---
|
|
|
|
## 🧬 Custom clients
|
|
|
|
Clients supply raw bytes to a scraper. For PDFs, subclass
|
|
`BasePDFClient` (or `FileSystemPDFClient`) and implement
|
|
`fetch(source) -> bytes`:
|
|
|
|
```python
|
|
from omniread.pdf.client import BasePDFClient
|
|
|
|
class MockPDFClient(BasePDFClient):
|
|
def fetch(self, source):
|
|
return b"%PDF ..." # bytes for the logical identifier
|
|
```
|
|
|
|
The same pattern applies to `BaseCsvClient` and `BaseXlsxClient`.
|
|
|
|
---
|
|
|
|
## 🚀 Custom scrapers
|
|
|
|
Festch something that a built-in scraper does not cover by extending
|
|
`BaseScraper`:
|
|
|
|
```python
|
|
from omniread import BaseScraper, Content, ContentType
|
|
|
|
class StorageScraper(BaseScraper):
|
|
def fetch(self, source, *, metadata=None):
|
|
raw = my_object_storage.download(source) # your I/O
|
|
return Content(raw=raw, source=source, content_type=ContentType.JSON)
|
|
```
|
|
|
|
---
|
|
|
|
## ✅ Extension checklist
|
|
|
|
1. Keep **scraper** and **parser** separate — never mix I/O into `parse()`.
|
|
2. Return `Content` from any scraper/client so downstream stays uniform.
|
|
3. Return a *typed* result from your parser (Pydantic model, dataclass, str).
|
|
4. Test your custom layers with a mock client, not a live network.
|
|
|
|
---
|
|
|
|
## 📚 Read Next
|
|
|
|
- [How to Use](02_how_to_use.md) — built-in example flows.
|
|
- [Development](04_development.md) — running tests and docs. |