chore(release): 0.0.3
Squash-merge lib-standardisation: adopt repo standards across packaging, tooling, docs, and CI. - standardize pyproject.toml (canonical packaging, lint config, extras, GSDFC pydoclint) - add py.typed marker, modernize type hints and Google-style docstrings - nest generated docs under docs/lib and docs/mcp; add hand-written wiki - adopt standard Type A PyPI CI pipeline (tag==version guard + quality gate) - promote generated CHANGELOG.md; bump version to 0.0.3
This commit is contained in:
104
docs/wiki/03_extending.md
Normal file
104
docs/wiki/03_extending.md
Normal file
@@ -0,0 +1,104 @@
|
||||
# 🧩 Extending OmniRead
|
||||
|
||||
OmniRead is meant to be extended by subclassing. All public extension points
|
||||
are generic over their result type, so your parser returns exactly the shape
|
||||
you need.
|
||||
|
||||
---
|
||||
|
||||
## 🧬 Custom parsers
|
||||
|
||||
Subclass `BaseParser[T]` (or a format parser) and implement `parse()`:
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel
|
||||
from omniread import HTMLParser
|
||||
|
||||
class Page(BaseModel):
|
||||
title: str
|
||||
content: str | None
|
||||
|
||||
class PageParser(HTMLParser[Page]):
|
||||
def parse(self) -> Page:
|
||||
soup = self._soup
|
||||
div = soup.find("div", id="content")
|
||||
return Page(
|
||||
title=soup.title.string,
|
||||
content=div.get_text() if div else None,
|
||||
)
|
||||
```
|
||||
|
||||
The parsed page is validated by Pydantic on construction — no manual
|
||||
assertions required.
|
||||
|
||||
---
|
||||
|
||||
## 🧬 Custom PDF parsers
|
||||
|
||||
PDF binary layout is format-specific, so parsers return your own model:
|
||||
|
||||
```python
|
||||
from typing import Literal
|
||||
from pydantic import BaseModel
|
||||
from omniread import PDFParser
|
||||
|
||||
class ParsedPDF(BaseModel):
|
||||
size_bytes: int
|
||||
magic: Literal[b"%PDF"]
|
||||
|
||||
class SimplePDFParser(PDFParser[ParsedPDF]):
|
||||
def parse(self) -> ParsedPDF:
|
||||
if not self.content.raw.startswith(b"%PDF"):
|
||||
raise ValueError("Not a valid PDF")
|
||||
return ParsedPDF(size_bytes=len(self.content.raw), magic=b"%PDF")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🧬 Custom clients
|
||||
|
||||
Clients supply raw bytes to a scraper. For PDFs, subclass
|
||||
`BasePDFClient` (or `FileSystemPDFClient`) and implement
|
||||
`fetch(source) -> bytes`:
|
||||
|
||||
```python
|
||||
from omniread.pdf.client import BasePDFClient
|
||||
|
||||
class MockPDFClient(BasePDFClient):
|
||||
def fetch(self, source):
|
||||
return b"%PDF ..." # bytes for the logical identifier
|
||||
```
|
||||
|
||||
The same pattern applies to `BaseCsvClient` and `BaseXlsxClient`.
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Custom scrapers
|
||||
|
||||
Festch something that a built-in scraper does not cover by extending
|
||||
`BaseScraper`:
|
||||
|
||||
```python
|
||||
from omniread import BaseScraper, Content, ContentType
|
||||
|
||||
class StorageScraper(BaseScraper):
|
||||
def fetch(self, source, *, metadata=None):
|
||||
raw = my_object_storage.download(source) # your I/O
|
||||
return Content(raw=raw, source=source, content_type=ContentType.JSON)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ✅ Extension checklist
|
||||
|
||||
1. Keep **scraper** and **parser** separate — never mix I/O into `parse()`.
|
||||
2. Return `Content` from any scraper/client so downstream stays uniform.
|
||||
3. Return a *typed* result from your parser (Pydantic model, dataclass, str).
|
||||
4. Test your custom layers with a mock client, not a live network.
|
||||
|
||||
---
|
||||
|
||||
## 📚 Read Next
|
||||
|
||||
- [How to Use](02_how_to_use.md) — built-in example flows.
|
||||
- [Development](04_development.md) — running tests and docs.
|
||||
Reference in New Issue
Block a user