Vishesh 'ironeagle' Bangotra
b984dd5f42
- Add hand-written wiki (index, overview, how-to, extending, dev) with MkDocs config following the platform anatomy - Complete the library nav by registering the csv and xlsx groups in docforge.nav.yml and docs/mkdocs.lib.yml - Regenerate lib/MCP outputs with the rebuilt nav
64 lines
2.2 KiB
Markdown
64 lines
2.2 KiB
Markdown
# 📄 OmniRead — Format-Agnostic Content Acquisition and Parsing
|
||
|
||
`OmniRead` is a composable content ingestion framework with pluggable scrapers
|
||
and parsers for HTML, PDF, CSV, and XLSX. It cleanly separates fetching
|
||
(I/O) from interpretation (parsing) through a canonical `Content` exchange
|
||
model, giving each format its own scraper and parser while sharing one
|
||
uniform pipeline.
|
||
|
||
> **Doc model:** this wiki is written for humans — how‑to guides, examples,
|
||
> and extension recipes. The authoritative API contracts live in the code
|
||
> (GSDFC docstrings) and the machine‑readable bundle under `docs/mcp/`.
|
||
|
||
---
|
||
|
||
## 🚀 Key Features
|
||
|
||
* 🧱 **Content/Scraper/Parser architecture** — fetch and interpret in
|
||
independent, testable layers
|
||
* 📄 **Canonical `Content` model** — raw bytes + metadata passed between all
|
||
layers via one dataclass
|
||
* 🌐 **HTML** — HTTP-based scraper with httpx; DOM parsing via BeautifulSoup
|
||
* 📕 **PDF** — client + scraper for raw PDF bytes; user-defined parsers
|
||
* 📊 **CSV** — standard-library CSV parser over `rows()` iterator
|
||
* 📑 **XLSX** — openpyxl-backed spreadsheet parser with sheet selection,
|
||
empty-row control, and ISO date conversion
|
||
|
||
---
|
||
|
||
## ⚡ Quick Start
|
||
|
||
```python
|
||
from omniread import HTMLScraper, HTMLParser
|
||
|
||
class TitleParser(HTMLParser[str]):
|
||
def parse(self) -> str:
|
||
return self._soup.title.string
|
||
|
||
content = HTMLScraper().fetch("https://example.com")
|
||
title = TitleParser(content).parse()
|
||
assert title == "Example Domain"
|
||
```
|
||
|
||
---
|
||
|
||
## 📁 Documentation Structure
|
||
|
||
| Section | What you'll find |
|
||
|---|---|
|
||
| [Overview](01_overview.md) | The three-layer architecture and the `Content` model |
|
||
| [How to Use](02_how_to_use.md) | Ready-to-run examples for HTML, PDF, CSV, and XLSX |
|
||
| [Extending OmniRead](03_extending.md) | Subclassing scrapers, parsers, and clients |
|
||
| [Development](04_development.md) | Setup, tests, and regenerating docs |
|
||
|
||
---
|
||
|
||
## 🔗 Related Resources
|
||
|
||
* **Source Code:** [Gitea Repository](https://git.aetoskia.com/aetos/omniread)
|
||
* **Internal PyPI:** [pip.aetoskia.com/simple/omniread](https://pip.aetoskia.com/simple/omniread)
|
||
* **CI:** Builds and publishes tagged releases, gated on black / ruff / mypy / pytest.
|
||
|
||
---
|
||
|
||
© Aetoskia Internal — `omniread` 0.0.1 |