Vishesh 'ironeagle' Bangotra
b984dd5f42
- Add hand-written wiki (index, overview, how-to, extending, dev) with MkDocs config following the platform anatomy - Complete the library nav by registering the csv and xlsx groups in docforge.nav.yml and docs/mkdocs.lib.yml - Regenerate lib/MCP outputs with the rebuilt nav
2.2 KiB
2.2 KiB
📄 OmniRead — Format-Agnostic Content Acquisition and Parsing
OmniRead is a composable content ingestion framework with pluggable scrapers
and parsers for HTML, PDF, CSV, and XLSX. It cleanly separates fetching
(I/O) from interpretation (parsing) through a canonical Content exchange
model, giving each format its own scraper and parser while sharing one
uniform pipeline.
Doc model: this wiki is written for humans — how‑to guides, examples, and extension recipes. The authoritative API contracts live in the code (GSDFC docstrings) and the machine‑readable bundle under
docs/mcp/.
🚀 Key Features
- 🧱 Content/Scraper/Parser architecture — fetch and interpret in independent, testable layers
- 📄 Canonical
Contentmodel — raw bytes + metadata passed between all layers via one dataclass - 🌐 HTML — HTTP-based scraper with httpx; DOM parsing via BeautifulSoup
- 📕 PDF — client + scraper for raw PDF bytes; user-defined parsers
- 📊 CSV — standard-library CSV parser over
rows()iterator - 📑 XLSX — openpyxl-backed spreadsheet parser with sheet selection, empty-row control, and ISO date conversion
⚡ Quick Start
from omniread import HTMLScraper, HTMLParser
class TitleParser(HTMLParser[str]):
def parse(self) -> str:
return self._soup.title.string
content = HTMLScraper().fetch("https://example.com")
title = TitleParser(content).parse()
assert title == "Example Domain"
📁 Documentation Structure
| Section | What you'll find |
|---|---|
| Overview | The three-layer architecture and the Content model |
| How to Use | Ready-to-run examples for HTML, PDF, CSV, and XLSX |
| Extending OmniRead | Subclassing scrapers, parsers, and clients |
| Development | Setup, tests, and regenerating docs |
🔗 Related Resources
- Source Code: Gitea Repository
- Internal PyPI: pip.aetoskia.com/simple/omniread
- CI: Builds and publishes tagged releases, gated on black / ruff / mypy / pytest.
© Aetoskia Internal — omniread 0.0.1