{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"\ud83d\udcc4 OmniRead \u2014 Format-Agnostic Content Acquisition and Parsing","text":"

OmniRead is a composable content ingestion framework with pluggable scrapers and parsers for HTML, PDF, CSV, and XLSX. It cleanly separates fetching (I/O) from interpretation (parsing) through a canonical Content exchange model, giving each format its own scraper and parser while sharing one uniform pipeline.

Doc model: this wiki is written for humans \u2014 how\u2011to guides, examples, and extension recipes. The authoritative API contracts live in the code (GSDFC docstrings) and the machine\u2011readable bundle under docs/mcp/.

"},{"location":"#key-features","title":"\ud83d\ude80 Key Features","text":""},{"location":"#quick-start","title":"\u26a1 Quick Start","text":"
from omniread import HTMLScraper, HTMLParser\n\nclass TitleParser(HTMLParser[str]):\n    def parse(self) -> str:\n        return self._soup.title.string\n\ncontent = HTMLScraper().fetch(\"https://example.com\")\ntitle = TitleParser(content).parse()\nassert title == \"Example Domain\"\n
"},{"location":"#documentation-structure","title":"\ud83d\udcc1 Documentation Structure","text":"Section What you'll find Overview The three-layer architecture and the Content model How to Use Ready-to-run examples for HTML, PDF, CSV, and XLSX Extending OmniRead Subclassing scrapers, parsers, and clients Development Setup, tests, and regenerating docs"},{"location":"#related-resources","title":"\ud83d\udd17 Related Resources","text":"

\u00a9 Aetoskia Internal \u2014 omniread 0.0.1

"},{"location":"01_overview/","title":"\ud83e\uddf1 Overview","text":"

OmniRead is designed as a decoupled content engine with three distinct layers. Understanding them is the key to using and extending the library.

"},{"location":"01_overview/#architecture","title":"\ud83c\udfd7\ufe0f Architecture","text":"
                 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n                 \u2502  Source (URL, file, storage) \u2502\n                 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n                              \u2502\n                    \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n                    \u2502   Scraper / Client  \u2502  fetches raw bytes\n                    \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n                              \u2502  returns\n                    \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n                    \u2502     Content         \u2502  raw + source + type\n                    \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n                              \u2502\n                    \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n                    \u2502      Parser         \u2502  parse() \u2192 structured T\n                    \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n
  1. Scraper (BaseScraper) \u2014 fetches raw bytes from a source (HTTP URL, filesystem path, object storage). Returns a Content instance. Scrapers never interpret content.
  2. Content (Content) \u2014 the canonical exchange model: raw bytes, source identifier, and optional ContentType enum.
  3. Parser (BaseParser[T]) \u2014 receives a Content and returns a structured result of type T via parse().
"},{"location":"01_overview/#the-content-model","title":"\ud83d\udce6 The Content model","text":"

Defined in omniread.core.content:

from dataclasses import dataclass\nfrom omniread import Content, ContentType\n\n@dataclass(slots=True)\nclass Content:\n    raw: bytes\n    source: str\n    content_type: ContentType | None = None\n
"},{"location":"01_overview/#the-contenttype-enum","title":"\ud83c\udfad The ContentType enum","text":"Value MIME Used by HTML text/html HTMLScraper PDF application/pdf PDFScraper XLSX application/vnd.openxmlformats-... XlsxScraper CSV text/csv CsvScraper JSON application/json \u2014 XML application/xml \u2014"},{"location":"01_overview/#format-modules-at-a-glance","title":"\ud83e\udde9 Format modules at a glance","text":"Module Scraper Parser Client Notes omniread.html HTMLScraper HTMLParser \u2014 httpx + BeautifulSoup omniread.pdf PDFScraper PDFParser FileSystemPDFClient explicit client layer omniread.csv CsvScraper CsvParser FileSystemCsvClient stdlib csv module omniread.xlsx XlsxScraper XlsxParser FileSystemXlsxClient openpyxl-backed"},{"location":"01_overview/#read-next","title":"\ud83d\udcda Read Next","text":""},{"location":"02_how_to_use/","title":"\ud83d\udda5\ufe0f How to Use","text":"

Every format follows the same pipeline: scrape \u2192 Content \u2192 parse. This page shows each supported format with working patterns.

"},{"location":"02_how_to_use/#html","title":"\ud83c\udf10 HTML","text":"
from omniread import HTMLScraper, HTMLParser\n\nclass TitleParser(HTMLParser[str]):\n    def parse(self) -> str:\n        return self._soup.title.string\n\nscraper = HTMLScraper()                     # httpx under the hood\ncontent = scraper.fetch(\"https://example.com\")\n\ntitle = TitleParser(content).parse()\n

HTMLScraper accepts an optional client (an httpx.Client) for transport control \u2014 the test suite wires one to a mock transport.

"},{"location":"02_how_to_use/#pdf","title":"\ud83d\udcd5 PDF","text":"

PDFs need a client to supply raw bytes before parsing:

from pathlib import Path\nfrom omniread import FileSystemPDFClient, PDFScraper, PDFParser\n\nclass TextPDFParser(PDFParser[str]):\n    def parse(self) -> str:\n        # implement your extraction logic\n        return self.content.raw  # bytes, decode as needed\n\nclient = FileSystemPDFClient()\nscraper = PDFScraper(client=client)\ncontent = scraper.fetch(Path(\"document.pdf\"))\n\nresult = TextPDFParser(content).parse()\n

PDFParser subclasses receive self.content and implement parse().

"},{"location":"02_how_to_use/#csv","title":"\ud83d\udcca CSV","text":"
from omniread import FileSystemCsvClient, CsvScraper, CsvParser\n\nscraper = CsvScraper(client=FileSystemCsvClient())\ncontent = scraper.fetch(\"data.csv\")\n\nparser = CsvParser(content)\nfor row in parser.rows():\n    print(row)\n

CsvParser.rows() yields string rows trimmed of empties by default (skip_empty=True).

"},{"location":"02_how_to_use/#xlsx","title":"\ud83d\udcd1 XLSX","text":"
from omniread import FileSystemXlsxClient, XlsxScraper, XlsxParser\n\nscraper = XlsxScraper(client=FileSystemXlsxClient())\ncontent = scraper.fetch(\"statement.xlsx\")\n\nparser = XlsxParser(content)\nprint(parser.sheet_names)                 # e.g. [\"Statement\"]\nrows = parser.rows(sheet=\"Statement\")     # by name or index\nall_rows = parser.parse()                 # alias for rows()\n

Key behaviors:

"},{"location":"02_how_to_use/#end-to-end-flow","title":"\ud83d\udd00 End-to-end flow","text":"
content = scraper.fetch(source)   # 1. acquire \u2192 Content\nassert isinstance(content.raw, bytes)\nassert content.content_type is not None\n\nparser = MyParser(content)        # 2. interpret \u2192 T\nresult = parser.parse()\n
"},{"location":"02_how_to_use/#read-next","title":"\ud83d\udcda Read Next","text":""},{"location":"03_extending/","title":"\ud83e\udde9 Extending OmniRead","text":"

OmniRead is meant to be extended by subclassing. All public extension points are generic over their result type, so your parser returns exactly the shape you need.

"},{"location":"03_extending/#custom-parsers","title":"\ud83e\uddec Custom parsers","text":"

Subclass BaseParser[T] (or a format parser) and implement parse():

from pydantic import BaseModel\nfrom omniread import HTMLParser\n\nclass Page(BaseModel):\n    title: str\n    content: str | None\n\nclass PageParser(HTMLParser[Page]):\n    def parse(self) -> Page:\n        soup = self._soup\n        div = soup.find(\"div\", id=\"content\")\n        return Page(\n            title=soup.title.string,\n            content=div.get_text() if div else None,\n        )\n

The parsed page is validated by Pydantic on construction \u2014 no manual assertions required.

"},{"location":"03_extending/#custom-pdf-parsers","title":"\ud83e\uddec Custom PDF parsers","text":"

PDF binary layout is format-specific, so parsers return your own model:

from typing import Literal\nfrom pydantic import BaseModel\nfrom omniread import PDFParser\n\nclass ParsedPDF(BaseModel):\n    size_bytes: int\n    magic: Literal[b\"%PDF\"]\n\nclass SimplePDFParser(PDFParser[ParsedPDF]):\n    def parse(self) -> ParsedPDF:\n        if not self.content.raw.startswith(b\"%PDF\"):\n            raise ValueError(\"Not a valid PDF\")\n        return ParsedPDF(size_bytes=len(self.content.raw), magic=b\"%PDF\")\n
"},{"location":"03_extending/#custom-clients","title":"\ud83e\uddec Custom clients","text":"

Clients supply raw bytes to a scraper. For PDFs, subclass BasePDFClient (or FileSystemPDFClient) and implement fetch(source) -> bytes:

from omniread.pdf.client import BasePDFClient\n\nclass MockPDFClient(BasePDFClient):\n    def fetch(self, source):\n        return b\"%PDF ...\"  # bytes for the logical identifier\n

The same pattern applies to BaseCsvClient and BaseXlsxClient.

"},{"location":"03_extending/#custom-scrapers","title":"\ud83d\ude80 Custom scrapers","text":"

Festch something that a built-in scraper does not cover by extending BaseScraper:

from omniread import BaseScraper, Content, ContentType\n\nclass StorageScraper(BaseScraper):\n    def fetch(self, source, *, metadata=None):\n        raw = my_object_storage.download(source)      # your I/O\n        return Content(raw=raw, source=source, content_type=ContentType.JSON)\n
"},{"location":"03_extending/#extension-checklist","title":"\u2705 Extension checklist","text":"
  1. Keep scraper and parser separate \u2014 never mix I/O into parse().
  2. Return Content from any scraper/client so downstream stays uniform.
  3. Return a typed result from your parser (Pydantic model, dataclass, str).
  4. Test your custom layers with a mock client, not a live network.
"},{"location":"03_extending/#read-next","title":"\ud83d\udcda Read Next","text":""},{"location":"04_development/","title":"\ud83d\udee0\ufe0f Development","text":"

Working on omniread itself.

"},{"location":"04_development/#repository-layout","title":"\ud83d\udcc2 Repository layout","text":"Path Purpose omniread/ The library package (core + html, pdf, csv, xlsx modules) omniread/*.pyi Type stubs kept in sync with implementations tests/ End-to-end and unit tests against mock transports/clients covers \u2014 docs/lib/ Generated library reference (docforge, flat layout) docs/mcp/ Machine-readable bundle served by the MCP server docs/wiki/ This hand-written wiki"},{"location":"04_development/#setup","title":"\ud83d\udd27 Setup","text":"
python -m venv .venv\n.venv/Scripts/pip install -e \".[dev]\"\n
"},{"location":"04_development/#tests","title":"\ud83e\uddea Tests","text":"

Run the suite (offline; a mock httpx transport and mock PDF client are used):

.venv/Scripts/pytest\n

Coverage spans end-to-end scrape \u2192 parse flows for HTML, PDF, and XLSX, plus client validation and CSV/XLSX parsing edge cases.

"},{"location":"04_development/#quality-gates","title":"\u2705 Quality gates","text":"

The CI quality gate runs, matching the Drone pipeline:

.venv/Scripts/black --check .\n.venv/Scripts/ruff check .\n.venv/Scripts/mypy\n.venv/Scripts/pytest\n
"},{"location":"04_development/#building-documentation-docforge","title":"\ud83d\udcdd Building documentation (docforge)","text":"

The site is generated by docforge and served per kind under site/{kind}:

doc-forge build \\\n  --mkdocs --mcp \\\n  --module-is-source --module omniread \\\n  --site-name \"OmniRead\"\n

Preview locally:

doc-forge serve --lib\ndoc-forge serve --wiki\ndoc-forge serve --mcp\n
"},{"location":"04_development/#read-next","title":"\ud83d\udcda Read Next","text":""}]}