docs/omniread/wiki/search/search_index.json

{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"\ud83d\udcc4 OmniRead \u2014 Format-Agnostic Content Acquisition and Parsing","text":"<p><code>OmniRead</code> is a composable content ingestion framework with pluggable scrapers and parsers for HTML, PDF, CSV, and XLSX. It cleanly separates fetching (I/O) from interpretation (parsing) through a canonical <code>Content</code> exchange model, giving each format its own scraper and parser while sharing one uniform pipeline.</p> <p>Doc model: this wiki is written for humans \u2014 how\u2011to guides, examples, and extension recipes. The authoritative API contracts live in the code (GSDFC docstrings) and the machine\u2011readable bundle under <code>docs/mcp/</code>.</p>"},{"location":"#key-features","title":"\ud83d\ude80 Key Features","text":"<ul> <li>\ud83e\uddf1 Content/Scraper/Parser architecture \u2014 fetch and interpret in   independent, testable layers</li> <li>\ud83d\udcc4 Canonical <code>Content</code> model \u2014 raw bytes + metadata passed between all   layers via one dataclass</li> <li>\ud83c\udf10 HTML \u2014 HTTP-based scraper with httpx; DOM parsing via BeautifulSoup</li> <li>\ud83d\udcd5 PDF \u2014 client + scraper for raw PDF bytes; user-defined parsers</li> <li>\ud83d\udcca CSV \u2014 standard-library CSV parser over <code>rows()</code> iterator</li> <li>\ud83d\udcd1 XLSX \u2014 openpyxl-backed spreadsheet parser with sheet selection,   empty-row control, and ISO date conversion</li> </ul>"},{"location":"#quick-start","title":"\u26a1 Quick Start","text":"<pre><code>from omniread import HTMLScraper, HTMLParser\n\nclass TitleParser(HTMLParser[str]):\n    def parse(self) -&gt; str:\n        return self._soup.title.string\n\ncontent = HTMLScraper().fetch(\"https://example.com\")\ntitle = TitleParser(content).parse()\nassert title == \"Example Domain\"\n</code></pre>"},{"location":"#documentation-structure","title":"\ud83d\udcc1 Documentation Structure","text":"Section What you'll find Overview The three-layer architecture and the <code>Content</code> model How to Use Ready-to-run examples for HTML, PDF, CSV, and XLSX Extending OmniRead Subclassing scrapers, parsers, and clients Development Setup, tests, and regenerating docs"},{"location":"#related-resources","title":"\ud83d\udd17 Related Resources","text":"<ul> <li>Source Code: Gitea Repository</li> <li>Internal PyPI: pip.aetoskia.com/simple/omniread</li> <li>CI: Builds and publishes tagged releases, gated on black / ruff / mypy / pytest.</li> </ul> <p>\u00a9 Aetoskia Internal \u2014 <code>omniread</code> 0.0.1</p>"},{"location":"01_overview/","title":"\ud83e\uddf1 Overview","text":"<p>OmniRead is designed as a decoupled content engine with three distinct layers. Understanding them is the key to using and extending the library.</p>"},{"location":"01_overview/#architecture","title":"\ud83c\udfd7\ufe0f Architecture","text":"<pre><code>                 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n                 \u2502  Source (URL, file, storage) \u2502\n                 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n                              \u2502\n                    \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n                    \u2502   Scraper / Client  \u2502  fetches raw bytes\n                    \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n                              \u2502  returns\n                    \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n                    \u2502     Content         \u2502  raw + source + type\n                    \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n                              \u2502\n                    \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n                    \u2502      Parser         \u2502  parse() \u2192 structured T\n                    \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n</code></pre> <ol> <li>Scraper (<code>BaseScraper</code>) \u2014 fetches raw bytes from a source (HTTP URL,    filesystem path, object storage). Returns a <code>Content</code> instance.    Scrapers never interpret content.</li> <li>Content (<code>Content</code>) \u2014 the canonical exchange model: raw bytes,    source identifier, and optional <code>ContentType</code> enum.</li> <li>Parser (<code>BaseParser[T]</code>) \u2014 receives a <code>Content</code> and returns a    structured result of type <code>T</code> via <code>parse()</code>.</li> </ol>"},{"location":"01_overview/#the-content-model","title":"\ud83d\udce6 The <code>Content</code> model","text":"<p>Defined in <code>omniread.core.content</code>:</p> <pre><code>from dataclasses import dataclass\nfrom omniread import Content, ContentType\n\n@dataclass(slots=True)\nclass Content:\n    raw: bytes\n    source: str\n    content_type: ContentType | None = None\n</code></pre> <ul> <li><code>raw</code> \u2014 the raw bytes exactly as retrieved.</li> <li><code>source</code> \u2014 URL, file path, or logical name identifying the origin.</li> <li><code>content_type</code> \u2014 optional <code>ContentType</code> enum value.</li> </ul>"},{"location":"01_overview/#the-contenttype-enum","title":"\ud83c\udfad The <code>ContentType</code> enum","text":"Value MIME Used by <code>HTML</code> <code>text/html</code> HTMLScraper <code>PDF</code> <code>application/pdf</code> PDFScraper <code>XLSX</code> <code>application/vnd.openxmlformats-...</code> XlsxScraper <code>CSV</code> <code>text/csv</code> CsvScraper <code>JSON</code> <code>application/json</code> \u2014 <code>XML</code> <code>application/xml</code> \u2014"},{"location":"01_overview/#format-modules-at-a-glance","title":"\ud83e\udde9 Format modules at a glance","text":"Module Scraper Parser Client Notes <code>omniread.html</code> <code>HTMLScraper</code> <code>HTMLParser</code> \u2014 httpx + BeautifulSoup <code>omniread.pdf</code> <code>PDFScraper</code> <code>PDFParser</code> <code>FileSystemPDFClient</code> explicit client layer <code>omniread.csv</code> <code>CsvScraper</code> <code>CsvParser</code> <code>FileSystemCsvClient</code> stdlib csv module <code>omniread.xlsx</code> <code>XlsxScraper</code> <code>XlsxParser</code> <code>FileSystemXlsxClient</code> openpyxl-backed"},{"location":"01_overview/#read-next","title":"\ud83d\udcda Read Next","text":"<ul> <li>How to Use \u2014 working examples per format.</li> <li>Extending OmniRead \u2014 subclassing scrapers and parsers.</li> </ul>"},{"location":"02_how_to_use/","title":"\ud83d\udda5\ufe0f How to Use","text":"<p>Every format follows the same pipeline: scrape \u2192 <code>Content</code> \u2192 parse. This page shows each supported format with working patterns.</p>"},{"location":"02_how_to_use/#html","title":"\ud83c\udf10 HTML","text":"<pre><code>from omniread import HTMLScraper, HTMLParser\n\nclass TitleParser(HTMLParser[str]):\n    def parse(self) -&gt; str:\n        return self._soup.title.string\n\nscraper = HTMLScraper()                     # httpx under the hood\ncontent = scraper.fetch(\"https://example.com\")\n\ntitle = TitleParser(content).parse()\n</code></pre> <p><code>HTMLScraper</code> accepts an optional <code>client</code> (an <code>httpx.Client</code>) for transport control \u2014 the test suite wires one to a mock transport.</p>"},{"location":"02_how_to_use/#pdf","title":"\ud83d\udcd5 PDF","text":"<p>PDFs need a client to supply raw bytes before parsing:</p> <pre><code>from pathlib import Path\nfrom omniread import FileSystemPDFClient, PDFScraper, PDFParser\n\nclass TextPDFParser(PDFParser[str]):\n    def parse(self) -&gt; str:\n        # implement your extraction logic\n        return self.content.raw  # bytes, decode as needed\n\nclient = FileSystemPDFClient()\nscraper = PDFScraper(client=client)\ncontent = scraper.fetch(Path(\"document.pdf\"))\n\nresult = TextPDFParser(content).parse()\n</code></pre> <p><code>PDFParser</code> subclasses receive <code>self.content</code> and implement <code>parse()</code>.</p>"},{"location":"02_how_to_use/#csv","title":"\ud83d\udcca CSV","text":"<pre><code>from omniread import FileSystemCsvClient, CsvScraper, CsvParser\n\nscraper = CsvScraper(client=FileSystemCsvClient())\ncontent = scraper.fetch(\"data.csv\")\n\nparser = CsvParser(content)\nfor row in parser.rows():\n    print(row)\n</code></pre> <p><code>CsvParser.rows()</code> yields string rows trimmed of empties by default (<code>skip_empty=True</code>).</p>"},{"location":"02_how_to_use/#xlsx","title":"\ud83d\udcd1 XLSX","text":"<pre><code>from omniread import FileSystemXlsxClient, XlsxScraper, XlsxParser\n\nscraper = XlsxScraper(client=FileSystemXlsxClient())\ncontent = scraper.fetch(\"statement.xlsx\")\n\nparser = XlsxParser(content)\nprint(parser.sheet_names)                 # e.g. [\"Statement\"]\nrows = parser.rows(sheet=\"Statement\")     # by name or index\nall_rows = parser.parse()                 # alias for rows()\n</code></pre> <p>Key behaviors:</p> <ul> <li><code>rows(skip_empty=False)</code> keeps blank rows (off by default).</li> <li>Cells render as trimmed strings; date cells convert to ISO format   (<code>2026-06-01T00:00:00</code>).</li> <li><code>rows(sheet=\"Missing\")</code> raises for an unknown sheet.</li> </ul>"},{"location":"02_how_to_use/#end-to-end-flow","title":"\ud83d\udd00 End-to-end flow","text":"<pre><code>content = scraper.fetch(source)   # 1. acquire \u2192 Content\nassert isinstance(content.raw, bytes)\nassert content.content_type is not None\n\nparser = MyParser(content)        # 2. interpret \u2192 T\nresult = parser.parse()\n</code></pre>"},{"location":"02_how_to_use/#read-next","title":"\ud83d\udcda Read Next","text":"<ul> <li>Overview \u2014 the full architecture.</li> <li>Extending OmniRead \u2014 custom clients and parsers.</li> </ul>"},{"location":"03_extending/","title":"\ud83e\udde9 Extending OmniRead","text":"<p>OmniRead is meant to be extended by subclassing. All public extension points are generic over their result type, so your parser returns exactly the shape you need.</p>"},{"location":"03_extending/#custom-parsers","title":"\ud83e\uddec Custom parsers","text":"<p>Subclass <code>BaseParser[T]</code> (or a format parser) and implement <code>parse()</code>:</p> <pre><code>from pydantic import BaseModel\nfrom omniread import HTMLParser\n\nclass Page(BaseModel):\n    title: str\n    content: str | None\n\nclass PageParser(HTMLParser[Page]):\n    def parse(self) -&gt; Page:\n        soup = self._soup\n        div = soup.find(\"div\", id=\"content\")\n        return Page(\n            title=soup.title.string,\n            content=div.get_text() if div else None,\n        )\n</code></pre> <p>The parsed page is validated by Pydantic on construction \u2014 no manual assertions required.</p>"},{"location":"03_extending/#custom-pdf-parsers","title":"\ud83e\uddec Custom PDF parsers","text":"<p>PDF binary layout is format-specific, so parsers return your own model:</p> <pre><code>from typing import Literal\nfrom pydantic import BaseModel\nfrom omniread import PDFParser\n\nclass ParsedPDF(BaseModel):\n    size_bytes: int\n    magic: Literal[b\"%PDF\"]\n\nclass SimplePDFParser(PDFParser[ParsedPDF]):\n    def parse(self) -&gt; ParsedPDF:\n        if not self.content.raw.startswith(b\"%PDF\"):\n            raise ValueError(\"Not a valid PDF\")\n        return ParsedPDF(size_bytes=len(self.content.raw), magic=b\"%PDF\")\n</code></pre>"},{"location":"03_extending/#custom-clients","title":"\ud83e\uddec Custom clients","text":"<p>Clients supply raw bytes to a scraper. For PDFs, subclass <code>BasePDFClient</code> (or <code>FileSystemPDFClient</code>) and implement <code>fetch(source) -&gt; bytes</code>:</p> <pre><code>from omniread.pdf.client import BasePDFClient\n\nclass MockPDFClient(BasePDFClient):\n    def fetch(self, source):\n        return b\"%PDF ...\"  # bytes for the logical identifier\n</code></pre> <p>The same pattern applies to <code>BaseCsvClient</code> and <code>BaseXlsxClient</code>.</p>"},{"location":"03_extending/#custom-scrapers","title":"\ud83d\ude80 Custom scrapers","text":"<p>Festch something that a built-in scraper does not cover by extending <code>BaseScraper</code>:</p> <pre><code>from omniread import BaseScraper, Content, ContentType\n\nclass StorageScraper(BaseScraper):\n    def fetch(self, source, *, metadata=None):\n        raw = my_object_storage.download(source)      # your I/O\n        return Content(raw=raw, source=source, content_type=ContentType.JSON)\n</code></pre>"},{"location":"03_extending/#extension-checklist","title":"\u2705 Extension checklist","text":"<ol> <li>Keep scraper and parser separate \u2014 never mix I/O into <code>parse()</code>.</li> <li>Return <code>Content</code> from any scraper/client so downstream stays uniform.</li> <li>Return a typed result from your parser (Pydantic model, dataclass, str).</li> <li>Test your custom layers with a mock client, not a live network.</li> </ol>"},{"location":"03_extending/#read-next","title":"\ud83d\udcda Read Next","text":"<ul> <li>How to Use \u2014 built-in example flows.</li> <li>Development \u2014 running tests and docs.</li> </ul>"},{"location":"04_development/","title":"\ud83d\udee0\ufe0f Development","text":"<p>Working on <code>omniread</code> itself.</p>"},{"location":"04_development/#repository-layout","title":"\ud83d\udcc2 Repository layout","text":"Path Purpose <code>omniread/</code> The library package (core + html, pdf, csv, xlsx modules) <code>omniread/*.pyi</code> Type stubs kept in sync with implementations <code>tests/</code> End-to-end and unit tests against mock transports/clients <code>covers</code> \u2014 <code>docs/lib/</code> Generated library reference (docforge, flat layout) <code>docs/mcp/</code> Machine-readable bundle served by the MCP server <code>docs/wiki/</code> This hand-written wiki"},{"location":"04_development/#setup","title":"\ud83d\udd27 Setup","text":"<pre><code>python -m venv .venv\n.venv/Scripts/pip install -e \".[dev]\"\n</code></pre>"},{"location":"04_development/#tests","title":"\ud83e\uddea Tests","text":"<p>Run the suite (offline; a mock httpx transport and mock PDF client are used):</p> <pre><code>.venv/Scripts/pytest\n</code></pre> <p>Coverage spans end-to-end scrape \u2192 parse flows for HTML, PDF, and XLSX, plus client validation and CSV/XLSX parsing edge cases.</p>"},{"location":"04_development/#quality-gates","title":"\u2705 Quality gates","text":"<p>The CI quality gate runs, matching the Drone pipeline:</p> <pre><code>.venv/Scripts/black --check .\n.venv/Scripts/ruff check .\n.venv/Scripts/mypy\n.venv/Scripts/pytest\n</code></pre>"},{"location":"04_development/#building-documentation-docforge","title":"\ud83d\udcdd Building documentation (docforge)","text":"<p>The site is generated by <code>docforge</code> and served per kind under <code>site/{kind}</code>:</p> <pre><code>doc-forge build \\\n  --mkdocs --mcp \\\n  --module-is-source --module omniread \\\n  --site-name \"OmniRead\"\n</code></pre> <ul> <li><code>--module-is-source</code> renders the flat <code>docs/lib/</code> layout (no nesting under   <code>omniread/</code>), matching <code>docforge.nav.yml</code> and <code>docs/mkdocs.lib.yml</code>.</li> <li><code>--mcp</code> regenerates the structured bundle in <code>docs/mcp/</code>.</li> <li><code>--wiki</code> builds this wiki.</li> <li><code>gen_api.py</code> (if present) regenerates the API docs.</li> </ul> <p>Preview locally:</p> <pre><code>doc-forge serve --lib\ndoc-forge serve --wiki\ndoc-forge serve --mcp\n</code></pre>"},{"location":"04_development/#read-next","title":"\ud83d\udcda Read Next","text":"<ul> <li>Extending OmniRead \u2014 custom parsers, scrapers, clients.</li> <li>Overview \u2014 the core architecture.</li> </ul>"}]}