{ "module": "omniread", "content": { "path": "omniread", "docstring": "# Summary\n\n`OmniRead` — format-agnostic content acquisition and parsing framework.\n\n`OmniRead` provides a **cleanly layered architecture** for fetching, parsing,\nand normalizing content from heterogeneous sources such as HTML documents\nand PDF files.\n\nThe library is structured around three core concepts:\n\n1. **`Content`**: A canonical, format-agnostic container representing raw content\n bytes and minimal contextual metadata.\n2. **`Scrapers`**: Components responsible for *acquiring* raw content from a\n source (HTTP, filesystem, object storage, etc.). `Scrapers` never interpret\n content.\n3. **`Parsers`**: Components responsible for *interpreting* acquired content and\n converting it into structured, typed representations.\n\n`OmniRead` deliberately separates these responsibilities to ensure:\n\n- Clear boundaries between IO and interpretation.\n- Replaceable implementations per format.\n- Predictable, testable behavior.\n\n# Installation\n\nInstall `OmniRead` using pip:\n\n```bash\npip install omniread\n```\n\nInstall OmniRead using Poetry:\n```bash\npoetry add omniread\n```\n\n---\n\n## Quick start\n\nExample:\n HTML example:\n ```python\n from omniread import HTMLScraper, HTMLParser\n\n scraper = HTMLScraper()\n content = scraper.fetch(\"https://example.com\")\n\n class TitleParser(HTMLParser[str]):\n def parse(self) -> str:\n return self._soup.title.string\n\n parser = TitleParser(content)\n title = parser.parse()\n ```\n\n PDF example:\n ```python\n from omniread import FileSystemPDFClient, PDFScraper, PDFParser\n from pathlib import Path\n\n client = FileSystemPDFClient()\n scraper = PDFScraper(client=client)\n content = scraper.fetch(Path(\"document.pdf\"))\n\n class TextPDFParser(PDFParser[str]):\n def parse(self) -> str:\n # implement PDF text extraction\n ...\n\n parser = TextPDFParser(content)\n result = parser.parse()\n ```\n\n---\n\n# Public API\n\nThis module re-exports the **recommended public entry points** of OmniRead.\nConsumers are encouraged to import from this namespace rather than from\nformat-specific submodules directly, unless advanced customization is\nrequired.\n\n- `Content`: Canonical content model.\n- `ContentType`: Supported media types.\n- `HTMLScraper`: HTTP-based HTML acquisition.\n- `HTMLParser`: Base parser for HTML DOM interpretation.\n- `FileSystemPDFClient`: Local filesystem PDF access.\n- `PDFScraper`: PDF-specific content acquisition.\n- `PDFParser`: Base parser for PDF binary interpretation.\n- `FileSystemXlsxClient`: Local filesystem spreadsheet access.\n- `XlsxScraper`: XLSX-specific content acquisition.\n- `XlsxParser`: Generic string-row parser for xlsx workbooks.\n\n---\n\n# Core Philosophy\n\n`OmniRead` is designed as a **decoupled content engine**:\n\n1. **Separation of Concerns**: Scrapers *fetch*, Parsers *interpret*. Neither\n knows about the other.\n2. **Normalized Exchange**: All components communicate via the `Content` model,\n ensuring a consistent contract.\n3. **Format Agnosticism**: The core logic is independent of whether the input\n is HTML, PDF, or JSON.\n\n---", "objects": { "Content": { "name": "Content", "kind": "class", "path": "omniread.Content", "signature": "Content(raw: bytes, source: str, content_type: ContentType | None = ..., metadata: Mapping[str, Any] | None = ...)", "docstring": "Normalized representation of extracted content.\n\nNotes:\n **Responsibilities:**\n\n - A `Content` instance represents a raw content payload along with\n minimal contextual metadata describing its origin and type.\n - This class is the primary exchange format between scrapers,\n parsers, and downstream consumers.", "members": { "raw": { "name": "raw", "kind": "attribute", "path": "omniread.Content.raw", "signature": null, "docstring": "Raw content bytes as retrieved from the source." }, "source": { "name": "source", "kind": "attribute", "path": "omniread.Content.source", "signature": null, "docstring": "Identifier of the content origin (URL, file path, or logical name)." }, "content_type": { "name": "content_type", "kind": "attribute", "path": "omniread.Content.content_type", "signature": null, "docstring": "Optional MIME type of the content, if known." }, "metadata": { "name": "metadata", "kind": "attribute", "path": "omniread.Content.metadata", "signature": null, "docstring": "Optional, implementation-defined metadata associated with the content (e.g., headers, encoding hints, extraction notes)." } } }, "ContentType": { "name": "ContentType", "kind": "class", "path": "omniread.ContentType", "signature": null, "docstring": "Supported MIME types for extracted content.\n\nNotes:\n **Guarantees:**\n\n - This enum represents the declared or inferred media type of the\n content source.\n - It is primarily used for routing content to the appropriate\n parser or downstream consumer.", "members": { "HTML": { "name": "HTML", "kind": "attribute", "path": "omniread.ContentType.HTML", "signature": null, "docstring": "HTML document content." }, "PDF": { "name": "PDF", "kind": "attribute", "path": "omniread.ContentType.PDF", "signature": null, "docstring": "PDF document content." }, "XLSX": { "name": "XLSX", "kind": "attribute", "path": "omniread.ContentType.XLSX", "signature": null, "docstring": "Office Open XML spreadsheet (xlsx/xlsm) content." }, "CSV": { "name": "CSV", "kind": "attribute", "path": "omniread.ContentType.CSV", "signature": null, "docstring": "Comma-separated-value document content." }, "JSON": { "name": "JSON", "kind": "attribute", "path": "omniread.ContentType.JSON", "signature": null, "docstring": "JSON document content." }, "XML": { "name": "XML", "kind": "attribute", "path": "omniread.ContentType.XML", "signature": null, "docstring": "XML document content." } } }, "BaseCsvClient": { "name": "BaseCsvClient", "kind": "class", "path": "omniread.BaseCsvClient", "signature": null, "docstring": "Abstract client responsible for retrieving csv bytes.\n\nRetrieves bytes from a specific backing store (filesystem, S3, FTP, etc.).\n\nNotes:\n **Responsibilities:**\n\n - Implementations must accept a source identifier appropriate to\n the backing store.\n - Return the full csv binary payload.\n - Raise retrieval-specific errors on failure.", "members": { "fetch": { "name": "fetch", "kind": "function", "path": "omniread.BaseCsvClient.fetch", "signature": "fetch(source: Any)", "docstring": "Fetch raw csv bytes from the given source.\n\nArgs:\n source (Any):\n Identifier of the csv location, such as a file path,\n object storage key, or remote reference.\n\nReturns:\n bytes:\n Raw csv bytes.\n\nRaises:\n Exception:\n Retrieval-specific errors defined by the implementation." } } }, "CsvParser": { "name": "CsvParser", "kind": "class", "path": "omniread.CsvParser", "signature": "CsvParser(content: Content)", "docstring": "Generic csv parser producing string rows from the document.\n\nNotes:\n **Responsibilities:**\n\n - Decode the payload (UTF-8 with BOM support, Latin-1 fallback).\n - Detect the delimiter from a leading sample (`,` `;` tab `|`),\n defaulting to `,`.\n - Normalize cells into deterministic stripped string values.\n - Expose row extraction helpers mirroring `XlsxParser.rows`.\n\n **Constraints:**\n\n - All values are strings; consumers requiring typed values must\n convert on their side.\n - Quoted fields containing delimiters/newlines are handled by\n the standard ``csv`` module.", "members": { "parse": { "name": "parse", "kind": "function", "path": "omniread.CsvParser.parse", "signature": "parse()", "docstring": "Parse the document into normalized string rows.\n\nReturns:\n list[list[str]]:\n Rows of the document." }, "rows": { "name": "rows", "kind": "function", "path": "omniread.CsvParser.rows", "signature": "rows(*, skip_empty: bool = True)", "docstring": "Extract normalized string rows from the document.\n\nArgs:\n skip_empty (bool):\n When True (default), rows whose cells are all blank are\n omitted.\n\nReturns:\n list[list[str]]:\n Normalized rows; trailing blank cells are trimmed per row." } } }, "CsvParserBase": { "name": "CsvParserBase", "kind": "class", "path": "omniread.CsvParserBase", "signature": null, "docstring": "Base csv parser.\n\nNotes:\n **Responsibilities:**\n\n - This class enforces csv content-type compatibility and provides\n the extension point for implementing concrete csv parsing\n strategies.\n\n **Constraints:**\n\n - Concrete implementations must define the output type `T` and\n implement the `parse()` method.", "members": { "supported_types": { "name": "supported_types", "kind": "attribute", "path": "omniread.CsvParserBase.supported_types", "signature": null, "docstring": "Set of content types supported by this parser (CSV only)." }, "parse": { "name": "parse", "kind": "function", "path": "omniread.CsvParserBase.parse", "signature": "parse()", "docstring": "Parse csv content into a structured output.\n\nReturns:\n T:\n Parsed representation of type `T`.\n\nRaises:\n Exception:\n Parsing-specific errors as defined by the implementation." } } }, "CsvScraper": { "name": "CsvScraper", "kind": "class", "path": "omniread.CsvScraper", "signature": "CsvScraper(*, client: BaseCsvClient)", "docstring": "Scraper for csv documents.\n\nNotes:\n **Responsibilities:**\n\n - Fetch raw csv bytes via the configured client.\n - Wrap the payload in a canonical `Content` instance with the\n CSV content type and source identifier.\n\n **Constraints:**\n\n - The scraper does not perform parsing or interpretation.\n - Does not assume a specific storage backend.", "members": { "fetch": { "name": "fetch", "kind": "function", "path": "omniread.CsvScraper.fetch", "signature": "fetch(source: Any, *, metadata: Mapping[str, Any] | None = None)", "docstring": "Fetch a csv document from the given source.\n\nArgs:\n source (Any):\n Identifier of the csv source as understood by the\n configured client.\n metadata (Mapping[str, Any] | None, optional):\n Optional metadata to attach to the returned content.\n\nReturns:\n Content:\n A `Content` instance containing raw csv bytes, source\n identifier, CSV content type, and optional metadata.\n\nRaises:\n Exception:\n Retrieval-specific errors raised by the client." } } }, "FileSystemCsvClient": { "name": "FileSystemCsvClient", "kind": "class", "path": "omniread.FileSystemCsvClient", "signature": null, "docstring": "CSV client that reads from the local filesystem.\n\nNotes:\n **Guarantees:**\n\n - This client reads csv files directly from the disk and\n returns their raw binary contents.", "members": { "fetch": { "name": "fetch", "kind": "function", "path": "omniread.FileSystemCsvClient.fetch", "signature": "fetch(path: Path)", "docstring": "Read a csv file from the local filesystem.\n\nArgs:\n path (Path):\n Filesystem path to the csv file.\n\nReturns:\n bytes:\n Raw csv bytes.\n\nRaises:\n FileNotFoundError:\n If the path does not exist.\n ValueError:\n If the path exists but is not a file." } } }, "HTMLScraper": { "name": "HTMLScraper", "kind": "class", "path": "omniread.HTMLScraper", "signature": "HTMLScraper(*, client: httpx.Client | None = None, timeout: float = 15.0, headers: Mapping[str, str] | None = None, follow_redirects: bool = True)", "docstring": "Base HTML scraper using `httpx`.\n\nNotes:\n **Responsibilities:**\n\n - This scraper retrieves HTML documents over HTTP(S) and returns\n them as raw content wrapped in a `Content` object.\n - Fetches raw bytes and metadata only.\n - The scraper uses `httpx.Client` for HTTP requests, enforces an\n HTML content type, and preserves HTTP response metadata.\n\n **Constraints:**\n\n - The scraper does not: Parse HTML, perform retries or backoff,\n handle non-HTML responses.", "members": { "content_type": { "name": "content_type", "kind": "attribute", "path": "omniread.HTMLScraper.content_type", "signature": null, "docstring": null }, "validate_content_type": { "name": "validate_content_type", "kind": "function", "path": "omniread.HTMLScraper.validate_content_type", "signature": "validate_content_type(response: httpx.Response)", "docstring": "Validate that the HTTP response contains HTML content.\n\nArgs:\n response (httpx.Response):\n HTTP response returned by `httpx`.\n\nRaises:\n ValueError:\n If the `Content-Type` header is missing or does not indicate HTML content." }, "fetch": { "name": "fetch", "kind": "function", "path": "omniread.HTMLScraper.fetch", "signature": "fetch(source: str, *, metadata: Mapping[str, Any] | None = None)", "docstring": "Fetch an HTML document from the given source.\n\nArgs:\n source (str):\n URL of the HTML document.\n metadata (Mapping[str, Any] | None, optional):\n Optional metadata to be merged into the returned content.\n\nReturns:\n Content:\n A `Content` instance containing raw HTML bytes, source URL, HTML content type, and HTTP response metadata.\n\nRaises:\n httpx.HTTPError:\n If the HTTP request fails.\n ValueError:\n If the response is not valid HTML." } } }, "HTMLParser": { "name": "HTMLParser", "kind": "class", "path": "omniread.HTMLParser", "signature": "HTMLParser(content: Content, features: str = 'html.parser')", "docstring": "Base HTML parser.\n\nNotes:\n **Responsibilities:**\n\n - This class extends the core `BaseParser` with HTML-specific behavior,\n including DOM parsing via BeautifulSoup and reusable extraction helpers.\n - Provides reusable helpers for HTML extraction. Concrete parsers must\n explicitly define the return type.\n\n **Guarantees:**\n\n - Accepts only HTML content.\n - Owns a parsed BeautifulSoup DOM tree.\n - Provides pure helper utilities for common HTML structures.\n\n **Constraints:**\n\n - Concrete subclasses must define the output type `T` and implement\n the `parse()` method.", "members": { "supported_types": { "name": "supported_types", "kind": "attribute", "path": "omniread.HTMLParser.supported_types", "signature": null, "docstring": "Set of content types supported by this parser (HTML only)." }, "parse": { "name": "parse", "kind": "function", "path": "omniread.HTMLParser.parse", "signature": "parse()", "docstring": "Fully parse the HTML content into structured output.\n\nReturns:\n T:\n Parsed representation of type `T`.\n\nNotes:\n **Responsibilities:**\n\n - Implementations must fully interpret the HTML DOM and return a\n deterministic, structured output." }, "parse_div": { "name": "parse_div", "kind": "function", "path": "omniread.HTMLParser.parse_div", "signature": "parse_div(div: Tag, *, separator: str = ' ')", "docstring": "Extract normalized text from a `