Vishesh 'ironeagle' Bangotra
b984dd5f42
- Add hand-written wiki (index, overview, how-to, extending, dev) with MkDocs config following the platform anatomy - Complete the library nav by registering the csv and xlsx groups in docforge.nav.yml and docs/mkdocs.lib.yml - Regenerate lib/MCP outputs with the rebuilt nav
62 lines
4.3 KiB
JSON
62 lines
4.3 KiB
JSON
{
|
|
"module": "omniread.core.scraper",
|
|
"content": {
|
|
"path": "omniread.core.scraper",
|
|
"docstring": "# Summary\n\nAbstract scraping contracts for OmniRead.\n\nThis module defines the **format-agnostic scraper interface** responsible for\nacquiring raw content from external sources.\n\nScrapers are responsible for:\n\n- Locating and retrieving raw content bytes\n- Attaching minimal contextual metadata\n- Returning normalized `Content` objects\n\nScrapers are explicitly NOT responsible for:\n\n- Parsing or interpreting content\n- Inferring structure or semantics\n- Performing content-type specific processing\n\nAll interpretation must be delegated to parsers.",
|
|
"objects": {
|
|
"Content": {
|
|
"name": "Content",
|
|
"kind": "class",
|
|
"path": "omniread.core.scraper.Content",
|
|
"signature": "Content(raw: bytes, source: str, content_type: ContentType | None = ..., metadata: Mapping[str, Any] | None = ...)",
|
|
"docstring": "Normalized representation of extracted content.\n\nNotes:\n **Responsibilities:**\n\n - A `Content` instance represents a raw content payload along with\n minimal contextual metadata describing its origin and type.\n - This class is the primary exchange format between scrapers,\n parsers, and downstream consumers.",
|
|
"members": {
|
|
"raw": {
|
|
"name": "raw",
|
|
"kind": "attribute",
|
|
"path": "omniread.core.scraper.Content.raw",
|
|
"signature": null,
|
|
"docstring": "Raw content bytes as retrieved from the source."
|
|
},
|
|
"source": {
|
|
"name": "source",
|
|
"kind": "attribute",
|
|
"path": "omniread.core.scraper.Content.source",
|
|
"signature": null,
|
|
"docstring": "Identifier of the content origin (URL, file path, or logical name)."
|
|
},
|
|
"content_type": {
|
|
"name": "content_type",
|
|
"kind": "attribute",
|
|
"path": "omniread.core.scraper.Content.content_type",
|
|
"signature": null,
|
|
"docstring": "Optional MIME type of the content, if known."
|
|
},
|
|
"metadata": {
|
|
"name": "metadata",
|
|
"kind": "attribute",
|
|
"path": "omniread.core.scraper.Content.metadata",
|
|
"signature": null,
|
|
"docstring": "Optional, implementation-defined metadata associated with the content (e.g., headers, encoding hints, extraction notes)."
|
|
}
|
|
}
|
|
},
|
|
"BaseScraper": {
|
|
"name": "BaseScraper",
|
|
"kind": "class",
|
|
"path": "omniread.core.scraper.BaseScraper",
|
|
"signature": null,
|
|
"docstring": "Base interface for all scrapers.\n\nNotes:\n **Responsibilities:**\n\n - A scraper is responsible ONLY for fetching raw content (bytes)\n from a source. It must not interpret or parse it.\n - A scraper is a stateless acquisition component that retrieves raw\n content from a source and returns it as a `Content` object.\n - Scrapers define how content is obtained, not what the content means.\n - Implementations may vary in transport mechanism, authentication\n strategy, retry and backoff behavior.\n\n **Constraints:**\n\n - Implementations must not parse content, modify content semantics,\n or couple scraping logic to a specific parser.",
|
|
"members": {
|
|
"fetch": {
|
|
"name": "fetch",
|
|
"kind": "function",
|
|
"path": "omniread.core.scraper.BaseScraper.fetch",
|
|
"signature": "fetch(source: str, *, metadata: Mapping[str, Any] | None = None) -> Content",
|
|
"docstring": "Fetch raw content from the given source.\n\nArgs:\n source (str):\n Location identifier (URL, file path, S3 URI, etc.).\n\n metadata (Mapping[str, Any] | None, optional):\n Optional hints for the scraper (headers, auth, etc.).\n\nReturns:\n Content:\n Content object containing raw bytes and metadata.\n\nRaises:\n Exception:\n Retrieval-specific errors as defined by the implementation.\n\nNotes:\n **Responsibilities:**\n\n - Implementations must retrieve the content referenced by `source`\n and return it as raw bytes wrapped in a `Content` object."
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
} |