chore(release): 0.0.3
Squash-merge lib-standardisation: adopt repo standards across packaging, tooling, docs, and CI. - standardize pyproject.toml (canonical packaging, lint config, extras, GSDFC pydoclint) - add py.typed marker, modernize type hints and Google-style docstrings - nest generated docs under docs/lib and docs/mcp; add hand-written wiki - adopt standard Type A PyPI CI pipeline (tag==version guard + quality gate) - promote generated CHANGELOG.md; bump version to 0.0.3
This commit is contained in:
86
docs/wiki/01_overview.md
Normal file
86
docs/wiki/01_overview.md
Normal file
@@ -0,0 +1,86 @@
|
||||
# 🧱 Overview
|
||||
|
||||
OmniRead is designed as a **decoupled content engine** with three distinct
|
||||
layers. Understanding them is the key to using and extending the library.
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Architecture
|
||||
|
||||
```text
|
||||
┌─────────────────────────────┐
|
||||
│ Source (URL, file, storage) │
|
||||
└────────────┬────────────────┘
|
||||
│
|
||||
┌─────────▼──────────┐
|
||||
│ Scraper / Client │ fetches raw bytes
|
||||
└─────────┬──────────┘
|
||||
│ returns
|
||||
┌─────────▼──────────┐
|
||||
│ Content │ raw + source + type
|
||||
└─────────┬──────────┘
|
||||
│
|
||||
┌─────────▼──────────┐
|
||||
│ Parser │ parse() → structured T
|
||||
└────────────────────┘
|
||||
```
|
||||
|
||||
1. **Scraper** (`BaseScraper`) — fetches raw bytes from a source (HTTP URL,
|
||||
filesystem path, object storage). Returns a `Content` instance.
|
||||
Scrapers never interpret content.
|
||||
2. **Content** (`Content`) — the canonical exchange model: raw bytes,
|
||||
source identifier, and optional `ContentType` enum.
|
||||
3. **Parser** (`BaseParser[T]`) — receives a `Content` and returns a
|
||||
structured result of type `T` via `parse()`.
|
||||
|
||||
---
|
||||
|
||||
## 📦 The `Content` model
|
||||
|
||||
Defined in `omniread.core.content`:
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass
|
||||
from omniread import Content, ContentType
|
||||
|
||||
@dataclass(slots=True)
|
||||
class Content:
|
||||
raw: bytes
|
||||
source: str
|
||||
content_type: ContentType | None = None
|
||||
```
|
||||
|
||||
- `raw` — the raw bytes exactly as retrieved.
|
||||
- `source` — URL, file path, or logical name identifying the origin.
|
||||
- `content_type` — optional `ContentType` enum value.
|
||||
|
||||
---
|
||||
|
||||
## 🎭 The `ContentType` enum
|
||||
|
||||
| Value | MIME | Used by |
|
||||
|---|---|---|
|
||||
| `HTML` | `text/html` | HTMLScraper |
|
||||
| `PDF` | `application/pdf` | PDFScraper |
|
||||
| `XLSX` | `application/vnd.openxmlformats-...` | XlsxScraper |
|
||||
| `CSV` | `text/csv` | CsvScraper |
|
||||
| `JSON` | `application/json` | — |
|
||||
| `XML` | `application/xml` | — |
|
||||
|
||||
---
|
||||
|
||||
## 🧩 Format modules at a glance
|
||||
|
||||
| Module | Scraper | Parser | Client | Notes |
|
||||
|---|---|---|---|---|
|
||||
| `omniread.html` | `HTMLScraper` | `HTMLParser` | — | httpx + BeautifulSoup |
|
||||
| `omniread.pdf` | `PDFScraper` | `PDFParser` | `FileSystemPDFClient` | explicit client layer |
|
||||
| `omniread.csv` | `CsvScraper` | `CsvParser` | `FileSystemCsvClient` | stdlib csv module |
|
||||
| `omniread.xlsx` | `XlsxScraper` | `XlsxParser` | `FileSystemXlsxClient` | openpyxl-backed |
|
||||
|
||||
---
|
||||
|
||||
## 📚 Read Next
|
||||
|
||||
- [How to Use](02_how_to_use.md) — working examples per format.
|
||||
- [Extending OmniRead](03_extending.md) — subclassing scrapers and parsers.
|
||||
109
docs/wiki/02_how_to_use.md
Normal file
109
docs/wiki/02_how_to_use.md
Normal file
@@ -0,0 +1,109 @@
|
||||
# 🖥️ How to Use
|
||||
|
||||
Every format follows the same pipeline: **scrape → `Content` → parse**.
|
||||
This page shows each supported format with working patterns.
|
||||
|
||||
---
|
||||
|
||||
## 🌐 HTML
|
||||
|
||||
```python
|
||||
from omniread import HTMLScraper, HTMLParser
|
||||
|
||||
class TitleParser(HTMLParser[str]):
|
||||
def parse(self) -> str:
|
||||
return self._soup.title.string
|
||||
|
||||
scraper = HTMLScraper() # httpx under the hood
|
||||
content = scraper.fetch("https://example.com")
|
||||
|
||||
title = TitleParser(content).parse()
|
||||
```
|
||||
|
||||
`HTMLScraper` accepts an optional `client` (an `httpx.Client`) for transport
|
||||
control — the test suite wires one to a mock transport.
|
||||
|
||||
---
|
||||
|
||||
## 📕 PDF
|
||||
|
||||
PDFs need a **client** to supply raw bytes before parsing:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
from omniread import FileSystemPDFClient, PDFScraper, PDFParser
|
||||
|
||||
class TextPDFParser(PDFParser[str]):
|
||||
def parse(self) -> str:
|
||||
# implement your extraction logic
|
||||
return self.content.raw # bytes, decode as needed
|
||||
|
||||
client = FileSystemPDFClient()
|
||||
scraper = PDFScraper(client=client)
|
||||
content = scraper.fetch(Path("document.pdf"))
|
||||
|
||||
result = TextPDFParser(content).parse()
|
||||
```
|
||||
|
||||
`PDFParser` subclasses receive `self.content` and implement `parse()`.
|
||||
|
||||
---
|
||||
|
||||
## 📊 CSV
|
||||
|
||||
```python
|
||||
from omniread import FileSystemCsvClient, CsvScraper, CsvParser
|
||||
|
||||
scraper = CsvScraper(client=FileSystemCsvClient())
|
||||
content = scraper.fetch("data.csv")
|
||||
|
||||
parser = CsvParser(content)
|
||||
for row in parser.rows():
|
||||
print(row)
|
||||
```
|
||||
|
||||
`CsvParser.rows()` yields string rows trimmed of empties by default
|
||||
(`skip_empty=True`).
|
||||
|
||||
---
|
||||
|
||||
## 📑 XLSX
|
||||
|
||||
```python
|
||||
from omniread import FileSystemXlsxClient, XlsxScraper, XlsxParser
|
||||
|
||||
scraper = XlsxScraper(client=FileSystemXlsxClient())
|
||||
content = scraper.fetch("statement.xlsx")
|
||||
|
||||
parser = XlsxParser(content)
|
||||
print(parser.sheet_names) # e.g. ["Statement"]
|
||||
rows = parser.rows(sheet="Statement") # by name or index
|
||||
all_rows = parser.parse() # alias for rows()
|
||||
```
|
||||
|
||||
Key behaviors:
|
||||
|
||||
- `rows(skip_empty=False)` keeps blank rows (off by default).
|
||||
- Cells render as trimmed strings; date cells convert to ISO format
|
||||
(`2026-06-01T00:00:00`).
|
||||
- `rows(sheet="Missing")` raises for an unknown sheet.
|
||||
|
||||
---
|
||||
|
||||
## 🔀 End-to-end flow
|
||||
|
||||
```python
|
||||
content = scraper.fetch(source) # 1. acquire → Content
|
||||
assert isinstance(content.raw, bytes)
|
||||
assert content.content_type is not None
|
||||
|
||||
parser = MyParser(content) # 2. interpret → T
|
||||
result = parser.parse()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📚 Read Next
|
||||
|
||||
- [Overview](01_overview.md) — the full architecture.
|
||||
- [Extending OmniRead](03_extending.md) — custom clients and parsers.
|
||||
104
docs/wiki/03_extending.md
Normal file
104
docs/wiki/03_extending.md
Normal file
@@ -0,0 +1,104 @@
|
||||
# 🧩 Extending OmniRead
|
||||
|
||||
OmniRead is meant to be extended by subclassing. All public extension points
|
||||
are generic over their result type, so your parser returns exactly the shape
|
||||
you need.
|
||||
|
||||
---
|
||||
|
||||
## 🧬 Custom parsers
|
||||
|
||||
Subclass `BaseParser[T]` (or a format parser) and implement `parse()`:
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel
|
||||
from omniread import HTMLParser
|
||||
|
||||
class Page(BaseModel):
|
||||
title: str
|
||||
content: str | None
|
||||
|
||||
class PageParser(HTMLParser[Page]):
|
||||
def parse(self) -> Page:
|
||||
soup = self._soup
|
||||
div = soup.find("div", id="content")
|
||||
return Page(
|
||||
title=soup.title.string,
|
||||
content=div.get_text() if div else None,
|
||||
)
|
||||
```
|
||||
|
||||
The parsed page is validated by Pydantic on construction — no manual
|
||||
assertions required.
|
||||
|
||||
---
|
||||
|
||||
## 🧬 Custom PDF parsers
|
||||
|
||||
PDF binary layout is format-specific, so parsers return your own model:
|
||||
|
||||
```python
|
||||
from typing import Literal
|
||||
from pydantic import BaseModel
|
||||
from omniread import PDFParser
|
||||
|
||||
class ParsedPDF(BaseModel):
|
||||
size_bytes: int
|
||||
magic: Literal[b"%PDF"]
|
||||
|
||||
class SimplePDFParser(PDFParser[ParsedPDF]):
|
||||
def parse(self) -> ParsedPDF:
|
||||
if not self.content.raw.startswith(b"%PDF"):
|
||||
raise ValueError("Not a valid PDF")
|
||||
return ParsedPDF(size_bytes=len(self.content.raw), magic=b"%PDF")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🧬 Custom clients
|
||||
|
||||
Clients supply raw bytes to a scraper. For PDFs, subclass
|
||||
`BasePDFClient` (or `FileSystemPDFClient`) and implement
|
||||
`fetch(source) -> bytes`:
|
||||
|
||||
```python
|
||||
from omniread.pdf.client import BasePDFClient
|
||||
|
||||
class MockPDFClient(BasePDFClient):
|
||||
def fetch(self, source):
|
||||
return b"%PDF ..." # bytes for the logical identifier
|
||||
```
|
||||
|
||||
The same pattern applies to `BaseCsvClient` and `BaseXlsxClient`.
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Custom scrapers
|
||||
|
||||
Festch something that a built-in scraper does not cover by extending
|
||||
`BaseScraper`:
|
||||
|
||||
```python
|
||||
from omniread import BaseScraper, Content, ContentType
|
||||
|
||||
class StorageScraper(BaseScraper):
|
||||
def fetch(self, source, *, metadata=None):
|
||||
raw = my_object_storage.download(source) # your I/O
|
||||
return Content(raw=raw, source=source, content_type=ContentType.JSON)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ✅ Extension checklist
|
||||
|
||||
1. Keep **scraper** and **parser** separate — never mix I/O into `parse()`.
|
||||
2. Return `Content` from any scraper/client so downstream stays uniform.
|
||||
3. Return a *typed* result from your parser (Pydantic model, dataclass, str).
|
||||
4. Test your custom layers with a mock client, not a live network.
|
||||
|
||||
---
|
||||
|
||||
## 📚 Read Next
|
||||
|
||||
- [How to Use](02_how_to_use.md) — built-in example flows.
|
||||
- [Development](04_development.md) — running tests and docs.
|
||||
87
docs/wiki/04_development.md
Normal file
87
docs/wiki/04_development.md
Normal file
@@ -0,0 +1,87 @@
|
||||
# 🛠️ Development
|
||||
|
||||
Working on `omniread` itself.
|
||||
|
||||
---
|
||||
|
||||
## 📂 Repository layout
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `omniread/` | The library package (core + html, pdf, csv, xlsx modules) |
|
||||
| `omniread/*.pyi` | Type stubs kept in sync with implementations |
|
||||
| `tests/` | End-to-end and unit tests against mock transports/clients |
|
||||
| `covers` | — |
|
||||
| `docs/lib/` | Generated library reference (docforge, flat layout) |
|
||||
| `docs/mcp/` | Machine-readable bundle served by the MCP server |
|
||||
| `docs/wiki/` | This hand-written wiki |
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Setup
|
||||
|
||||
```bash
|
||||
python -m venv .venv
|
||||
.venv/Scripts/pip install -e ".[dev]"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Tests
|
||||
|
||||
Run the suite (offline; a mock httpx transport and mock PDF client are used):
|
||||
|
||||
```bash
|
||||
.venv/Scripts/pytest
|
||||
```
|
||||
|
||||
Coverage spans end-to-end scrape → parse flows for HTML, PDF, and XLSX,
|
||||
plus client validation and CSV/XLSX parsing edge cases.
|
||||
|
||||
---
|
||||
|
||||
## ✅ Quality gates
|
||||
|
||||
The CI quality gate runs, matching the Drone pipeline:
|
||||
|
||||
```bash
|
||||
.venv/Scripts/black --check .
|
||||
.venv/Scripts/ruff check .
|
||||
.venv/Scripts/mypy
|
||||
.venv/Scripts/pytest
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📝 Building documentation (docforge)
|
||||
|
||||
The site is generated by [`docforge`](https://git.aetoskia.com/aetos/doc-forge)
|
||||
and served per kind under `site/{kind}`:
|
||||
|
||||
```bash
|
||||
doc-forge build \
|
||||
--mkdocs --mcp \
|
||||
--module-is-source --module omniread \
|
||||
--site-name "OmniRead"
|
||||
```
|
||||
|
||||
- `--module-is-source` renders the flat `docs/lib/` layout (no nesting under
|
||||
`omniread/`), matching `docforge.nav.yml` and `docs/mkdocs.lib.yml`.
|
||||
- `--mcp` regenerates the structured bundle in `docs/mcp/`.
|
||||
- `--wiki` builds this wiki.
|
||||
- `gen_api.py` (if present) regenerates the API docs.
|
||||
|
||||
Preview locally:
|
||||
|
||||
```bash
|
||||
doc-forge serve --lib
|
||||
doc-forge serve --wiki
|
||||
doc-forge serve --mcp
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📚 Read Next
|
||||
|
||||
- [Extending OmniRead](03_extending.md) — custom parsers, scrapers, clients.
|
||||
- [Overview](01_overview.md) — the core architecture.
|
||||
64
docs/wiki/index.md
Normal file
64
docs/wiki/index.md
Normal file
@@ -0,0 +1,64 @@
|
||||
# 📄 OmniRead — Format-Agnostic Content Acquisition and Parsing
|
||||
|
||||
`OmniRead` is a composable content ingestion framework with pluggable scrapers
|
||||
and parsers for HTML, PDF, CSV, and XLSX. It cleanly separates fetching
|
||||
(I/O) from interpretation (parsing) through a canonical `Content` exchange
|
||||
model, giving each format its own scraper and parser while sharing one
|
||||
uniform pipeline.
|
||||
|
||||
> **Doc model:** this wiki is written for humans — how‑to guides, examples,
|
||||
> and extension recipes. The authoritative API contracts live in the code
|
||||
> (GSDFC docstrings) and the machine‑readable bundle under `docs/mcp/`.
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Key Features
|
||||
|
||||
* 🧱 **Content/Scraper/Parser architecture** — fetch and interpret in
|
||||
independent, testable layers
|
||||
* 📄 **Canonical `Content` model** — raw bytes + metadata passed between all
|
||||
layers via one dataclass
|
||||
* 🌐 **HTML** — HTTP-based scraper with httpx; DOM parsing via BeautifulSoup
|
||||
* 📕 **PDF** — client + scraper for raw PDF bytes; user-defined parsers
|
||||
* 📊 **CSV** — standard-library CSV parser over `rows()` iterator
|
||||
* 📑 **XLSX** — openpyxl-backed spreadsheet parser with sheet selection,
|
||||
empty-row control, and ISO date conversion
|
||||
|
||||
---
|
||||
|
||||
## ⚡ Quick Start
|
||||
|
||||
```python
|
||||
from omniread import HTMLScraper, HTMLParser
|
||||
|
||||
class TitleParser(HTMLParser[str]):
|
||||
def parse(self) -> str:
|
||||
return self._soup.title.string
|
||||
|
||||
content = HTMLScraper().fetch("https://example.com")
|
||||
title = TitleParser(content).parse()
|
||||
assert title == "Example Domain"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📁 Documentation Structure
|
||||
|
||||
| Section | What you'll find |
|
||||
|---|---|
|
||||
| [Overview](01_overview.md) | The three-layer architecture and the `Content` model |
|
||||
| [How to Use](02_how_to_use.md) | Ready-to-run examples for HTML, PDF, CSV, and XLSX |
|
||||
| [Extending OmniRead](03_extending.md) | Subclassing scrapers, parsers, and clients |
|
||||
| [Development](04_development.md) | Setup, tests, and regenerating docs |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 Related Resources
|
||||
|
||||
* **Source Code:** [Gitea Repository](https://git.aetoskia.com/aetos/omniread)
|
||||
* **Internal PyPI:** [pip.aetoskia.com/simple/omniread](https://pip.aetoskia.com/simple/omniread)
|
||||
* **CI:** Builds and publishes tagged releases, gated on black / ruff / mypy / pytest.
|
||||
|
||||
---
|
||||
|
||||
© Aetoskia Internal — `omniread` 0.0.1
|
||||
Reference in New Issue
Block a user