8 lines
3.2 KiB
JSON
8 lines
3.2 KiB
JSON
{
|
|
"module": "worker_and_progress",
|
|
"content": {
|
|
"path": "worker_and_progress",
|
|
"docstring": "# Worker & Progress\n\n## Summary\n\nLong-running work runs in a background worker. Jobs are durable documents that\nmove through an explicit status machine. Progress is streamed to clients.\n\n---\n\n## The status machine\n\n```\npending -> processing -> completed\n \\-> failed (retried up to N times)\n \\-> paused (waiting for human input)\n```\n\n- `pending` — queued, eligible for pickup.\n- `processing` — claimed by a worker.\n- `paused` — blocked on a human-in-the-loop question.\n- `failed` — error; `retry_count` increments until a max, then skipped.\n- `completed` — terminal success.\n\n## The poll loop\n\n```python\nasync def poll(resources):\n backoff = MIN_POLL\n while True:\n jobs = await job_svc.search(filter={\"status\": \"pending\"}, limit=100)\n backoff = MIN_POLL if jobs else min(backoff * 2, MAX_POLL)\n for job in jobs:\n if (job.retry_count or 0) >= MAX_RETRIES:\n continue\n await job_svc.update_status(job.id, \"processing\")\n try:\n await process(job, resources)\n except PauseForResolution:\n await job_svc.update_status(job.id, \"paused\")\n except Exception as e:\n await job_svc.update_status(\n job.id, \"failed\", error=str(e),\n retry_count=(job.retry_count or 0) + 1,\n )\n await asyncio.sleep(backoff)\n```\n\nTunables live in environment variables: `WORKER_POLL_MIN`, `WORKER_POLL_MAX`,\n`WORKER_MAX_RETRIES`.\n\n## Progress events\n\nA worker publishes structured progress events to a **bus** (a port). The HTTP\nedge exposes a streaming endpoint that replays events for a given job id.\n\n```python\nclass ProgressEvent(BaseModel):\n job_id: str\n step: str\n status: str # started | progress | completed | skipped | paused\n message: dict | None = None\n```\n\nThe bus is an in-memory adapter by default; because it is a port, it can be\nswapped for a durable broker. Emit from the orchestrator at each stage:\n\n```python\nawait self.bus.publish(ProgressEvent(job_id=jid, step=\"normalize\",\n status=\"completed\", message={\"count\": n}))\n```\n\n## Streaming endpoint\n\n```python\nasync def stream_job_events(request, id):\n bus = request.app.state.db_resources.get(\"progress_bus\")\n if bus is None:\n return {\"events\": []}\n\n async def gen():\n sent = 0\n while True:\n events = bus.get_events(id)\n for e in events[sent:]:\n yield f\"data: {e.model_dump_json(exclude_none=True)}\\n\\n\"\n sent += 1\n await asyncio.sleep(0.5)\n\n return StreamingResponse(gen(), media_type=\"text/event-stream\")\n```\n\n> **Known limitation.** The default in-memory bus is per-process. If the worker\n> and the API run as separate processes, events published by the worker are not\n> visible to the API. For production, back the bus with a shared broker. The\n> port boundary makes this a one-adapter change.\n",
|
|
"objects": {}
|
|
}
|
|
} |