Vishesh 'ironeagle' Bangotra
49a00351bb
- mongo-ops now serves /mongo-ops/wiki/ and /mongo-ops/lib/ independently - add mongo-ops MCP bundle under mcp/mongo-ops - fix copy-paste mcp server (jwtlib -> mongo_ops) in config.yml - .drone.yml/Dockerfile: publish port 8007 for mongo-ops MCP
1 line
147 KiB
JSON
1 line
147 KiB
JSON
{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"\ud83e\udde9 mongo-ops \u2014 Async MongoDB Operations Layer for FastAPI","text":"<p><code>mongo-ops</code> is a modular, high-performance MongoDB operations library for FastAPI microservices. It standardizes connection lifecycle, Pydantic v2 document models, and repository-style async CRUD \u2014 with added layers for caching, reference population, and multi-document transactions. It is built on top of Motor.</p> <p>Doc model: this wiki is written for humans \u2014 how\u2011to guides, examples, and testing recipes. The authoritative API contracts live in the code (docstrings) and the machine\u2011readable bundle under <code>docs/mcp/</code>.</p>"},{"location":"#key-features","title":"\ud83d\ude80 Key Features","text":"<ul> <li>\ud83e\uddf1 Unified repository pattern \u2014 <code>BaseRepository</code> with generic CRUD</li> <li>\u26a1 Fully asynchronous (Motor-based)</li> <li>\ud83e\uddec Pydantic v2 data models with auto\u2011timestamps (<code>BaseDocument</code>)</li> <li>\ud83d\udd12 Model registry for multi-collection startup (<code>ModelRegistry</code>)</li> <li>\ud83d\uddc4\ufe0f ID-based caching \u2014 in-memory or Redis (<code>cache/</code>)</li> <li>\ud83d\udd17 Reference population with cycle detection (<code>populate/</code>)</li> <li>\ud83d\udd01 Transaction and session helpers (<code>transactions/</code>)</li> <li>\ud83e\uddea Mock-friendly architecture \u2014 the whole repo test suite runs without MongoDB</li> </ul>"},{"location":"#installation","title":"\ud83d\udce6 Installation","text":"<p>From your internal PyPI:</p> Bash<pre><code>pip install --extra-index-url https://$PYPI_USERNAME:$PYPI_PASSWORD@pip.aetoskia.com/simple mongo-ops\n</code></pre> <p>With the extra batteries your workload needs:</p> Bash<pre><code>pip install --extra-index-url https://$PYPI_USERNAME:$PYPI_PASSWORD@pip.aetoskia.com/simple \"mongo-ops[redis]\" # Redis cache backend\npip install --extra-index-url https://$PYPI_USERNAME:$PYPI_PASSWORD@pip.aetoskia.com/simple \"mongo-ops[all]\" # dev, redis, beanie, fastapi\n</code></pre> <p>From local source:</p> Bash<pre><code>pip install -e .\n</code></pre>"},{"location":"#documentation-structure","title":"\ud83d\udcc1 Documentation Structure","text":"Section Description Overview Architecture, lifecycle, and repository decision table Core Components Validated reference for the public API surface Use cases Step-by-step recipes, grouped by theme \u00b7 Getting Started \u00b7 01 \u2013 Basic CRUD API FastAPI user management app \u00b7 02 \u2013 Custom repository Product catalog with business logic \u00b7 06 \u2013 Multi-model One service, many collections \u00b7 Data & Queries \u00b7 04 \u2013 Pagination Filtering, sorting, page metadata \u00b7 05 \u2013 Soft deletes Recoverable delete pattern \u00b7 13 \u2013 Indexes Single, compound, and optioned indexes \u00b7 Caching \u00b7 07 \u2013 Caching In-memory / Redis read cache \u00b7 10 \u2013 Cache + population Reading through cache, populating on hit \u00b7 11 \u2013 Cache lifecycle Startup/shutdown hygiene \u00b7 16 \u2013 Cached repository intricacies What's stored, JSON round-trip, typed-FK trap \u00b7 Population \u00b7 08 \u2013 Population Resolve references on read \u00b7 09 \u2013 Advanced population Nested + circular references \u00b7 15 \u2013 PopulatingRepository internals Object \u21c4 ObjectId write/read lifecycle, wiring \u00b7 17 \u2013 Populate & depopulate examples Worked shapes for depopulate/populate + cached round trip \u00b7 Advanced Usage \u00b7 03 \u2013 Transactions Atomic order + inventory updates \u00b7 12 \u2013 Transaction helper <code>execute_transaction</code> \u00b7 18 \u2013 Aggregation pipelines <code>$group</code>, <code>$lookup</code>, reporting \u00b7 19 \u2013 Bulk operations <code>update_many</code> / <code>delete_many</code> / <code>bulk_write</code> Best Practices Layering, lifecycle, and gotchas Error Handling Library exceptions and FastAPI mapping Testing Mock Motor, engine, cache, transactions \u00b7 Testing example Mock-based quickstart"},{"location":"#related-resources","title":"\ud83d\udd17 Related Resources","text":"<ul> <li>Source Code: Gitea Repository</li> <li>Internal PyPI: pip.aetoskia.com/simple/mongo-ops</li> <li>Drone CI: Auto-builds and publishes tagged releases, gated on black / ruff / mypy / pytest.</li> </ul> <p>\u00a9 Aetoskia Internal \u2014 <code>mongo-ops</code> 0.1.5</p>"},{"location":"01_overview/","title":"Library Overview","text":"<p><code>mongo-ops</code> gives FastAPI/microservice teams a small, opinionated toolkit for talking to MongoDB asynchronously. This page builds the mental model: what the layers are, in what order they must be wired up, and which repository to reach for.</p>"},{"location":"01_overview/#architecture","title":"\ud83c\udfd7\ufe0f Architecture","text":"Text Only<pre><code>\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 Application (FastAPI) \u2502\n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n\u2502 \u2502 CachedBase \u2502 Populating \u2502 BaseRepository\u2502 Raw \u2502 \u2502\n\u2502 \u2502 Repository \u2502 Repository \u2502 (CRUDMixin) \u2502 Motor \u2502 \u2502\n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n\u2502 \u2502 \u2502 \u2502 \u2502\n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n\u2502 \u2502 Cache backend\u2502 \u2502 Population\u2502 \u2502 TransactionManager \u2502 \u2502\n\u2502 \u2502 memory/redis \u2502 \u2502 Engine \u2502 \u2502 (session helpers) \u2502 \u2502\n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n\u2502 \u2502 ModelRegistry (models, indexes, cache lifecycle) \u2502 \u2502\n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n\u2502 \u2502 MongoConnectionManager (Motor client + database) \u2502 \u2502\n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u25bc\n MongoDB (Motor / PyMongo)\n</code></pre> <p>The layers build on each other:</p> <ol> <li>Connection \u2014 <code>MongoConnectionManager</code> owns the Motor client and active database.</li> <li>Models \u2014 <code>BaseDocument</code> (Pydantic v2) gives every document <code>id</code> (aliased <code>_id</code>) plus <code>created_at</code> / <code>updated_at</code>.</li> <li>CRUD \u2014 <code>CRUDMixin</code> / <code>BaseRepository</code> implement generic async CRUD against a collection.</li> <li>Startup \u2014 <code>ModelRegistry</code> centralizes model + index registration and cache lifecycle for multi-collection services.</li> <li>Advanced layers \u2014 caching and population compose around a repository:</li> <li><code>CachedBaseRepository</code> wraps <code>BaseRepository</code> with a <code>CacheBackend</code>.</li> <li><code>PopulatingRepository</code> wraps <code>BaseRepository</code> with a <code>PopulationEngine</code>.</li> <li>Transactions \u2014 <code>TransactionManager</code> runs a list of async operations atomically on a session.</li> </ol>"},{"location":"01_overview/#lifecycle-rules","title":"\ud83d\udd04 Lifecycle Rules","text":"<p><code>mongo-ops</code> has a strict startup order. Violating it raises fast, loud exceptions:</p> Step Call Why 1 <code>await MongoConnectionManager.connect(uri, db_name)</code> Without a connection, <code>get_database()</code>/<code>get_client()</code> raise <code>RuntimeError(\"Database not connected...\")</code>. 2 <code>await ModelRegistry.initialize_all()</code> Create registered indexes (idempotent via <code>create_index</code>). 3 <code>await ModelRegistry.set_cache_backend(...)</code> then <code>await ModelRegistry.initialize_cache()</code> Must happen after connection, before any cache-backed operation. Raises <code>RuntimeError(\"No cache backend registered...\")</code> if skipped. 4 Use repositories Constructed repos resolve the collection from the live database. 5 <code>await ModelRegistry.shutdown_cache()</code> + <code>await MongoConnectionManager.disconnect()</code> On shutdown (in-memory TTL task cancelled; Redis pub/sub closed). <p>The canonical wiring is the FastAPI lifespan context manager (see use case 01).</p>"},{"location":"01_overview/#which-repository-should-you-use","title":"\ud83e\udded Which Repository Should You Use?","text":"Repository Use when Adds <code>BaseRepository[T]</code> Plain CRUD \u2014 the default nothing extra <code>CachedBaseRepository[T]</code> Read\u2011heavy, low\u2011write fields (lookups by <code>_id</code>) cache\u2011first <code>get_by_id</code>, cache on <code>create</code>, invalidate on <code>update</code>/<code>delete</code>, <code>warm_cache</code> <code>PopulatingRepository[T]</code> You return related docs (FK references) denormalized <code>_populate</code> on read, <code>_depopulate</code> on write, FK\u2011guarded <code>patch</code> <code>CRUDMixin</code> Reusing CRUD inside an existing class raw CRUD against a collection you already have <p>There is no built\u2011in <code>CachedPopulatingRepository</code>. Use case 10 shows how to compose caching + population in a small subclass.</p>"},{"location":"01_overview/#read-next","title":"\u27a1\ufe0f Read Next","text":"<ul> <li>Core Components \u2014 the validated public API surface.</li> <li>Use case 01 \u2014 the minimal FastAPI app.</li> <li>Best Practices \u2014 team-wide conventions.</li> </ul>"},{"location":"02_components/","title":"Core Components","text":"<p>A validated reference to the public API. Signatures match the code in <code>mongo_ops/</code> exactly. See the use cases for runnable recipes.</p>"},{"location":"02_components/#1-mongoconnectionmanager","title":"1. MongoConnectionManager","text":"<p>Singleton-style owner of the Motor client and active database.</p> Method Signature Behavior <code>connect</code> <code>async (uri: str, db_name: str, **kwargs) -> AsyncIOMotorDatabase</code> Creates <code>AsyncIOMotorClient(uri, **kwargs)</code> and selects the database. Idempotent. <code>disconnect</code> <code>async () -> None</code> Closes the client and clears state. <code>get_database</code> <code>() -> AsyncIOMotorDatabase</code> Returns the active database; raises <code>RuntimeError(\"Database not connected. Call connect() first.\")</code>. <code>get_client</code> <code>() -> AsyncIOMotorClient</code> Returns the active client; raises <code>RuntimeError(\"Client not connected. Call connect() first.\")</code>. <code>lifespan</code> <code>async ctx manager (uri, db_name, **kwargs)</code> <code>connect()</code> on entry, yields the database, <code>disconnect()</code> on exit. Designed for FastAPI lifespan."},{"location":"02_components/#2-basedocument-pyobjectid","title":"2. BaseDocument & PyObjectId","text":"<p><code>BaseDocument(BaseModel)</code> \u2014 inherit for every Mongo entity.</p> <ul> <li><code>id: Optional[PyObjectId]</code> \u2014 aliased to <code>_id</code>, serialized to <code>str</code>.</li> <li><code>created_at: datetime</code> \u2014 defaults to <code>datetime.utcnow()</code>.</li> <li><code>updated_at: datetime</code> \u2014 defaults to <code>datetime.utcnow()</code>.</li> <li><code>Config.populate_by_name = True</code>, <code>arbitrary_types_allowed = True</code>, <code>json_encoders = {ObjectId: str}</code>.</li> </ul> <p><code>PyObjectId(ObjectId)</code> \u2014 Pydantic v2-compatible <code>ObjectId</code> that accepts <code>str</code> or <code>ObjectId</code> and validates with <code>ObjectId.is_valid</code>.</p>"},{"location":"02_components/#3-crudmixint","title":"3. CRUDMixin[T]","text":"<p>Generic CRUD over a Motor collection. The building block of all repositories.</p> Method Signature Notes <code>create</code> <code>async (data: T) -> T</code> Dumps model (excludes <code>id</code>, <code>None</code>), stamps <code>created_at</code>/<code>updated_at</code>, inserts, returns model with assigned <code>_id</code>. <code>get_by_id</code> <code>async (id: str \\| ObjectId) -> Optional[T]</code> <code>str</code> is accepted and converted to <code>ObjectId</code>. <code>get_many</code> <code>async (filter: dict \\| None = None, skip: int = 0, limit: int = 100, sort: list[tuple] \\| None = None) -> list[T]</code> Cursor <code>.skip().limit().sort(...)</code> then <code>to_list(limit)</code>. <code>limit=0</code> disables the limit clause. <code>update</code> <code>async (id, data: dict[str, Any]) -> Optional[T]</code> <code>$set</code> + refreshed <code>updated_at</code> via <code>find_one_and_update</code>. <code>patch</code> <code>async (id, data: dict[str, Any]) -> Optional[T]</code> Same as <code>update</code> but intended for REST PATCH semantics. <code>delete</code> <code>async (id) -> bool</code> True if a document was deleted. <code>count</code> <code>async (filter: dict \\| None = None) -> int</code> <code>count_documents</code>."},{"location":"02_components/#4-baserepositoryt","title":"4. BaseRepository[T]","text":"<p><code>BaseRepository(collection_name: str, model: type[T])</code> \u2014 resolves the collection from <code>MongoConnectionManager.get_database()[collection_name]</code>. Requires an active connection at construction time. Provides everything in <code>CRUDMixin</code> plus <code>collection_name</code>.</p>"},{"location":"02_components/#5-populatingrepositoryt","title":"5. PopulatingRepository[T]","text":"<p><code>PopulatingRepository(collection_name, model, population_engine: PopulationEngine | None = None, populate_rules: list[PopulateRule] | None = None)</code>.</p> <ul> <li><code>set_population_engine(engine)</code> / <code>set_populate_rules(rules)</code> \u2014 swap engine/rules at runtime.</li> <li><code>data_to_model</code> hooks <code>_populate</code> \u2014 <code>get_by_id</code>/<code>get_many</code> return fully populated models.</li> <li><code>_depopulate(document)</code> \u2014 collapses populated FK fields back to <code>ObjectId</code> before <code>create</code>/<code>update</code>.</li> <li><code>create</code> / <code>update</code> accept a model <code>T</code> (not a dict) so depopulation can run.</li> <li><code>patch</code> blocks FK fields \u2014 raises <code>ValueError(\"Cannot patch FK fields via patch(): ... Use update() to change FK fields.\")</code>.</li> </ul> <p>Populate semantics (important): a <code>PopulateRule</code> names a field that holds either an <code>ObjectId</code> or a <code>list[ObjectId]</code> and is the same field that gets replaced with the resolved document(s). There is no separate \"ref field\" vs \"target field\". See use case 08.</p>"},{"location":"02_components/#6-transactionmanager","title":"6. TransactionManager","text":"Method Signature Behavior <code>start_session</code> <code>async ctx manager (**kwargs) -> AsyncIOMotorClientSession</code> Yields a session with an active transaction. Pass <code>session=</code> to every collection call inside. <code>execute_transaction</code> <code>async (operations: list[Callable[[session], Awaitable[Any]]], **kwargs) -> list[Any]</code> Runs each op inside one transaction and returns results in order; any exception aborts the transaction and propagates."},{"location":"02_components/#7-modelregistry","title":"7. ModelRegistry","text":"<p>Centralized models, indexes, and cache lifecycle for multi-collection services.</p> Method Signature Behavior <code>register</code> <code>(collection_name: str, model: type[BaseDocument], indexes: list[Any] \\| None = None)</code> Records model + index specs. Index specs are passed as-is to pymongo <code>create_index</code> \u2014 single tuples, compound lists, or dicts with <code>keys</code>/<code>options</code>. <code>initialize_all</code> <code>async (db: AsyncIOMotorDatabase \\| None = None) -> None</code> <code>create_index</code> per registered spec (idempotent). Uses the manager database if <code>db</code> omitted. <code>get_model</code> <code>(collection_name) -> type[BaseDocument]</code> Raises <code>KeyError</code> if unregistered. <code>list_collections</code> <code>() -> list[str]</code> Registered collection names. <code>set_cache_backend</code> <code>(backend: CacheBackend) -> None</code> Register the single shared backend. <code>initialize_cache</code> <code>async () -> None</code> Starts the backend (background TTL cleanup) \u2014 raises <code>RuntimeError</code> if no backend registered. <code>shutdown_cache</code> <code>async () -> None</code> Stops the backend cleanly and clears it. <code>get_cache_backend</code> <code>() -> Optional[CacheBackend]</code> Current backend, if any."},{"location":"02_components/#8-cache-layer","title":"8. Cache Layer","text":""},{"location":"02_components/#81-cachebackend-abstract","title":"8.1 CacheBackend (abstract)","text":"<p>Async interface: <code>get(key) -> Optional[bytes]</code>, <code>set(key, value: bytes, ttl | None)</code>, <code>delete(key)</code>, <code>exists(key)</code>, <code>clear_pattern(pattern)</code>, <code>get_stats() -> CacheStats</code>, <code>initialize()</code>, <code>shutdown()</code>. Values are bytes (JSON-encoded).</p>"},{"location":"02_components/#82-cachestats","title":"8.2 CacheStats","text":"<p>Dataclass: <code>hits</code>, <code>misses</code>, <code>sets</code>, <code>deletes</code>, <code>current_size</code>, <code>max_size</code>.</p>"},{"location":"02_components/#83-cacheconfig","title":"8.3 CacheConfig","text":"<p>Dataclass: <code>enabled: bool = True</code>, <code>backend: Literal[\"memory\", \"redis\"] = \"memory\"</code>, <code>redis_client</code>, <code>default_ttl: int = 300</code>, <code>max_entries: int = 10000</code>, <code>key_prefix: str = \"\"</code>, <code>cleanup_interval: int = 60</code>. Raises <code>ValueError</code> if <code>backend=\"redis\"</code> without a client, and <code>ImportError</code> if <code>redis</code> is not installed.</p>"},{"location":"02_components/#84-inmemorycachebackend","title":"8.4 InMemoryCacheBackend","text":"<p><code>InMemoryCacheBackend(max_entries=10000, default_ttl=300, cleanup_interval=60)</code> \u2014 LRU <code>OrderedDict</code> + TTL heap; <code>initialize()</code> spawns the periodic cleanup task (TTL 0 expires immediately).</p>"},{"location":"02_components/#85-rediscachebackend","title":"8.5 RedisCacheBackend","text":"<p><code>RedisCacheBackend(redis_client, key_prefix=\"\", default_ttl=300)</code> \u2014 <code>setex</code> storage, SCAN-based <code>clear_pattern</code>, and <code>publish_invalidate(key)</code> for cross-service invalidation on <code>delete</code> via the <code>mongo_ops:cache:invalidate</code> channel.</p>"},{"location":"02_components/#86-cachedbaserepositoryt","title":"8.6 CachedBaseRepository[T]","text":"<p><code>CachedBaseRepository(collection_name, model, cache_backend: CacheBackend, config: CacheConfig | None = None)</code>.</p> <ul> <li>Cache keys are <code>\"{key_prefix}{id}\"</code> (prefix defaults to <code>\"{collection_name}:\"</code>).</li> <li><code>get_by_id</code> \u2014 cache-first; cache miss reads DB and stores <code>model_dump(by_alias=True)</code> (JSON-encoded) for <code>default_ttl</code>. Honors <code>config.enabled=False</code> (bypass).</li> <li><code>create</code> \u2014 inserts then caches the result.</li> <li><code>update</code>/<code>delete</code> \u2014 refresh or remove the cache entry.</li> <li><code>warm_cache(ids) -> int</code> \u2014 prefetch a list of IDs, returns count warmed.</li> <li><code>invalidate_cache(id)</code> \u2014 manual eviction.</li> </ul>"},{"location":"02_components/#9-population-layer","title":"9. Population Layer","text":""},{"location":"02_components/#91-populaterule","title":"9.1 PopulateRule","text":"<p>Dataclass:</p> Python<pre><code>@dataclass\nclass PopulateRule:\n field_name: str # field holding the ObjectId / list[ObjectId]; replaced in-place with the resolved doc(s)\n collection_name: str # collection the references point at\n nested_rules: list[PopulateRule] | None = None\n max_depth: int = 1\n filter: dict | None = None # DECLARED but NOT yet applied by the engine\n projection: dict | None = None # DECLARED but NOT yet applied by the engine\n</code></pre> <p>\u26a0\ufe0f <code>filter</code> and <code>projection</code> are accepted but currently ignored by <code>PopulationEngine</code> \u2014 do not rely on them yet.</p>"},{"location":"02_components/#92-populationengine","title":"9.2 PopulationEngine","text":"<p><code>PopulationEngine(repos: dict[str, Any], global_max_depth: int = 10)</code> \u2014 <code>repos</code> maps <code>collection_name</code> \u2192 repository.</p> <ul> <li><code>register_repo(collection_name, repo)</code> \u2014 add repositories at runtime.</li> <li><code>populate(document, rules, depth=0)</code> \u2014 resolves refs recursively, replacing <code>field_name</code> in place; raises <code>CircularReferenceError(collection, doc_id, path)</code> when a <code>(Class, id)</code> pair is revisited.</li> <li><code>depopulate(document, rules)</code> \u2014 collapse populated docs back to IDs (for storage).</li> <li><code>global_max_depth</code> caps recursion; per-rule <code>max_depth</code> bounds a rule's descent.</li> </ul>"},{"location":"02_components/#93-circularreferenceerrorvalueerror","title":"9.3 CircularReferenceError(ValueError)","text":"<p>Holds <code>collection</code>, <code>doc_id</code>, and the visited <code>path</code> for debugging cycle messages.</p>"},{"location":"02_components/#related","title":"Related","text":"<ul> <li>Use case 01 \u2013 Basic CRUD \u00b7 07 \u2013 Caching \u00b7 08 \u2013 Population</li> </ul>"},{"location":"04_best_practices/","title":"Best Practices","text":"<p>Team-wide conventions for building fast, testable MongoDB services with <code>mongo-ops</code>.</p>"},{"location":"04_best_practices/#layering","title":"\ud83c\udfd7\ufe0f Layering","text":"<ol> <li>One repository per collection. Encapsulate every query the domain needs behind repository methods; keep Mongo details (<code>$regex</code>, <code>$inc</code>, projections) inside the repository.</li> <li>Keep models thin. <code>BaseDocument</code> for the shape; use Pydantic <code>Field</code> constraints for validation; never put business rules in the model.</li> <li>Use services for cross-repository logic. A <code>Service</code> composes multiple repositories (and <code>TransactionManager</code>) \u2014 routes stay thin.</li> <li>Expose <code>get_many(filter=..., skip=..., limit=..., sort=...)</code> instead of raw <code>find</code> for list endpoints \u2014 you get controlled pagination for free.</li> </ol>"},{"location":"04_best_practices/#lifecycle","title":"\ud83d\udd04 Lifecycle","text":"<ol> <li>Connect once, in the lifespan. <code>MongoConnectionManager.lifespan(...)</code> (or explicit <code>connect</code>/<code>disconnect</code>) \u2014 never lazily per request.</li> <li>Construct repositories after <code>connect()</code>. Module-level <code>Repo()</code> before connection raises <code>RuntimeError(\"Database not connected...\")</code>. Use dependencies or construct inside the lifespan/request.</li> <li>Order the cache lifecycle strictly: <code>set_cache_backend(backend)</code> \u2192 <code>initialize_cache()</code> (after connect, before use) \u2192 <code>shutdown_cache()</code> on exit.</li> <li>Register all models up front via <code>ModelRegistry.register(...)</code> and let <code>initialize_all()</code> create indexes once at startup (idempotent).</li> </ol>"},{"location":"04_best_practices/#data-performance","title":"\ud83d\uddc4\ufe0f Data & Performance","text":"<ol> <li>Declare indexes for every hot query. Single-field, composite, and optioned (unique/TTL) specs all work via <code>ModelRegistry.register</code> \u2014 see use case 13.</li> <li>Cache only hot, low-write <code>_id</code> reads. Use <code>CachedBaseRepository</code> for lookups-by-id; invalidate (<code>update</code>/<code>delete</code> handle it) and pick a sensible <code>default_ttl</code>.</li> <li>Populate at the repository boundary. <code>PopulatingRepository</code> resolves refs on read and depopulates on write; do not hand-roll joins in endpoints.</li> <li>Respect the populate invariants: rules name the field that holds the reference and that is replaced; <code>patch()</code> cannot touch FK fields \u2014 use <code>update()</code>.</li> </ol>"},{"location":"04_best_practices/#transactions-errors","title":"\ud83d\udd01 Transactions & Errors","text":"<ol> <li>Use transactions for multi-document writes. <code>TransactionManager.start_session</code> (inline) or <code>execute_transaction</code> (list of ops) \u2014 and pass <code>session=</code> to every collection call inside.</li> <li>Handle the library's real exceptions at the edges: <code>DuplicateKeyError</code> \u2192 409, <code>InvalidId</code> \u2192 400, <code>CircularReferenceError</code> \u2192 409, <code>ValueError</code> guides \u2192 400/422 (see Error Handling).</li> </ol>"},{"location":"04_best_practices/#testing","title":"\ud83e\uddea Testing","text":"<ol> <li>Default to mock-based unit tests. Patch <code>MongoConnectionManager.get_database</code>, use <code>AsyncMock</code> collections and cursor chains \u2014 the whole suite runs without MongoDB (use case 14).</li> <li>Mirror the library tests. <code>tests/test_{repository,populating_repository,cache,registry,transactions}.py</code> are canonical examples of every pattern above.</li> <li>Use type hints end-to-end \u2014 mypy-gated CI (see pyproject) catches drift early.</li> </ol>"},{"location":"04_best_practices/#related","title":"Related","text":"<ul> <li>Overview \u00b7 Core Components \u00b7 Use Cases</li> </ul>"},{"location":"06_error_handling/","title":"Error Handling","text":"<p>What can raise, what it means, and how to map it in a FastAPI app.</p>"},{"location":"06_error_handling/#library-raised-exceptions","title":"\ud83d\udccb Library-Raised Exceptions","text":"Exception Source Meaning / fix <code>RuntimeError(\"Database not connected. Call connect() first.\")</code> <code>get_database()</code> / <code>get_client()</code> and any repository constructed first <code>MongoConnectionManager.connect()</code> hasn't run \u2014 wire the lifespan. <code>RuntimeError(\"No cache backend registered. Call set_cache_backend() first.\")</code> <code>ModelRegistry.initialize_cache()</code> Call <code>set_cache_backend(backend)</code> before <code>initialize_cache()</code>. <code>KeyError(\"Model for collection '...' not registered\")</code> <code>ModelRegistry.get_model()</code> Collection was never <code>register</code>ed (or typo). <code>ValueError(\"Cannot patch FK fields via patch(): ...\")</code> <code>PopulatingRepository.patch()</code> <code>patch</code> must not touch populated ref fields \u2014 use <code>update()</code> with a model. <code>ValueError(\"...contains embedded dict(s) \u2014 run repair script\")</code> <code>_populate</code> on read A FK field holds an embedded document instead of an <code>ObjectId</code> \u2014 migrate the data. <code>ValueError(\"...contains ObjectId \u2014 was populate skipped?\")</code> <code>_depopulate</code> on write A populate-ruled field is still an <code>ObjectId</code> at depopulate time \u2014 the read must have populated it first. <code>CircularReferenceError(collection, doc_id, path)</code> <code>PopulationEngine.populate</code> A <code>(Class, id)</code> pair was revisited \u2014 raise <code>max_depth</code> or fix the graph. <code>ImportError(\"redis package required ... mongo-ops[redis]\")</code> <code>CacheConfig</code> / <code>RedisCacheBackend</code> Missing <code>redis</code> extra. <code>ValueError(\"redis_client required when backend='redis'\")</code> <code>CacheConfig</code> <code>backend=\"redis\"</code> without a client. Pymongo <code>DuplicateKeyError</code> any insert/update Unique index violation (e.g., duplicate email). <code>bson.errors.InvalidId</code> <code>ObjectId(...)</code> on a bad string Wrapped by <code>PyObjectId</code> model validation on API inputs. <p><code>KeyError</code> for <code>ModelRegistry.get_model</code> and the <code>RuntimeError</code>/<code>ValueError</code> guards are by design \u2014 they fail loudly at startup or first call instead of misbehaving silently.</p>"},{"location":"06_error_handling/#fastapi-mapping-example","title":"\ud83d\ude80 FastAPI Mapping Example","text":"Python<pre><code>from fastapi import FastAPI, HTTPException, Request\nfrom fastapi.responses import JSONResponse\nfrom bson.errors import InvalidId\nfrom pymongo.errors import DuplicateKeyError\nfrom mongo_ops.cache import CircularReferenceError\n\n\napp = FastAPI()\n\n\n@app.exception_handler(DuplicateKeyError)\nasync def duplicate_key_handler(_: Request, __: DuplicateKeyError) -> JSONResponse:\n return JSONResponse(status_code=409, content={\"detail\": \"Resource already exists\"})\n\n\n@app.exception_handler(InvalidId)\nasync def invalid_id_handler(_: Request, __: InvalidId) -> JSONResponse:\n return JSONResponse(status_code=400, content={\"detail\": \"Invalid ID format\"})\n\n\n@app.exception_handler(CircularReferenceError)\nasync def circular_ref_handler(_: Request, exc: CircularReferenceError) -> JSONResponse:\n return JSONResponse(\n status_code=409,\n content={\"detail\": f\"Circular reference detected: {exc.path}\"},\n )\n\n\n# Or handle inline for route-specific responses:\n@app.post(\"/users/\")\nasync def create_user(user: User, repo=Depends(get_user_repository)):\n try:\n return await repo.create(user)\n except DuplicateKeyError:\n raise HTTPException(status_code=409, detail=\"User already exists\")\n except Exception as exc:\n raise HTTPException(status_code=500, detail=str(exc))\n\n\n@app.get(\"/users/{user_id}\")\nasync def get_user(user_id: str, repo=Depends(get_user_repository)):\n try:\n user = await repo.get_by_id(user_id)\n except InvalidId:\n raise HTTPException(status_code=400, detail=\"Invalid user ID\")\n if not user:\n raise HTTPException(status_code=404, detail=\"User not found\")\n return user\n</code></pre> <p>Prefer exception handlers for the library-level exceptions (409/400 above) and per-endpoint <code>try/except</code> for domain decisions (404).</p>"},{"location":"06_error_handling/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>The <code>ValueError</code> populate guards are your friends: they surface data-shape drift (embedded docs, skipped population) at the exact call site.</li> <li>In dev, log the <code>CircularReferenceError.path</code> \u2014 it prints the visited <code>Class:id</code> chain.</li> <li>Never swallow <code>RuntimeError</code>s at startup; let the app fail to load so the misconfiguration is obvious.</li> </ul>"},{"location":"06_error_handling/#related","title":"Related","text":"<ul> <li>03 \u2013 Transactions \u00b7 08 \u2013 Population \u00b7 11 \u2013 Cache lifecycle</li> </ul>"},{"location":"07_testing_example/","title":"Testing Example","text":"<p>A zero-MongoDB unit-test quickstart. The full guide is use case 14.</p>"},{"location":"07_testing_example/#mock-based-quickstart","title":"\ud83d\ude80 Mock-Based Quickstart","text":"<p>Patch the connection manager and stub the collection \u2014 no network, no docker.</p> Python<pre><code>import pytest\nfrom unittest.mock import AsyncMock, patch\n\nfrom bson import ObjectId\nfrom mongo_ops import BaseDocument, BaseRepository\nfrom mongo_ops.repository import MongoConnectionManager\n\n\nclass User(BaseDocument):\n username: str = \"\"\n email: str = \"\"\n\n\nclass UserRepository(BaseRepository[User]):\n def __init__(self):\n super().__init__(\"users\", User)\n\n\n@pytest.fixture\ndef repo():\n mock_collection = AsyncMock()\n with patch.object(MongoConnectionManager, \"get_database\") as mock_db:\n mock_db.return_value.__getitem__.return_value = mock_collection\n r = UserRepository()\n r.collection = mock_collection\n return r\n\n\n@pytest.mark.asyncio\nasync def test_create_user(repo):\n mock_collection = repo.collection\n oid = ObjectId()\n mock_collection.insert_one.return_value.inserted_id = oid\n\n user = await repo.create(User(username=\"test\", email=\"test@example.com\"))\n\n assert user.id == oid\n assert user.username == \"test\"\n\n\n@pytest.mark.asyncio\nasync def test_get_by_id(repo):\n oid = ObjectId()\n repo.collection.find_one.return_value = {\n \"_id\": oid,\n \"username\": \"test\",\n \"email\": \"test@example.com\",\n \"created_at\": \"2024-01-01T00:00:00\",\n \"updated_at\": \"2024-01-01T00:00:00\",\n }\n\n fetched = await repo.get_by_id(str(oid))\n\n assert fetched is not None\n assert fetched.username == \"test\"\n</code></pre>"},{"location":"07_testing_example/#notes","title":"\ud83d\udca1 Notes","text":"<ul> <li><code>from mongo_ops.repository import MongoConnectionManager</code> \u2014 patch where it is used (<code>mongo_ops.repository.MongoConnectionManager</code>), matching the library's own tests.</li> <li>pytest-asyncio runs as <code>auto</code> mode per <code>pyproject.toml</code>, so <code>@pytest.mark.asyncio</code> works without extra config.</li> <li>For population, cache, registry, and transaction mockups \u2014 see the full testing guide.</li> </ul>"},{"location":"07_testing_example/#related","title":"Related","text":"<ul> <li>14 \u2013 Testing guide \u00b7 Best Practices</li> </ul>"},{"location":"03_use_cases/01_basic_crud/","title":"Use Case 1: Basic FastAPI CRUD API","text":"<p>Scenario: A simple user management API with CRUD endpoints, index registration, and a correct connection lifecycle.</p>"},{"location":"03_use_cases/01_basic_crud/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>MongoConnectionManager.lifespan</code> Connect on startup, disconnect on shutdown. <code>BaseRepository[User]</code> Generic CRUD \u2014 no endpoint-level Mongo code. <code>ModelRegistry.register</code> Declares the <code>email</code> index; <code>initialize_all</code> creates it at startup."},{"location":"03_use_cases/01_basic_crud/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>import os\nfrom contextlib import asynccontextmanager\n\nfrom fastapi import Depends, FastAPI, HTTPException\nfrom pydantic import Field\n\nfrom mongo_ops import BaseDocument, BaseRepository, ModelRegistry, MongoConnectionManager\n\n# ---------------------------\n# Model\n# ---------------------------\nclass User(BaseDocument):\n username: str = Field(..., min_length=3, max_length=50)\n email: str = Field(...)\n is_active: bool = True\n\n\n# ---------------------------\n# Repository\n# ---------------------------\nclass UserRepository(BaseRepository[User]):\n def __init__(self):\n super().__init__(\"users\", User)\n\n\n# ---------------------------\n# Registration + index\n# ---------------------------\nModelRegistry.register(\"users\", User, indexes=[(\"email\", 1)])\n\n\n# ---------------------------\n# Lifecycle\n# ---------------------------\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=os.getenv(\"MONGO_URI\", \"mongodb://localhost:27017\"),\n db_name=\"mydb\",\n ):\n await ModelRegistry.initialize_all()\n yield\n\n\napp = FastAPI(lifespan=lifespan)\n\n\n# ---------------------------\n# Dependency \u2014 created per request, after connect()\n# ---------------------------\ndef get_user_repository() -> UserRepository:\n return UserRepository()\n\n\n# ---------------------------\n# Routes\n# ---------------------------\n@app.post(\"/users/\", response_model=User)\nasync def create_user(user: User, repo: UserRepository = Depends(get_user_repository)):\n return await repo.create(user)\n\n\n@app.get(\"/users/{user_id}\", response_model=User)\nasync def get_user(user_id: str, repo: UserRepository = Depends(get_user_repository)):\n user = await repo.get_by_id(user_id)\n if not user:\n raise HTTPException(status_code=404, detail=\"User not found\")\n return user\n\n\n@app.get(\"/users/\", response_model=list[User])\nasync def list_users(\n skip: int = 0,\n limit: int = 10,\n repo: UserRepository = Depends(get_user_repository),\n):\n return await repo.get_many(skip=skip, limit=limit)\n\n\n@app.put(\"/users/{user_id}\", response_model=User)\nasync def update_user(\n user_id: str,\n email: str,\n repo: UserRepository = Depends(get_user_repository),\n):\n user = await repo.update(user_id, {\"email\": email})\n if not user:\n raise HTTPException(status_code=404, detail=\"User not found\")\n return user\n\n\n@app.delete(\"/users/{user_id}\")\nasync def delete_user(user_id: str, repo: UserRepository = Depends(get_user_repository)):\n deleted = await repo.delete(user_id)\n if not deleted:\n raise HTTPException(status_code=404, detail=\"User not found\")\n return {\"message\": \"User deleted successfully\"}\n</code></pre>"},{"location":"03_use_cases/01_basic_crud/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li><code>UserRepository()</code> is created inside the dependency, i.e., only after the lifespan has connected. Instantiating a repository at module import time raises <code>RuntimeError</code> because the database isn't connected yet.</li> <li><code>BaseRepository.update(id, {...})</code> takes a dict; only <code>PopulatingRepository.update</code> takes a model.</li> <li>Use <code>patch()</code> for REST PATCH semantics \u2014 it accepts a partial dict like <code>update</code>, but <code>PopulatingRepository</code> rejects FK fields.</li> </ul>"},{"location":"03_use_cases/01_basic_crud/#related","title":"Related","text":"<ul> <li>02 \u2013 Custom repository \u00b7 04 \u2013 Pagination \u00b7 Overview</li> </ul>"},{"location":"03_use_cases/02_custom_repo/","title":"Use Case 2: Custom Repository with Business Logic","text":"<p>Scenario: An e-commerce product catalog needs search, filtering, and stock updates without Mongo leaking into routes.</p>"},{"location":"03_use_cases/02_custom_repo/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description Repository methods Encapsulate queries (<code>$regex</code>, filters, <code>$inc</code>) behind domain methods. <code>get_many</code> Filtering + default pagination via the base repository. Direct <code>collection</code> access For operations with no base-repo helper (regex search, atomic <code>$inc</code>)."},{"location":"03_use_cases/02_custom_repo/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from fastapi import FastAPI, HTTPException, Query\nfrom mongo_ops import BaseDocument, BaseRepository\n\n# ---------------------------\n# Model\n# ---------------------------\nclass Product(BaseDocument):\n name: str\n description: str = \"\"\n price: float = 0.0\n category: str = \"\"\n in_stock: bool = True\n quantity: int = 0\n tags: list[str] = []\n\n\n# ---------------------------\n# Repository\n# ---------------------------\nclass ProductRepository(BaseRepository[Product]):\n def __init__(self):\n super().__init__(\"products\", Product)\n\n async def search_by_name(self, query: str) -> list[Product]:\n \"\"\"Case-insensitive name search.\"\"\"\n docs = await self.collection.find(\n {\"name\": {\"$regex\": query, \"$options\": \"i\"}}\n ).to_list(length=100)\n return [self.model(**doc) for doc in docs]\n\n async def get_by_category(self, category: str, in_stock_only: bool = True) -> list[Product]:\n filter_query = {\"category\": category}\n if in_stock_only:\n filter_query[\"in_stock\"] = True\n return await self.get_many(filter=filter_query)\n\n async def get_low_stock(self, threshold: int = 10) -> list[Product]:\n return await self.get_many(filter={\"quantity\": {\"$lt\": threshold}, \"in_stock\": True})\n\n async def update_stock(self, product_id: str, quantity_delta: int) -> Product | None:\n \"\"\"Atomically increment/decrement stock.\"\"\"\n from bson import ObjectId\n from datetime import datetime\n\n result = await self.collection.find_one_and_update(\n {\"_id\": ObjectId(product_id)},\n {\"$inc\": {\"quantity\": quantity_delta}, \"$set\": {\"updated_at\": datetime.utcnow()}},\n return_document=True,\n )\n return self.model(**result) if result else None\n\n\napp = FastAPI()\nproduct_repo = ProductRepository()\n\n\n@app.get(\"/products/search\", response_model=list[Product])\nasync def search_products(q: str = Query(..., min_length=1)):\n return await product_repo.search_by_name(q)\n\n\n@app.get(\"/products/category/{category}\", response_model=list[Product])\nasync def products_by_category(category: str, in_stock: bool = True):\n return await product_repo.get_by_category(category, in_stock)\n\n\n@app.get(\"/products/low-stock\", response_model=list[Product])\nasync def low_stock_products(threshold: int = 10):\n return await product_repo.get_low_stock(threshold)\n\n\n@app.patch(\"/products/{product_id}/stock\")\nasync def update_product_stock(product_id: str, quantity_delta: int):\n product = await product_repo.update_stock(product_id, quantity_delta)\n if not product:\n raise HTTPException(status_code=404, detail=\"Product not found\")\n return product\n</code></pre> <p>Note: this snippet omits the FastAPI <code>lifespan</code> connection wiring for brevity \u2014 copy it from use case 01 so <code>ProductRepository()</code> is created only after <code>MongoConnectionManager.connect()</code>.</p>"},{"location":"03_use_cases/02_custom_repo/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Methods that hit <code>self.collection</code> directly (regex search, <code>$inc</code>) bypass the caching and population layers. If a feature composes them \u2014 extend <code>CachedBaseRepository</code> or <code>PopulatingRepository</code> instead and add the domain methods there.</li> <li>Prefer <code>get_many(filter=...)</code> over raw <code>find</code> when you want pagination/sort defaults for free.</li> <li>Reuse <code>self.model(**doc)</code> to convert raw dicts to model instances consistently.</li> </ul>"},{"location":"03_use_cases/02_custom_repo/#related","title":"Related","text":"<ul> <li>01 \u2013 Basic CRUD \u00b7 04 \u2013 Pagination \u00b7 05 \u2013 Soft deletes</li> </ul>"},{"location":"03_use_cases/03_transactions/","title":"Use Case 3: Transaction Support for Multi-Document Operations","text":"<p>Scenario: Order processing must update inventory and create an order atomically. Any failure rolls both back.</p>"},{"location":"03_use_cases/03_transactions/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>TransactionManager.start_session</code> Async context manager yielding a session with an active transaction. <code>session=</code> kwarg Pass to every <code>insert_one</code> / <code>update_one</code> / <code>find_one</code> inside the block."},{"location":"03_use_cases/03_transactions/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from contextlib import asynccontextmanager\nfrom datetime import datetime\n\nfrom bson import ObjectId\nfrom fastapi import FastAPI, HTTPException\nfrom mongo_ops import BaseDocument, BaseRepository, MongoConnectionManager, TransactionManager\n\n\nclass Order(BaseDocument):\n user_id: str\n items: list[dict] # [{\"product_id\": \"...\", \"quantity\": 2}]\n total_amount: float = 0.0\n status: str = \"pending\"\n\n\nclass Inventory(BaseDocument):\n product_id: ObjectId = None\n quantity: int = 0\n\n\nclass OrderRepository(BaseRepository[Order]):\n def __init__(self):\n super().__init__(\"orders\", Order)\n\n\nclass InventoryRepository(BaseRepository[Inventory]):\n def __init__(self):\n super().__init__(\"inventory\", Inventory)\n\n\nasync def create_order_with_inventory_update(\n order: Order,\n order_repo: OrderRepository,\n inv_repo: InventoryRepository,\n) -> Order:\n async def insert_order(session):\n doc = order.model_dump(exclude={\"id\"}, exclude_none=True)\n doc[\"created_at\"] = doc[\"updated_at\"] = datetime.utcnow()\n result = await order_repo.collection.insert_one(doc, session=session)\n return await order_repo.collection.find_one({\"_id\": result.inserted_id}, session=session)\n\n async def update_inventory(session):\n for item in order.items:\n await inv_repo.collection.update_one(\n {\"product_id\": ObjectId(item[\"product_id\"])},\n {\n \"$inc\": {\"quantity\": -item[\"quantity\"]},\n \"$set\": {\"updated_at\": datetime.utcnow()},\n },\n session=session,\n )\n return None\n\n async with TransactionManager.start_session() as session:\n created = await insert_order(session)\n await update_inventory(session)\n return Order(**created)\n\n\n# ---------------------------\n# FastAPI wiring\n# ---------------------------\napp = FastAPI()\n\norder_repo = OrderRepository()\ninv_repo = InventoryRepository()\n\n\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"shop\"\n ):\n yield\n\n\napp.lifespan = lifespan\n\n\n@app.post(\"/orders/\", response_model=Order)\nasync def create_order(order: Order):\n try:\n return await create_order_with_inventory_update(order, order_repo, inv_repo)\n except Exception as exc:\n raise HTTPException(status_code=400, detail=str(exc))\n</code></pre>"},{"location":"03_use_cases/03_transactions/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Repositories are created inside the lifespan (or a dependency) so the database is connected \u2014 see the note in use case 01.</li> <li>Every operation inside the transaction context must receive <code>session=<session></code> \u2014 a missing session silently runs outside the transaction.</li> <li>MongoDB transactions require a replica set (or the local standalone test server that emulates one).</li> <li>For a list-of-operations style, see <code>TransactionManager.execute_transaction</code> in use case 12.</li> </ul>"},{"location":"03_use_cases/03_transactions/#related","title":"Related","text":"<ul> <li>06 \u2013 Multi-model \u00b7 12 \u2013 Transaction helper \u00b7 18 \u2013 Aggregation pipelines \u00b7 19 \u2013 Bulk operations \u00b7 Error Handling</li> </ul>"},{"location":"03_use_cases/04_pagination/","title":"Use Case 4: Pagination & Filtering","text":"<p>Scenario: A blog post API lists posts with page metadata, filtering, and sorting.</p>"},{"location":"03_use_cases/04_pagination/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>BaseRepository.get_many</code> <code>filter</code>, <code>skip</code>, <code>limit</code>, <code>sort</code> in one call. <code>BaseRepository.count</code> Total matching documents for page metadata. <code>PaginatedResponse</code> Reusable generic response wrapper."},{"location":"03_use_cases/04_pagination/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from typing import Generic, TypeVar\n\nfrom fastapi import FastAPI, Query\nfrom pydantic import BaseModel\nfrom mongo_ops import BaseDocument, BaseRepository\n\nT = TypeVar(\"T\")\n\n\nclass PaginatedResponse(BaseModel, Generic[T]):\n items: list[T]\n total: int\n page: int\n page_size: int\n total_pages: int\n has_next: bool\n has_prev: bool\n\n\nclass BlogPost(BaseDocument):\n title: str = \"\"\n content: str = \"\"\n author_id: str = \"\"\n published: bool = False\n tags: list[str] = []\n views: int = 0\n\n\nclass BlogPostRepository(BaseRepository[BlogPost]):\n def __init__(self):\n super().__init__(\"blog_posts\", BlogPost)\n\n async def paginate(\n self,\n page: int = 1,\n page_size: int = 10,\n filter_dict: dict | None = None,\n sort_by: str = \"created_at\",\n sort_order: int = -1,\n ) -> PaginatedResponse[BlogPost]:\n filter_dict = filter_dict or {}\n skip = (page - 1) * page_size\n total = await self.count(filter_dict)\n items = await self.get_many(\n filter=filter_dict,\n skip=skip,\n limit=page_size,\n sort=[(sort_by, sort_order)],\n )\n total_pages = (total + page_size - 1) // page_size\n return PaginatedResponse(\n items=items,\n total=total,\n page=page,\n page_size=page_size,\n total_pages=total_pages,\n has_next=page < total_pages,\n has_prev=page > 1,\n )\n\n async def get_by_author(self, author_id: str, published_only: bool = True) -> list[BlogPost]:\n filter_dict = {\"author_id\": author_id}\n if published_only:\n filter_dict[\"published\"] = True\n return await self.get_many(filter=filter_dict, sort=[(\"created_at\", -1)])\n\n async def search_by_tags(self, tags: list[str]) -> list[BlogPost]:\n return await self.get_many(filter={\"tags\": {\"$in\": tags}, \"published\": True})\n\n\napp = FastAPI()\nblog_repo = BlogPostRepository()\n\n\n@app.get(\"/posts/\", response_model=PaginatedResponse[BlogPost])\nasync def list_posts(\n page: int = 1,\n page_size: int = 10,\n published: bool | None = None,\n author_id: str | None = None,\n):\n filter_dict = {}\n if published is not None:\n filter_dict[\"published\"] = published\n if author_id:\n filter_dict[\"author_id\"] = author_id\n return await blog_repo.paginate(page, page_size, filter_dict)\n\n\n@app.get(\"/posts/author/{author_id}\", response_model=list[BlogPost])\nasync def posts_by_author(author_id: str, published: bool = True):\n return await blog_repo.get_by_author(author_id, published)\n\n\n@app.get(\"/posts/tags\", response_model=list[BlogPost])\nasync def posts_by_tags(tags: list[str] = Query(...)):\n return await blog_repo.search_by_tags(tags)\n</code></pre> <p>Note: add the <code>lifespan</code> wiring from use case 01 so <code>blog_repo</code> is created after connection.</p>"},{"location":"03_use_cases/04_pagination/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li><code>get_many</code> calls <code>cursor.to_list(limit)</code>; pass <code>limit=0</code> to skip the limit entirely, otherwise a large explicit limit is safer than unbounded reads.</li> <li>Always <code>count</code> first for stable metadata \u2014 it uses the same filter as the page query.</li> <li>Combine with <code>sort=[(field, -1|1)]</code> for deterministic ordering; create matching indexes to avoid full collection scans (see use case 13).</li> </ul>"},{"location":"03_use_cases/04_pagination/#related","title":"Related","text":"<ul> <li>02 \u2013 Custom repository \u00b7 13 \u2013 Indexes \u00b7 05 \u2013 Soft deletes</li> </ul>"},{"location":"03_use_cases/05_soft_deletes/","title":"Use Case 5: Soft Deletes Pattern","text":"<p>Scenario: Deleting a task should be recoverable. Instead of removing the document, set a tombstone flag and filter it from normal queries.</p>"},{"location":"03_use_cases/05_soft_deletes/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>SoftDeleteDocument</code> Base model carrying <code>is_deleted</code>, <code>deleted_at</code>, <code>deleted_by</code>. <code>SoftDeleteRepository[T]</code> Repository-level soft delete / restore / filtering using <code>update</code> + <code>get_many</code>. <code>permanent_delete</code> Escapes to the real <code>delete</code>."},{"location":"03_use_cases/05_soft_deletes/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from datetime import datetime\n\nfrom fastapi import FastAPI, HTTPException\nfrom mongo_ops import BaseDocument, BaseRepository\n\n\nclass SoftDeleteDocument(BaseDocument):\n is_deleted: bool = False\n deleted_at: datetime | None = None\n deleted_by: str | None = None\n\n\nclass Task(SoftDeleteDocument):\n title: str = \"\"\n description: str = \"\"\n assignee_id: str = \"\"\n status: str = \"pending\"\n priority: str = \"medium\"\n\n\nclass SoftDeleteRepository(BaseRepository[T]):\n async def soft_delete(self, id: str, deleted_by: str | None = None):\n return await self.update(\n id,\n {\"is_deleted\": True, \"deleted_at\": datetime.utcnow(), \"deleted_by\": deleted_by},\n )\n\n async def restore(self, id: str):\n return await self.update(\n id, {\"is_deleted\": False, \"deleted_at\": None, \"deleted_by\": None}\n )\n\n async def get_active(self, skip: int = 0, limit: int = 100):\n return await self.get_many(filter={\"is_deleted\": False}, skip=skip, limit=limit)\n\n async def get_deleted(self, skip: int = 0, limit: int = 100):\n return await self.get_many(filter={\"is_deleted\": True}, skip=skip, limit=limit)\n\n async def permanent_delete(self, id: str) -> bool:\n return await self.delete(id)\n\n\nclass TaskRepository(SoftDeleteRepository[Task]):\n def __init__(self):\n super().__init__(\"tasks\", Task)\n\n\napp = FastAPI()\ntask_repo = TaskRepository()\n\n\n@app.delete(\"/tasks/{task_id}\")\nasync def soft_delete_task(task_id: str, user_id: str):\n task = await task_repo.soft_delete(task_id, deleted_by=user_id)\n if not task:\n raise HTTPException(status_code=404, detail=\"Task not found\")\n return {\"message\": \"Task deleted\", \"task\": task}\n\n\n@app.post(\"/tasks/{task_id}/restore\")\nasync def restore_task(task_id: str):\n task = await task_repo.restore(task_id)\n if not task:\n raise HTTPException(status_code=404, detail=\"Task not found\")\n return {\"message\": \"Task restored\", \"task\": task}\n\n\n@app.get(\"/tasks/\", response_model=list[Task])\nasync def list_active_tasks(skip: int = 0, limit: int = 10):\n return await task_repo.get_active(skip, limit)\n\n\n@app.get(\"/tasks/deleted\", response_model=list[Task])\nasync def list_deleted_tasks(skip: int = 0, limit: int = 10):\n return await task_repo.get_deleted(skip, limit)\n</code></pre> <p>Note: add the <code>lifespan</code> wiring from use case 01 so <code>task_repo</code> is created after connection.</p>"},{"location":"03_use_cases/05_soft_deletes/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li><code>BaseRepository[T]</code> is generic \u2014 subclasses must parameterize it (see <code>SoftDeleteRepository[T]</code> above). <code>from mongo_ops import BaseRepository; BaseRepository[T]</code> works out of the box.</li> <li>Keep a compound index on <code>{\"is_deleted\": 1, \"status\": 1}</code> for active-list queries (see use case 13).</li> <li>Soft-deleted documents should be excluded at the repository boundary, never re-filtered ad hoc in endpoints.</li> </ul>"},{"location":"03_use_cases/05_soft_deletes/#related","title":"Related","text":"<ul> <li>01 \u2013 Basic CRUD \u00b7 13 \u2013 Indexes \u00b7 04 \u2013 Pagination</li> </ul>"},{"location":"03_use_cases/06_multi_model/","title":"Use Case 6: Multi-Model Service with Registration","text":"<p>Scenario: A social app manages <code>users</code>, <code>posts</code>, and <code>comments</code>. Each has its own model, repository, and indexes \u2014 registered centrally and initialized at startup.</p>"},{"location":"03_use_cases/06_multi_model/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>ModelRegistry.register</code> One call per collection \u2014 model + indexes together. <code>ModelRegistry.initialize_all</code> Creates every registered index at startup (idempotent). Repositories One repository class per collection, all sharing the same connection."},{"location":"03_use_cases/06_multi_model/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from contextlib import asynccontextmanager\n\nfrom fastapi import FastAPI\nfrom mongo_ops import BaseDocument, BaseRepository, ModelRegistry, MongoConnectionManager\n\n\nclass User(BaseDocument):\n username: str = \"\"\n email: str = \"\"\n role: str = \"user\"\n\n\nclass Post(BaseDocument):\n title: str = \"\"\n content: str = \"\"\n author_id: str = \"\"\n likes: int = 0\n\n\nclass Comment(BaseDocument):\n post_id: str = \"\"\n user_id: str = \"\"\n content: str = \"\"\n\n\nclass UserRepository(BaseRepository[User]):\n def __init__(self):\n super().__init__(\"users\", User)\n\n\nclass PostRepository(BaseRepository[Post]):\n def __init__(self):\n super().__init__(\"posts\", Post)\n\n\nclass CommentRepository(BaseRepository[Comment]):\n def __init__(self):\n super().__init__(\"comments\", Comment)\n\n\n# ---------------------------\n# Central registration + indexes\n# ---------------------------\nModelRegistry.register(\"users\", User, indexes=[(\"email\", 1), (\"username\", 1)])\nModelRegistry.register(\"posts\", Post, indexes=[(\"author_id\", 1), (\"created_at\", -1)])\nModelRegistry.register(\"comments\", Comment, indexes=[(\"post_id\", 1), (\"user_id\", 1)])\n\n\n# ---------------------------\n# Lifecycle\n# ---------------------------\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"social_app\"\n ):\n await ModelRegistry.initialize_all()\n yield\n\n\napp = FastAPI(lifespan=lifespan)\n\n\n# Created after connect() \u2014 inside the lifespan body is fine, or use a dependency.\nuser_repo = UserRepository()\npost_repo = PostRepository()\ncomment_repo = CommentRepository()\n\n\n@app.post(\"/users/\", response_model=User)\nasync def create_user(user: User):\n return await user_repo.create(user)\n\n\n@app.post(\"/posts/\", response_model=Post)\nasync def create_post(post: Post):\n return await post_repo.create(post)\n\n\n@app.post(\"/comments/\", response_model=Comment)\nasync def create_comment(comment: Comment):\n return await comment_repo.create(comment)\n</code></pre>"},{"location":"03_use_cases/06_multi_model/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Place <code>user_repo = ...</code> inside the lifespan/after connect. Module-level instantiation before <code>connect()</code> raises <code>RuntimeError(\"Database not connected...\")</code>.</li> <li>Model relationships here are plain <code>ObjectId</code> strings stored on the child docs. To resolve them on read, see use case 08 \u2013 Population.</li> <li>Registering indexes on <code>created_at</code>/<code>author_id</code>/<code>post_id</code> keeps the common queries indexed (see use case 13).</li> </ul>"},{"location":"03_use_cases/06_multi_model/#related","title":"Related","text":"<ul> <li>01 \u2013 Basic CRUD \u00b7 08 \u2013 Population \u00b7 13 \u2013 Indexes</li> </ul>"},{"location":"03_use_cases/07_caching/","title":"Use Case 7: Caching for High-Performance Reads","text":"<p>Scenario: A read-heavy API (product catalog) reduces DB load by caching documents by <code>_id</code> \u2014 in-memory locally, or shared via Redis.</p>"},{"location":"03_use_cases/07_caching/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>InMemoryCacheBackend</code> TTL + LRU cache with a background cleanup task. <code>RedisCacheBackend</code> Distributed cache on <code>redis.asyncio</code> with pub/sub invalidation. <code>CachedBaseRepository[T]</code> Extends <code>BaseRepository</code> \u2014 cache-first <code>get_by_id</code>, cache on <code>create</code>, invalidate on <code>update</code>/<code>delete</code>, <code>warm_cache(ids)</code>, <code>invalidate_cache(id)</code>. Backend lifecycle The same backend instance must be both passed to the repository AND registered via <code>ModelRegistry.set_cache_backend</code> so <code>initialize_cache()</code> starts its task."},{"location":"03_use_cases/07_caching/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from contextlib import asynccontextmanager\n\nfrom fastapi import FastAPI, HTTPException\nfrom mongo_ops import BaseDocument, CachedBaseRepository, ModelRegistry, MongoConnectionManager\nfrom mongo_ops.cache import CacheConfig, InMemoryCacheBackend\n\n\nclass Product(BaseDocument):\n name: str = \"\"\n price: float = 0.0\n\n\n# One shared backend \u2014 used by both the repository and the registry lifecycle.\ncache = InMemoryCacheBackend(max_entries=10_000, default_ttl=300)\nModelRegistry.set_cache_backend(cache)\n\n\nclass ProductRepo(CachedBaseRepository[Product]):\n def __init__(self):\n super().__init__(\n collection_name=\"products\",\n model=Product,\n cache_backend=cache,\n config=CacheConfig(enabled=True, backend=\"memory\"),\n )\n\n\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"shop\"\n ):\n await ModelRegistry.initialize_all()\n await ModelRegistry.initialize_cache() # starts the TTL cleanup task\n yield\n await ModelRegistry.shutdown_cache() # cancels it on exit\n\n\napp = FastAPI(lifespan=lifespan)\n\n\n@app.post(\"/products/\", response_model=Product)\nasync def create_product(product: Product):\n return await ProductRepo().create(product) # created after connect()\n\n\n@app.get(\"/products/{product_id}\", response_model=Product)\nasync def get_product(product_id: str):\n product = await ProductRepo().get_by_id(product_id)\n if not product:\n raise HTTPException(status_code=404, detail=\"Product not found\")\n return product\n\n\n@app.put(\"/products/{product_id}\", response_model=Product)\nasync def update_product(product_id: str, name: str | None = None, price: float | None = None):\n data = {}\n if name is not None:\n data[\"name\"] = name\n if price is not None:\n data[\"price\"] = price\n return await ProductRepo().update(product_id, data)\n\n\n# Outside request handlers:\n# await ProductRepo().warm_cache([object_id_1, object_id_2]) # pre-load n ids -> int\n# await ProductRepo().invalidate_cache(object_id_3) # manual eviction\n</code></pre>"},{"location":"03_use_cases/07_caching/#redis-backend","title":"\ud83d\udd04 Redis Backend","text":"<p>Swap the backend \u2014 the repository code stays identical:</p> Python<pre><code>from mongo_ops.cache.redis_backend import RedisCacheBackend\nfrom redis.asyncio import Redis\n\nredis_client = Redis(host=\"localhost\", port=6379)\nredis_backend = RedisCacheBackend(redis_client, key_prefix=\"prod:\")\n\nModelRegistry.set_cache_backend(redis_backend) # for the lifecycle in lifespan()\n\nclass ProductRepo(CachedBaseRepository[Product]):\n def __init__(self):\n super().__init__(\n collection_name=\"products\",\n model=Product,\n cache_backend=redis_backend,\n config=CacheConfig(enabled=True, backend=\"redis\"),\n )\n</code></pre>"},{"location":"03_use_cases/07_caching/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Cache keys are <code>\"{key_prefix}{id}\"</code> (default prefix <code>\"products:\"</code>). <code>clear_pattern(\"products:*\")</code> wipes a whole collection's entries.</li> <li>Values are JSON-encoded (<code>json.dumps(obj, default=str)</code>) \u2014 nested models inside a cached document are stored as dicts, not objects.</li> <li>Enable/disable per repository with <code>CacheConfig(enabled=False)</code>; a disabled repo bypasses the cache entirely.</li> <li>Distinguish the two <code>initialize*</code> calls: <code>initialize_cache()</code> starts the backend task; <code>initialize_all()</code> creates indexes. Both belong in the lifespan, after <code>connect()</code>.</li> </ul>"},{"location":"03_use_cases/07_caching/#related","title":"Related","text":"<ul> <li>11 \u2013 Cache lifecycle \u00b7 10 \u2013 Cache + population \u00b7 Components: cache</li> </ul>"},{"location":"03_use_cases/08_population/","title":"Use Case 8: Document Population","text":"<p>Scenario: An API returns a <code>User</code> with its related <code>Profile</code> embedded in one JSON payload \u2014 no second round-trip from the client, no joins.</p>"},{"location":"03_use_cases/08_population/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>PopulateRule</code> Declares which field to resolve. <code>field_name</code> holds the <code>ObjectId</code> (or <code>list[ObjectId]</code>) and is the same field the resolved document replaces in place. <code>collection_name</code> says where the referenced documents live. <code>PopulationEngine</code> Holds repositories by collection name and resolves rules recursively, detecting cycles (<code>CircularReferenceError</code>). <code>PopulatingRepository[T]</code> <code>get_by_id</code>/<code>get_many</code> populate on read; <code>create</code>/<code>update</code> depopulate on write; <code>patch</code> rejects FK fields. <p>\u26a0\ufe0f <code>PopulateRule</code> does not have a separate \"ref field\" vs \"target field\" \u2014 the ref field is the populated field. <code>filter</code>/<code>projection</code> on <code>PopulateRule</code> are declared but not yet applied by the engine.</p>"},{"location":"03_use_cases/08_population/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from contextlib import asynccontextmanager\n\nfrom fastapi import FastAPI, HTTPException\nfrom mongo_ops import BaseDocument, MongoConnectionManager, ModelRegistry, PopulatingRepository\nfrom mongo_ops.populate import PopulateRule, PopulationEngine\n\n\n# 1. Models \u2014 `profile` holds an ObjectId in the DB and becomes a Profile on read.\nclass Profile(BaseDocument):\n avatar_url: str = \"\"\n bio: str = \"\"\n\n\nclass User(BaseDocument):\n username: str = \"\"\n email: str = \"\"\n profile: Profile | None = None # ObjectId in MongoDB, Profile in memory\n\n\n# 2. Engine + rule \u2014 repositories are registered later (after connect()).\nengine = PopulationEngine({})\nprofile_rule = PopulateRule(\n field_name=\"profile\",\n collection_name=\"profiles\",\n)\n\n\n# 3. The repository the app uses.\nclass UserRepository(PopulatingRepository[User]):\n def __init__(self):\n super().__init__(\n collection_name=\"users\",\n model=User,\n population_engine=engine,\n populate_rules=[profile_rule],\n )\n\n\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"mydb\"\n ):\n # Tells the engine which repository resolves \"profiles\" refs.\n engine.register_repo(\"profiles\", PopulatingRepository[Profile](\"profiles\", Profile))\n await ModelRegistry.initialize_all()\n yield\n\n\napp = FastAPI(lifespan=lifespan)\n\n\n# 4. Writing \u2014 pass the model; the repository depopulates to an ObjectId.\n@app.post(\"/users/\", response_model=User)\nasync def create_user():\n user = User(\n username=\"alice\",\n email=\"alice@example.com\",\n profile=Profile(avatar_url=\"alice.png\", bio=\"hi\"),\n )\n return await UserRepository().create(user)\n # ^ stored as {\"profile\": <ObjectId>}, returned with profile populated\n\n\n# 5. Reading \u2014 get_by_id resolves the reference on the way back.\n@app.get(\"/users/{user_id}\", response_model=User)\nasync def get_user(user_id: str):\n user = await UserRepository().get_by_id(user_id)\n if not user:\n raise HTTPException(status_code=404, detail=\"User not found\")\n return user # user.profile is a Profile instance\n</code></pre>"},{"location":"03_use_cases/08_population/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li><code>Patch</code>ing an FK field raises <code>ValueError</code> \u2014 switch FK changes to <code>update(user_id, model)</code> instead. See use case 05 for the reasoning with soft deletes.</li> <li>Missing references resolve to <code>None</code>, not an error.</li> <li>A field holding an embedded dict (instead of an <code>ObjectId</code>) raises a <code>ValueError</code> (\"run repair script\") \u2014 migrate embedded docs to a separate collection first.</li> <li>If <code>create</code>/<code>update</code> receives a field that is already an <code>ObjectId</code> under a populate rule, <code>_depopulate</code> raises <code>ValueError(\"...was populate skipped?\")</code> \u2014 populate-then-depopulate pairs must be balanced.</li> </ul>"},{"location":"03_use_cases/08_population/#related","title":"Related","text":"<ul> <li>09 \u2013 Advanced population \u00b7 15 \u2013 PopulatingRepository internals \u00b7 17 \u2013 Populate & depopulate examples \u00b7 10 \u2013 Cache + population \u00b7 14 \u2013 Testing guide</li> </ul>"},{"location":"03_use_cases/09_advanced_population/","title":"Use Case 9: Nested Document Population & Circular-Ref Handling","text":"<p>Scenario: An <code>Author</code> has <code>books</code> (a list of references), each <code>Book</code> references a <code>publisher</code>, and an <code>Author</code> may reference a <code>mentor</code> \u2014 which is another <code>Author</code> (a potential cycle).</p>"},{"location":"03_use_cases/09_advanced_population/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>nested_rules</code> Populates deeper levels: resolve <code>books</code> on <code>Author</code>, then <code>publisher</code> inside each <code>Book</code>. <code>max_depth</code> Per-rule recursion bound \u2014 the safety net for cyclic graphs. <code>global_max_depth</code> <code>PopulationEngine(repos, global_max_depth=10)</code> global cap. <code>CircularReferenceError</code> Raised when a <code>(Class, id)</code> pair is revisited; carries the visited path."},{"location":"03_use_cases/09_advanced_population/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from contextlib import asynccontextmanager\n\nfrom bson import ObjectId\nfrom fastapi import FastAPI, HTTPException\nfrom mongo_ops import BaseDocument, ModelRegistry, MongoConnectionManager, PopulatingRepository\nfrom mongo_ops.cache import CircularReferenceError\nfrom mongo_ops.populate import PopulateRule, PopulationEngine\n\n\n# 1. Models \u2014 each ref field holds ObjectId(s) in the DB and becomes model(s) in memory.\nclass Publisher(BaseDocument):\n name: str = \"\"\n country: str = \"\"\n\n\nclass Book(BaseDocument):\n title: str = \"\"\n publisher: Publisher | None = None # ObjectId in DB, Publisher in memory\n\n\nclass Author(BaseDocument):\n name: str = \"\"\n books: list[Book] | None = None # list[ObjectId] in DB, list[Book] in memory\n mentor: \"Author\" | None = None # self-reference \u2014 potential cycle\n\n\n# 2. Engine + nested rules.\nengine = PopulationEngine({})\n\npublisher_rule = PopulateRule(\n field_name=\"publisher\",\n collection_name=\"publishers\",\n)\n\nbook_rule = PopulateRule(\n field_name=\"books\",\n collection_name=\"books\",\n nested_rules=[publisher_rule], # fetch each book, then its publisher\n max_depth=3,\n)\n\nmentor_rule = PopulateRule(\n field_name=\"mentor\",\n collection_name=\"authors\",\n max_depth=2, # stops mentor chains early \u2014 also avoids unbounded cycles\n)\n\n\nclass AuthorRepository(PopulatingRepository[Author]):\n def __init__(self):\n super().__init__(\n collection_name=\"authors\",\n model=Author,\n population_engine=engine,\n populate_rules=[book_rule, mentor_rule],\n )\n\n\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"library\"\n ):\n engine.register_repo(\"publishers\", PopulatingRepository[Publisher](\"publishers\", Publisher))\n engine.register_repo(\"books\", PopulatingRepository[Book](\"books\", Book))\n engine.register_repo(\"authors\", AuthorRepository())\n await ModelRegistry.initialize_all()\n yield\n\n\napp = FastAPI()\n\n\n@app.get(\"/authors/{author_id}\")\nasync def get_author(author_id: str):\n try:\n author = await AuthorRepository().get_by_id(author_id)\n except CircularReferenceError as exc:\n raise HTTPException(status_code=409, detail=f\"Circular reference: {exc}\")\n if not author:\n raise HTTPException(status_code=404, detail=\"Author not found\")\n return author\n</code></pre>"},{"location":"03_use_cases/09_advanced_population/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li><code>CircularReferenceError</code> is a <code>ValueError</code> subtype exposing <code>.collection</code>, <code>.doc_id</code>, and the visited <code>.path</code> \u2014 use it in error responses and logging.</li> <li>Keep <code>max_depth</code> conservative (2\u20133) for most graphs; combine with the global <code>global_max_depth=10</code> default.</li> <li>List refs depopulate back to <code>list[ObjectId]</code> on <code>create</code>/<code>update</code>, mirroring the scalar case.</li> </ul>"},{"location":"03_use_cases/09_advanced_population/#related","title":"Related","text":"<ul> <li>08 \u2013 Population \u00b7 15 \u2013 PopulatingRepository internals \u00b7 17 \u2013 Populate & depopulate examples \u00b7 10 \u2013 Cache + population \u00b7 Error Handling</li> </ul>"},{"location":"03_use_cases/10_cache_and_population/","title":"Use Case 10: Caching + Population (Read-Through, Populated on Read)","text":"<p>Scenario: A service needs fast reads of a <code>User</code> and its related <code>Profile</code>. We want the cache-hit path to skip the DB entirely while still returning a populated document.</p>"},{"location":"03_use_cases/10_cache_and_population/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>CachedBaseRepository</code> Provides the cache layer (<code>cache-first get_by_id</code>, invalidation on <code>update</code>/<code>delete</code>). <code>PopulationEngine</code> Provides the reference resolution. Composed subclass A small <code>CachedBaseRepository</code> subclass adds two <code>_populate</code> helpers so cache hits return populated models. <p>\u26a0\ufe0f Known limitation: there is no built-in <code>CachedPopulatingRepository</code> in the library. This use case documents the composition. We cache the raw (depopulated) document \u2014 references stay <code>ObjectId</code>s until read time, so <code>Profile</code> changes are reflected on the next fetch (within TTL) and <code>update</code>/<code>delete</code> invalidation remains correct.</p>"},{"location":"03_use_cases/10_cache_and_population/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from contextlib import asynccontextmanager\nfrom datetime import datetime\nfrom typing import Any, Union\n\nfrom bson import ObjectId\nfrom fastapi import FastAPI\nfrom mongo_ops import BaseDocument, CachedBaseRepository, ModelRegistry, MongoConnectionManager\nfrom mongo_ops.cache import CacheConfig, InMemoryCacheBackend\nfrom mongo_ops.cache.in_memory import decode_value, encode_value\nfrom mongo_ops.populate import PopulateRule, PopulationEngine\n\n\n# 1. Models \u2014 `profile` holds an ObjectId in DB, a Profile in memory.\nclass Profile(BaseDocument):\n avatar_url: str = \"\"\n bio: str = \"\"\n\n\nclass User(BaseDocument):\n username: str = \"\"\n email: str = \"\"\n profile: Profile | None = None\n\n\n# 2. Engine + rule.\nengine = PopulationEngine({})\nprofile_rule = PopulateRule(field_name=\"profile\", collection_name=\"profiles\")\n\n\n# 3. Composed repository.\nclass CachedUserRepository(CachedBaseRepository[User]):\n def __init__(\n self,\n cache_backend: InMemoryCacheBackend,\n population_engine: PopulationEngine,\n populate_rules: list[PopulateRule],\n config: CacheConfig | None = None,\n ):\n super().__init__(\"users\", User, cache_backend, config)\n self.population_engine = population_engine\n self._populate_rules = populate_rules\n\n async def _populate(self, data: dict[str, Any]) -> dict[str, Any]:\n \"\"\"Dict-level resolution (mirrors PopulatingRepository._populate).\"\"\"\n for rule in self._populate_rules:\n ref = data.get(rule.field_name)\n if ref is None:\n continue\n if isinstance(ref, str): # JSON round-trip turns ObjectId into hex str\n ref = ObjectId(ref)\n if isinstance(ref, list):\n resolved = []\n for item in ref:\n item = ObjectId(item) if isinstance(item, str) else item\n repo = self.population_engine._repos.get(rule.collection_name)\n doc = await repo.get_by_id(item) if repo else None\n resolved.append(doc)\n data[rule.field_name] = resolved\n elif isinstance(ref, ObjectId):\n repo = self.population_engine._repos.get(rule.collection_name)\n data[rule.field_name] = await repo.get_by_id(ref) if repo else None\n return data\n\n async def _depopulate(self, data: User) -> dict[str, Any]:\n \"\"\"Model -> raw dict, FK fields collapsed to ObjectId (mirrors _depopulate).\"\"\"\n doc = data.model_dump(exclude={\"id\"}, exclude_none=True)\n for rule in self._populate_rules:\n value = getattr(data, rule.field_name, None)\n if isinstance(value, list):\n doc[rule.field_name] = [item.id for item in value]\n elif isinstance(value, BaseDocument):\n doc[rule.field_name] = value.id\n return doc\n\n async def create(self, data: User) -> User:\n doc = await self._depopulate(data) # store FK refs as ObjectIds\n doc[\"created_at\"] = datetime.utcnow()\n doc[\"updated_at\"] = datetime.utcnow()\n result = await self.collection.insert_one(doc)\n doc[\"_id\"] = result.inserted_id\n if self._cache_config.enabled:\n await self._cache.set(self._cache_key(doc[\"_id\"]), encode_value(doc), self._cache_config.default_ttl)\n return self.model(**await self._populate(doc))\n\n async def update(self, id: Union[str, ObjectId], data: User) -> User | None:\n if isinstance(id, str):\n id = ObjectId(id)\n doc = await self._depopulate(data) # update() takes a full model here\n doc[\"updated_at\"] = datetime.utcnow()\n result = await self.collection.find_one_and_update(\n {\"_id\": id}, {\"$set\": doc}, return_document=True\n )\n key = self._cache_key(id)\n if result is None:\n if self._cache_config.enabled:\n await self._cache.delete(key)\n return None\n raw = dict(result)\n if self._cache_config.enabled:\n await self._cache.set(key, encode_value(raw), self._cache_config.default_ttl)\n return self.model(**await self._populate(raw))\n\n async def get_by_id(self, id: Union[str, ObjectId]) -> User | None:\n if not self._cache_config.enabled:\n # No cache: still a raw fetch + populate \u2014 the base get_by_id()\n # would build a typed model from the raw doc and fail on FK fields.\n raw = await self.collection.find_one({\"_id\": ObjectId(id) if isinstance(id, str) else id})\n return self.model(**await self._populate(raw)) if raw else None\n\n cached = await self._cache.get(self._cache_key(id))\n if cached is not None:\n # Cache hit: decode the RAW doc, then populate before returning.\n data = await self._populate(decode_value(cached))\n return self.model(**data)\n\n # Cache miss: one RAW DB read. The base get_by_id() would rebuild the\n # model from the raw doc, which fails for model-typed FK fields \u2014\n # so fetch the raw dict, cache it, and populate before materializing.\n key = self._cache_key(id)\n raw = await self.collection.find_one({\"_id\": ObjectId(id) if isinstance(id, str) else id})\n if raw is None:\n return None\n await self._cache.set(key, encode_value(raw), self._cache_config.default_ttl)\n return self.model(**await self._populate(raw))\n\n\n# 4. Wire-up \u2014 one backend for both the repo and the registry lifecycle.\ncache = InMemoryCacheBackend(max_entries=20_000, default_ttl=600)\nModelRegistry.set_cache_backend(cache)\n\n\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"app_db\"\n ):\n # Register the repositories the engine resolves refs against:\n # engine.register_repo(\"profiles\", ProfileRepo())\n engine.register_repo(\"users\", CachedUserRepository(cache, engine, [profile_rule]))\n await ModelRegistry.initialize_all()\n await ModelRegistry.initialize_cache()\n yield\n await ModelRegistry.shutdown_cache()\n\n\napp = FastAPI(lifespan=lifespan)\n</code></pre> <p>Note: replace the placeholder <code>engine.register_repo(\"profiles\", ...)</code> line with a repository for <code>Profile</code>. A matching <code>CachedBaseRepository[Profile](\"profiles\", Profile, cache)</code> keeps profile reads cached too \u2014 e.g. <code>engine.register_repo(\"profiles\", CachedBaseRepository[Profile](\"profiles\", Profile, cache))</code>.</p>"},{"location":"03_use_cases/10_cache_and_population/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Cache the raw doc, populate at read time. Ref fields stay <code>ObjectId</code>s in the cache; JSON round-trips them to hex strings, so <code>_populate</code> must normalize <code>str \u2192 ObjectId</code> (shown above).</li> <li><code>update</code>/<code>create</code> above are model-based (<code>update(id, User)</code>), mirroring <code>PopulatingRepository</code> \u2014 they depopulate before writing and cache a raw snapshot. The plain <code>CachedBaseRepository.update(id, dict)</code> and <code>delete(id)</code> keep working and invalidate the same key.</li> <li>Set a sensible <code>default_ttl</code> \u2014 cached <code>User</code> entries resolve <code>Profile</code> on each read, so profile edits show up within the TTL (or call <code>invalidate_cache(user_id)</code> explicitly).</li> <li>To reuse this compose logic across many collections, extract the <code>_populate</code> / <code>_depopulate</code> helpers plus the <code>create</code> / <code>update</code> / <code>get_by_id</code> overrides into a mixin and parameterize the rules per subclass.</li> </ul>"},{"location":"03_use_cases/10_cache_and_population/#related","title":"Related","text":"<ul> <li>07 \u2013 Caching \u00b7 08 \u2013 Population \u00b7 15 \u2013 PopulatingRepository internals \u00b7 16 \u2013 Cached repository intricacies \u00b7 17 \u2013 Populate & depopulate examples</li> </ul>"},{"location":"03_use_cases/11_cache_lifecycle/","title":"Use Case 11: Proper Cache Lifecycle in FastAPI","text":"<p>Scenario: A microservice uses an in-memory (or Redis) cache backend and must start the cleanup task on app start, then shut it down cleanly on termination.</p>"},{"location":"03_use_cases/11_cache_lifecycle/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>ModelRegistry.set_cache_backend(backend)</code> Registers the single shared backend. <code>ModelRegistry.initialize_cache()</code> Starts the backend (spawns the in-memory TTL cleanup task). Raises <code>RuntimeError</code> if no backend was registered. <code>ModelRegistry.shutdown_cache()</code> Cancels the cleanup task, closes pub/sub, clears the backend. If a repo uses a backend directly Same instance must be registered so <code>initialize_cache</code> starts its task."},{"location":"03_use_cases/11_cache_lifecycle/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from contextlib import asynccontextmanager\n\nfrom fastapi import FastAPI\nfrom mongo_ops import ModelRegistry, MongoConnectionManager\nfrom mongo_ops.cache import InMemoryCacheBackend\n\ncache = InMemoryCacheBackend(max_entries=10_000, default_ttl=300)\nModelRegistry.set_cache_backend(cache) # before any cache-backed repo is used\n\n\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"mydb\"\n ):\n await ModelRegistry.initialize_all() # create indexes (idempotent)\n await ModelRegistry.initialize_cache() # start the TTL cleanup task\n yield\n await ModelRegistry.shutdown_cache() # cancel task + close cleanly\n\n\napp = FastAPI(lifespan=lifespan)\n</code></pre>"},{"location":"03_use_cases/11_cache_lifecycle/#what-actually-happens","title":"\ud83d\udd01 What Actually Happens","text":"<ul> <li><code>InMemoryCacheBackend.initialize()</code> spawns an <code>asyncio.Task</code> that evicts expired entries every <code>cleanup_interval</code> seconds. Without a shutdown, the event loop flags the dangling task on exit \u2014 <code>shutdown_cache()</code> cancels it and awaits it.</li> <li><code>shutdown_cache()</code> also NULs the registry cache backend and closes the Redis pub/sub handle (if Redis).</li> <li>If <code>initialize_cache()</code> is called before <code>set_cache_backend()</code>, it raises:</li> </ul> Text Only<pre><code>RuntimeError: No cache backend registered. Call set_cache_backend() first.\n</code></pre>"},{"location":"03_use_cases/11_cache_lifecycle/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Per-repository toggling is independent of the lifecycle: <code>CacheConfig(enabled=False)</code> bypasses the cache for that repo even after <code>initialize_cache()</code>.</li> <li>The cleanup task uses <code>cleanup_interval</code> seconds for scans; entries also expire on access via the TTL heap (<code>default_ttl=0</code> expires immediately).</li> <li>Register the backend before constructing any <code>CachedBaseRepository</code> that references it \u2014 otherwise a repo may hold an uninitialized backend (no cleanup task, no Redis pub/sub).</li> </ul>"},{"location":"03_use_cases/11_cache_lifecycle/#related","title":"Related","text":"<ul> <li>07 \u2013 Caching \u00b7 Components: ModelRegistry \u00b7 10 \u2013 Cache + population</li> </ul>"},{"location":"03_use_cases/12_transaction_helper/","title":"Use Case 12: Using <code>TransactionManager.execute_transaction</code>","text":"<p>Scenario: Perform several writes across different collections atomically \u2014 e.g., create an <code>Order</code> and decrement <code>Inventory</code>. The low-level <code>start_session</code> context works, but <code>execute_transaction</code> collects results from a list of async operations.</p>"},{"location":"03_use_cases/12_transaction_helper/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>TransactionManager.execute_transaction</code> Runs a list of async callables (each receives a session) inside one transaction; returns the result of each callable, in order. Automatic rollback Any raised exception aborts the transaction and propagates to the caller. <code>start_session</code> The underlying async context manager <code>async with TransactionManager.start_session() as session:</code>."},{"location":"03_use_cases/12_transaction_helper/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from contextlib import asynccontextmanager\nfrom datetime import datetime\n\nfrom bson import ObjectId\nfrom fastapi import FastAPI, HTTPException\nfrom mongo_ops import BaseDocument, ModelRegistry, MongoConnectionManager\nfrom mongo_ops.repository import BaseRepository\nfrom mongo_ops.transactions import TransactionManager\n\n\n# 1. Models\nclass Order(BaseDocument):\n user_id: str = \"\"\n items: list[dict] = [] # [{\"product_id\": ObjectId, \"qty\": int}]\n total: float = 0.0\n\n\nclass Inventory(BaseDocument):\n product_id: ObjectId = None\n quantity: int = 0\n\n\n# 2. Repositories (constructed inside the lifespan \u2014 after connect()).\nclass OrderRepo(BaseRepository[Order]):\n def __init__(self):\n super().__init__(\"orders\", Order)\n\n\nclass InventoryRepo(BaseRepository[Inventory]):\n def __init__(self):\n super().__init__(\"inventory\", Inventory)\n\n\n# 3. The atomic operation.\nasync def create_order_with_inventory(order: Order, order_repo: OrderRepo, inv_repo: InventoryRepo):\n async def insert_order(session):\n doc = order.model_dump(exclude={\"id\"}, exclude_none=True)\n doc[\"created_at\"] = doc[\"updated_at\"] = datetime.utcnow()\n result = await order_repo.collection.insert_one(doc, session=session)\n return await order_repo.collection.find_one({\"_id\": result.inserted_id}, session=session)\n\n async def update_inventory(session):\n for item in order.items:\n await inv_repo.collection.update_one(\n {\"product_id\": ObjectId(item[\"product_id\"])},\n {\"$inc\": {\"quantity\": -item[\"qty\"]}},\n session=session,\n )\n return \"inventory-updated\"\n\n results = await TransactionManager.execute_transaction(\n [insert_order, update_inventory]\n )\n return results[0] # created order document; results[1] == \"inventory-updated\"\n\n\n# 4. FastAPI wiring.\napp = FastAPI()\n\n\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"shop\"\n ):\n await ModelRegistry.initialize_all()\n yield\n\n\napp.lifespan = lifespan\n\n\n@app.post(\"/orders/\", response_model=Order)\nasync def create_order(order: Order):\n created = await create_order_with_inventory(order, OrderRepo(), InventoryRepo())\n if created is None:\n raise HTTPException(status_code=400, detail=\"Transaction failed\")\n return created\n</code></pre>"},{"location":"03_use_cases/12_transaction_helper/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Return values: each callable can return whatever you need; results are collected in the same order.</li> <li>Errors: raise inside any callable \u2192 the whole transaction aborts (session rolls back) and the exception propagates.</li> <li>Reads inside a transaction: pass <code>session=session</code> to <code>find_one</code>/<code>find</code> too.</li> <li>Testing (no Mongo): <code>monkeypatch</code> a fake client on <code>mongo_ops.transactions.MongoConnectionManager.get_client</code> and stub <code>start_session</code> \u2014 see <code>tests/test_transactions.py</code>.</li> <li>Repo models still carry <code>created_at</code>/<code>updated_at</code>; for raw collection inserts inside the transaction you set them manually (as shown).</li> </ul>"},{"location":"03_use_cases/12_transaction_helper/#related","title":"Related","text":"<ul> <li>03 \u2013 Transactions \u00b7 18 \u2013 Aggregation pipelines \u00b7 19 \u2013 Bulk operations \u00b7 14 \u2013 Testing guide \u00b7 Error Handling</li> </ul>"},{"location":"03_use_cases/13_index_creation/","title":"Use Case 13: Declaring Indexes (Single-Field, Composite, Unique)","text":"<p>Scenario: Ensure each collection has the right indexes for fast queries and data integrity \u2014 declared in one place and created at startup.</p>"},{"location":"03_use_cases/13_index_creation/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>ModelRegistry.register(indexes=...)</code> Each spec is passed as-is to pymongo <code>create_index</code>. Supported forms: tuple <code>(\"field\", direction)</code>, compound list <code>[(\"a\", 1), (\"b\", -1)]</code>, or a dict with <code>keys</code> + <code>options</code>. <code>ModelRegistry.initialize_all</code> Creates every registered index during startup (idempotent \u2014 <code>create_index</code> skips existing indexes)."},{"location":"03_use_cases/13_index_creation/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from contextlib import asynccontextmanager\n\nfrom pymongo import ASCENDING, DESCENDING\nfrom mongo_ops import BaseDocument, ModelRegistry, MongoConnectionManager\n\n\nclass User(BaseDocument):\n username: str = \"\"\n email: str = \"\"\n\n\nclass BlogPost(BaseDocument):\n author_id: str = \"\"\n created_at: str = \"\"\n title: str = \"\"\n\n\nclass Passenger(BaseDocument):\n email: str = \"\"\n seat: str = \"\"\n\n\n# 1\ufe0f\u20e3 Single-field index \u2014 email lookups.\nModelRegistry.register(\n collection_name=\"users\",\n model=User,\n indexes=[(\"email\", ASCENDING)],\n)\n\n# 2\ufe0f\u20e3 Composite index \u2014 queries filtering by author + creation date.\nModelRegistry.register(\n collection_name=\"posts\",\n model=BlogPost,\n indexes=[[(\"author_id\", ASCENDING), (\"created_at\", DESCENDING)]],\n)\n\n# 3\ufe0f\u20e3 Unique index with a custom name \u2014 enforce unique emails.\nModelRegistry.register(\n collection_name=\"passengers\",\n model=Passenger,\n indexes=[\n {\n \"keys\": [(\"email\", ASCENDING)],\n \"options\": {\"unique\": True, \"name\": \"uq_passenger_email\"},\n }\n ],\n)\n\n\n# 4\ufe0f\u20e3 Everything is created on startup.\n@asynccontextmanager\nasync def lifespan(_app):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\",\n db_name=\"mydb\",\n ):\n await ModelRegistry.initialize_all()\n yield\n</code></pre>"},{"location":"03_use_cases/13_index_creation/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Every spec passes through to <code>collection.create_index(spec)</code> \u2014 so MongoDB options like <code>unique</code>, <code>sparse</code>, and <code>expireAfterSeconds</code> (TTL) belong in the <code>options</code> dict.</li> <li>Idempotent by construction: <code>create_index</code> is a no-op when a same-shape index already exists.</li> <li>Verify with the Mongo shell:</li> </ul> JavaScript<pre><code>db.<collection>.getIndexes()\n</code></pre> <ul> <li>A unique index on an already-duplicated field will fail with <code>DuplicateKeyError</code> on startup \u2014 clean the data first.</li> </ul>"},{"location":"03_use_cases/13_index_creation/#related","title":"Related","text":"<ul> <li>01 \u2013 Basic CRUD \u00b7 06 \u2013 Multi-model \u00b7 Components: ModelRegistry</li> </ul>"},{"location":"03_use_cases/14_testing_guide/","title":"Use Case 14: Testing Guide \u2014 Mocking Motor, Cache, Registry & Transactions","text":"<p>Scenario: Write unit tests that never touch a real MongoDB \u2014 mock the collection, the cache backend, and the client, exactly like the library's own test suite (<code>tests/</code>).</p>"},{"location":"03_use_cases/14_testing_guide/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>AsyncMock</code> collections Stub <code>find_one</code>, <code>insert_one</code>, <code>find_one_and_update</code>, cursor chains. Patching <code>MongoConnectionManager.get_database</code> Gives repositories a mock collection without a live connection. <code>monkeypatch</code> on <code>get_client</code> Fakes sessions for <code>TransactionManager</code> tests. Reference tests <code>tests/test_populating_repository.py</code>, <code>tests/test_cache.py</code>, <code>tests/test_registry.py</code>, <code>tests/test_transactions.py</code>."},{"location":"03_use_cases/14_testing_guide/#example-boilerplate","title":"\ud83d\ude80 Example Boilerplate","text":"Python<pre><code>import pytest\nfrom unittest.mock import AsyncMock, MagicMock, patch\n\nfrom bson import ObjectId\nfrom mongo_ops.cache import CacheConfig, InMemoryCacheBackend\nfrom mongo_ops.cache.repository import CachedBaseRepository\nfrom mongo_ops.models import BaseDocument\nfrom mongo_ops.registry import ModelRegistry\nfrom mongo_ops.populate import PopulateRule, PopulationEngine\nfrom mongo_ops.repository import PopulatingRepository\n\n\n# ----------------------------------------------------------------------\n# 1. Models (same shape as the library tests)\n# ----------------------------------------------------------------------\nclass Profile(BaseDocument):\n avatar_url: str = \"\"\n\n\nclass User(BaseDocument):\n name: str = \"\"\n profile: Profile | None = None # ObjectId in DB, Profile in memory\n\n\n# ----------------------------------------------------------------------\n# 2. Fixtures\n# ----------------------------------------------------------------------\n@pytest.fixture\ndef mock_collection():\n return AsyncMock()\n\n\n@pytest.fixture\ndef engine():\n profile_repo = AsyncMock()\n return PopulationEngine({\"profiles\": profile_repo})\n\n\n@pytest.fixture\ndef repo(mock_collection, engine):\n with patch(\"mongo_ops.repository.MongoConnectionManager.get_database\") as mock_db:\n mock_db.return_value.__getitem__.return_value = mock_collection\n r = PopulatingRepository(\n \"users\",\n User,\n population_engine=engine,\n populate_rules=[PopulateRule(field_name=\"profile\", collection_name=\"profiles\")],\n )\n r.collection = mock_collection\n return r\n\n\n# ----------------------------------------------------------------------\n# 3. Population \u2014 get_by_id resolves the reference\n# ----------------------------------------------------------------------\n@pytest.mark.asyncio\nasync def test_get_by_id_populates(repo, mock_collection, engine):\n uid, pid = ObjectId(), ObjectId()\n mock_collection.find_one.return_value = {\n \"_id\": uid,\n \"name\": \"Alice\",\n \"profile\": pid, # ObjectId stored in DB\n \"created_at\": \"2024-01-01T00:00:00\",\n \"updated_at\": \"2024-01-01T00:00:00\",\n }\n engine._repos[\"profiles\"].get_by_id.return_value = Profile(id=pid, avatar_url=\"pic.png\")\n\n result = await repo.get_by_id(uid)\n\n assert result is not None\n assert result.name == \"Alice\"\n assert isinstance(result.profile, Profile)\n assert result.profile.avatar_url == \"pic.png\"\n\n\n# ----------------------------------------------------------------------\n# 4. Patch FK guard\n# ----------------------------------------------------------------------\n@pytest.mark.asyncio\nasync def test_patch_rejects_fk_field(repo):\n with pytest.raises(ValueError, match=\"Cannot patch FK fields\"):\n await repo.patch(ObjectId(), {\"profile\": ObjectId()})\n\n\n# ----------------------------------------------------------------------\n# 5. Registry \u2014 index specs pass through to create_index\n# ----------------------------------------------------------------------\n@pytest.mark.asyncio\nasync def test_initialize_all_creates_indexes():\n ModelRegistry.register(\"users\", User, indexes=[(\"email\", 1)])\n\n fake_collection = AsyncMock()\n await ModelRegistry.initialize_all(db={\"users\": fake_collection})\n\n fake_collection.create_index.assert_awaited_once_with((\"email\", 1))\n\n\n# ----------------------------------------------------------------------\n# 6. Cached repository \u2014 cache-first reads + invalidation\n# ----------------------------------------------------------------------\n@pytest.mark.asyncio\nasync def test_cached_get_by_id_populates_cache():\n backend = InMemoryCacheBackend(\n max_entries=100, default_ttl=300, cleanup_interval=9999\n )\n await backend.initialize()\n try:\n with patch(\"mongo_ops.repository.MongoConnectionManager.get_database\") as mock_db:\n mock_collection = AsyncMock()\n mock_db.return_value.__getitem__.return_value = mock_collection\n repo = CachedBaseRepository(\n \"users\", User, backend, CacheConfig(enabled=True)\n )\n repo.collection = mock_collection\n\n oid = ObjectId()\n mock_collection.find_one.return_value = {\n \"_id\": oid,\n \"name\": \"cached\",\n \"created_at\": \"2024-01-01T00:00:00\",\n \"updated_at\": \"2024-01-01T00:00:00\",\n }\n\n first = await repo.get_by_id(oid)\n assert first is not None\n\n mock_collection.find_one.return_value = None # DB now \"empty\"\n second = await repo.get_by_id(oid) # served from cache\n\n assert second is not None\n assert second.name == \"cached\"\n mock_collection.find_one.assert_awaited_once() # only one DB read\n finally:\n await backend.shutdown()\n\n\n# ----------------------------------------------------------------------\n# 7. Transactions \u2014 fake the client's start_session\n# ----------------------------------------------------------------------\nfrom mongo_ops.transactions import TransactionManager\n\n\n@pytest.mark.asyncio\nasync def test_execute_transaction(monkeypatch):\n session_ctx = AsyncMock()\n session_ctx.__aenter__.return_value = AsyncMock()\n session_ctx.__aexit__.return_value = None\n\n client = MagicMock()\n client.start_session = AsyncMock(return_value=session_ctx)\n\n # NOTE: start_transaction must return a context manager, not a coroutine.\n async_session = session_ctx.__aenter__.return_value\n async_session.start_transaction = lambda **_: session_ctx\n\n monkeypatch.setattr(\n \"mongo_ops.transactions.MongoConnectionManager.get_client\",\n lambda: client,\n )\n\n async def fake_op(session):\n return \"ok\"\n\n results = await TransactionManager.execute_transaction([fake_op])\n assert results == [\"ok\"]\n</code></pre>"},{"location":"03_use_cases/14_testing_guide/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>pytest-asyncio is already configured in <code>pyproject.toml</code> (<code>asyncio_mode = \"auto\"</code>), so <code>@pytest.mark.asyncio</code> tests work out of the box. Run with <code>pytest</code> (coverage reports are enabled there too).</li> <li>Never hit the network. Keep the patches in fixtures (or a <code>conftest.py</code>) and reuse them.</li> <li>Mock <code>cursor</code> chains with <code>MagicMock()</code> + <code>.to_list = AsyncMock(...)</code>, exactly like <code>tests/test_repository.py</code>.</li> <li>For an optional integration check (real Mongo), use <code>MongoConnectionManager.lifespan</code> against a local replica set and drop the test database in teardown \u2014 keep it separate from the unit suite.</li> <li>The pattern works symmetrically for Redis: mock the <code>RedisCacheBackend</code> methods (<code>get</code>, <code>set</code>, <code>delete</code>) \u2014 no Redis process required.</li> </ul>"},{"location":"03_use_cases/14_testing_guide/#related","title":"Related","text":"<ul> <li>08 \u2013 Population \u00b7 07 \u2013 Caching \u00b7 12 \u2013 Transaction helper</li> </ul>"},{"location":"03_use_cases/15_populating_repository_wiring/","title":"Use Case 15: Inside <code>PopulatingRepository</code> \u2014 the Object \u21c4 ObjectId Lifecycle","text":"<p>Scenario: You want to see what the repository actually does, in which order, before you trust it with your data \u2014 how a <code>Profile</code> becomes an <code>ObjectId</code> for storage and comes back as a <code>Profile</code> on read, and where every piece is wired.</p>"},{"location":"03_use_cases/15_populating_repository_wiring/#two-representations-one-field","title":"\ud83d\udce6 Two Representations, One Field","text":"<p>A populate-ruled field is a shape shifter \u2014 the same name holds different things depending on where you look:</p> Place Value held in the field In the MongoDB doc <code>ObjectId</code> (or <code>list[ObjectId]</code>) In the cache/JSON hex string (see use case 16) In the app model the referenced model (or <code>None</code>) <p><code>PopulatingRepository.create</code> / <code>update</code> depopulate (model \u2192 <code>ObjectId</code>) before writing; <code>get_by_id</code> / <code>get_many</code> populate (<code>ObjectId</code> \u2192 model) after reading:</p> Text Only<pre><code>App model MongoDB document App model\nUser(profile=Profile) \u2500\u2500depopulate\u2500\u2500\u25b6 { profile: <ObjectId> } \u2500\u2500populate\u2500\u2500\u25b6 User(profile=Profile)\n (write path) (storage shape) (read path)\n</code></pre>"},{"location":"03_use_cases/15_populating_repository_wiring/#write-path-create-step-by-step","title":"\ud83d\ude80 Write Path \u2014 <code>create()</code> Step by Step","text":"<p>Given <code>user = User(username=\"alice\", profile=saved_profile)</code> where <code>saved_profile.id</code> exists:</p> Python<pre><code>class UserRepository(PopulatingRepository[User]):\n def __init__(self):\n super().__init__(\n collection_name=\"users\",\n model=User,\n population_engine=engine,\n populate_rules=[profile_rule],\n )\n</code></pre> <ol> <li>Guard check (<code>_depopulate</code>, <code>repository.py</code>): for each rule field, if the value is already an <code>ObjectId</code> (or <code>list[ObjectId]</code>) an error is raised \u2014 see _depopulate guards below. Saved models pass.</li> <li><code>engine.depopulate(user, rules)</code> collapses the graph in place:</li> <li><code>saved_profile</code> (a <code>BaseDocument</code> with <code>.id</code>) \u2192 becomes <code>saved_profile.id</code> \u2192 <code>ObjectId</code></li> <li>a <code>Profile</code> without <code>.id</code> (unsaved) \u2192 is left as a model object, which <code>model_dump</code> then embeds as a dict \u2014 see intricacy #3</li> <li>when <code>nested_rules</code> are present, the nested model is depopulated recursively first, then collapsed</li> <li><code>model_dump(exclude={\"id\"}, exclude_none=True)</code> produces the raw insert dict; timestamps are added.</li> <li><code>insert_one(doc)</code> writes <code>{..., \"profile\": ObjectId(\"...\"), ...}</code> to MongoDB.</li> <li>On the way out, <code>data_to_model</code> runs <code>_populate</code> (the read path below) so <code>create</code> returns a fully populated model, not the raw one.</li> </ol>"},{"location":"03_use_cases/15_populating_repository_wiring/#the-_depopulate-guards-loud-failures-silent-corruption","title":"The <code>_depopulate</code> guards (loud failures > silent corruption)","text":"Stored/held value under a rule field Behaviour <code>ObjectId</code> / <code>list[ObjectId]</code> <code>ValueError(\"...contains ObjectId \u2014 was populate skipped?\")</code> \u2014 the doc was loaded raw (e.g. via <code>BaseRepository</code> or an unpopulated read) and handed back to <code>create</code>/<code>update</code> a saved <code>BaseDocument</code> (has <code>.id</code>) collapse to <code>.id</code> (<code>ObjectId</code>) an unsaved <code>BaseDocument</code> (no <code>.id</code>) kept as a model \u2192 embedded dict below a <code>dict</code> not <code>BaseDocument</code> \u2192 left as-is \u2192 embedded non-<code>BaseDocument</code>, non-dict value <code>AttributeError</code> raised by <code>engine.depopulate</code> no engine / no rules passthrough \u2014 <code>_depopulate</code> just does <code>model_dump</code>, whatever shape you gave is stored"},{"location":"03_use_cases/15_populating_repository_wiring/#read-path-get_by_id-step-by-step","title":"\ud83d\ude80 Read Path \u2014 <code>get_by_id()</code> Step by Step","text":"<p><code>get_by_id</code> inherits CRUD and only changes <code>data_to_model</code> (<code>repository.py:247</code>):</p> Python<pre><code>doc = await self.collection.find_one({\"_id\": id}) # {\"profile\": ObjectId, ...}\nreturn await self.data_to_model(doc) # data_to_model \u2192 self._populate(doc)\n</code></pre> <p><code>_populate(data)</code> walks each rule field in the raw dict:</p> <ol> <li><code>ref = data.get(\"profile\")</code> \u2192 <code>ObjectId</code>. If <code>ref is None</code> \u2192 field untouched (stays absent).</li> <li>Look up the repo: <code>repo = engine._repos.get(\"profiles\")</code>. If it's not registered, <code>doc = None</code>.</li> <li><code>doc = await repo.get_by_id(ref)</code> \u2014 a single read on the referenced collection.</li> <li>If <code>doc</code> is <code>None</code> \u2192 <code>data[\"profile\"] = None</code> (missing refs resolve to <code>None</code>, never raise).</li> <li>If the rule has <code>nested_rules</code> \u2192 <code>engine.populate(doc, nested_rules, depth=1)</code> deepens the result.</li> <li><code>data[\"profile\"] = doc</code> \u2192 <code>self.model(**data)</code> builds the <code>User</code> with a real <code>Profile</code>.</li> </ol> <p>Lists behave the same per item; a <code>ref</code> that is a dict (an embedded document) raises <code>ValueError(\"...contains embedded dict(s) \u2014 run repair script\")</code>.</p> <p>Engine is optional. Set <code>population_engine=None</code> and <code>populate_rules=[]</code>, and <code>PopulatingRepository</code> is just a <code>BaseRepository</code> \u2014 FK fields come back as raw <code>ObjectId</code>, and your model field type must agree (that is the whole point of the guard in the WRITE path: a <code>Profile | None</code>-typed field populated with a raw <code>ObjectId</code> is a broken round-trip waiting to happen).</p>"},{"location":"03_use_cases/15_populating_repository_wiring/#how-its-wired","title":"\ud83d\udd0c How It's Wired","text":"Python<pre><code>from mongo_ops import BaseDocument, PopulatingRepository\nfrom mongo_ops.populate import PopulateRule, PopulationEngine\n\nclass Profile(BaseDocument):\n avatar_url: str = \"\"\n\nclass User(BaseDocument):\n username: str = \"\"\n profile: Profile | None = None # ObjectId in DB, Profile in memory\n\nengine = PopulationEngine({}) # repositories live here, keyed by collection name\n\nprofile_rule = PopulateRule(field_name=\"profile\", collection_name=\"profiles\")\n\nclass UserRepository(PopulatingRepository[User]):\n def __init__(self):\n super().__init__(\"users\", User, population_engine=engine, populate_rules=[profile_rule])\n\ndef wire() -> None:\n \"\"\"Call AFTER connect() \u2014 repositories need a live database.\"\"\"\n engine.register_repo(\"profiles\", PopulatingRepository[Profile](\"profiles\", Profile))\n engine.register_repo(\"users\", UserRepository())\n</code></pre> <p>Wiring rules:</p> <ul> <li><code>engine._repos</code> is keyed by <code>collection_name</code> as written in the rule \u2014 typo \u2192 silent <code>None</code> refs.</li> <li><code>register_repo</code> needs an already-constructed repository \u2192 call it inside the lifespan (after <code>connect()</code> / <code>MongoConnectionManager.lifespan</code>), not at module import.</li> <li>The referenced repository only needs a <code>get_by_id</code> that returns a <code>BaseDocument</code> \u2014 it can be a plain <code>BaseRepository</code>, another <code>PopulatingRepository</code>, or even a cached repo (see use case 10). Population requires no extra DB index on the referenced collection's <code>_id</code>.</li> <li>Swap at runtime: <code>repo.set_population_engine(new_engine)</code> and <code>repo.set_populate_rules(new_rules)</code> \u2014 the tests exercise both.</li> </ul>"},{"location":"03_use_cases/15_populating_repository_wiring/#intricacy-nested_rules-do-not-round-trip-through-depopulate","title":"\u26a0\ufe0f Intricacy \u2014 <code>nested_rules</code> Do NOT Round-Trip Through <code>depopulate</code>","text":"<p><code>nested_rules</code> are designed for read-side deep population. If the same rules run through the write path (<code>create</code> / <code>update</code>), <code>engine.depopulate</code> behaves differently from plain refs \u2014 verified against the engine:</p> Rule shape What <code>depopulate</code> does to it Effect on the stored doc scalar FK, no <code>nested_rules</code> collapse to <code>.id</code> stored as <code>ObjectId</code> \u2713 <code>list[ObjectId]</code> FK, no <code>nested_rules</code> collapse each item to <code>.id</code> stored as <code>list[ObjectId]</code> \u2713 <code>list</code> FK with <code>nested_rules</code> items are kept as models stored as embedded dicts \u2717 \u2014 reading back raises <code>\"contains embedded dict(s) \u2014 run repair script\"</code> scalar FK with <code>nested_rules</code> the recursion runs, then the field is assigned <code>None</code> reference lost \u2717 \u2014 written as absent/null <p>In other words: a document carrying deep <code>nested_rules</code> (like UC 09's <code>Author \u2192 books \u2192 publisher</code>) cannot be <code>create</code>d / <code>update</code>d as-is. If the graph must be written back, materialize references separately (save each <code>Book</code> to its collection first, then store <code>list[ObjectId]</code> without <code>nested_rules</code>), and keep <code>nested_rules</code> only on read rules you never hand back to <code>depopulate</code>.</p>"},{"location":"03_use_cases/15_populating_repository_wiring/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Never pass a raw <code>ObjectId</code>-holding model to <code>create</code>/<code>update</code>: the \"was populate skipped?\" <code>ValueError</code> is the guard. Read through the repo so the read path can populate first.</li> <li>Unsaved references are a write-once trap: they embed as dicts, and re-reading raises the \"run repair script\" <code>ValueError</code>. Save referenced docs to their collection first, then reference their <code>id</code>.</li> <li><code>filter</code> / <code>projection</code> on <code>PopulateRule</code> are declared but not applied by the engine \u2014 don't rely on them.</li> <li>Reads cost 1 query per reference (no <code>$lookup</code> yet); batch-heavy endpoints should add caching (next use case).</li> </ul>"},{"location":"03_use_cases/15_populating_repository_wiring/#related","title":"Related","text":"<ul> <li>08 \u2013 Population \u00b7 09 \u2013 Advanced population \u00b7 17 \u2013 Populate & depopulate examples \u00b7 16 \u2013 Cached repository intricacies \u00b7 Components</li> </ul>"},{"location":"03_use_cases/16_cached_repository_intricacies/","title":"Use Case 16: Inside the Cached Repository \u2014 What's Actually Stored & Returned","text":"<p>Scenario: You want the exact contract of <code>CachedBaseRepository</code> \u2014 what goes into the cache, in what shape, and why a cached read occasionally looks \"wrong\" for populated models \u2014 before wiring it into a service.</p>"},{"location":"03_use_cases/16_cached_repository_intricacies/#the-cache-contract","title":"\ud83d\udce6 The Cache Contract","text":"Aspect Value Key <code>\"{key_prefix}{id}\"</code> \u2014 default prefix <code>\"{collection_name}:\"</code> Value <code>json.dumps(model_dump(by_alias=True), default=str)</code> \u2014 a byte string of JSON <code>get_by_id</code> hit <code>decode_value(cached)</code> \u2192 <code>self.model(**data)</code> \u2014 no DB hit <code>get_by_id</code> miss DB read, then the raw doc is cached (<code>model_dump(by_alias=True)</code>) <code>create</code> creates in DB, then caches the result (<code>model_dump</code>) <code>update</code> DB update, then set new value or delete the key if the doc vanished <code>delete</code> DB delete, then removes the key TTL <code>config.default_ttl</code> (default 300 s); in-memory eviction is LRU + expiry <p>So the cache stores JSON snapshots of whole documents \u2014 it is a read-through cache keyed by document id, not a query cache.</p>"},{"location":"03_use_cases/16_cached_repository_intricacies/#intricacy-1-objectids-become-hex-strings","title":"\u26a0\ufe0f Intricacy #1 \u2014 ObjectIds Become Hex Strings","text":"<p><code>json.dumps(..., default=str)</code> stringifies every non-JSON value \u2014 most importantly an <code>ObjectId</code> in a FK field:</p> Python<pre><code># model in memory: User(id=..., profile=ObjectId(\"507f1f77bcf86cd799439011\"))\n# cached bytes: b'{\"_id\":\"507f1f77bcf86cd799439011\",\"profile\":\"507f1f77bcf86cd799439011\", ...}'\n</code></pre> <p>On a cache hit, <code>self.model(**data)</code> must therefore accept a hex string where the raw doc held an <code>ObjectId</code>. This is fine for: - <code>id</code> (typed <code>PyObjectId</code>, which accepts both <code>str</code> and <code>ObjectId</code>), and - <code>created_at</code> / <code>updated_at</code> (ISO strings coerce to <code>datetime</code>).</p> <p>It is not fine for a field typed as a model.</p>"},{"location":"03_use_cases/16_cached_repository_intricacies/#intricacy-2-model-typed-fk-fields-fail-on-a-cache-hit","title":"\u26a0\ufe0f Intricacy #2 \u2014 Model-Typed FK Fields Fail on a Cache Hit","text":"<p>If your document has a populated field, e.g.:</p> Python<pre><code>class User(BaseDocument):\n profile: Profile | None = None # populate-ruled\n</code></pre> <p>then the base-class hit path <code>self.model(**data)</code> receives <code>profile=\"507f\u2026\"</code> and Pydantic raises a <code>ValidationError</code> \u2014 a string cannot coerce into a <code>Profile</code>. The miss path fails the same way: <code>BaseRepository.get_by_id</code> materializes the model from the raw doc whose <code>profile</code> is an <code>ObjectId</code> \u2014 also a <code>ValidationError</code>. So a plain <code>CachedBaseRepository</code> cannot materialize a model-typed FK field at all, hit or miss \u2014 the problem is the base classes build a typed model straight from raw docs.</p> <p>The tension is structural: <code>PopulatingRepository</code> types the field as the model; <code>CachedBaseRepository</code> stores the raw (depopulated) shape. You cannot have a single typed model serve both at once. The two resolutions:</p> <ol> <li>Compose \u2014 keep <code>profile: Profile | None</code> and cache raw, populating on read. Exact recipe in use case 10: its miss path fetches the raw doc (bypassing the base model-build) and both paths normalize <code>str \u2192 ObjectId</code> before populate, because the JSON round-trip hands you strings.</li> <li>Type it as an id \u2014 <code>profile: PyObjectId | None</code> and no populate rules; then cached hits validate cleanly, but you've given up population entirely.</li> </ol> <p>Do not cache a populated model object through the base class: <code>create</code> caches <code>result.model_dump</code>, so the first cache write stores the embedded-dict shape while later <code>get_by_id</code> misses would repopulate \u2014 inconsistent shapes for the same key, and <code>update</code> overwrites with yet another. Pick one canonical raw shape and stick to it.</p>"},{"location":"03_use_cases/16_cached_repository_intricacies/#example-inspecting-what-gets-stored","title":"\ud83d\ude80 Example \u2014 Inspecting What Gets Stored","text":"<p>Python<pre><code>from mongo_ops import BaseDocument, CachedBaseRepository, ModelRegistry, MongoConnectionManager\nfrom mongo_ops.cache import CacheConfig, InMemoryCacheBackend\n\nclass Product(BaseDocument):\n name: str = \"\"\n price: float = 0.0\n\ncache = InMemoryCacheBackend(max_entries=10_000, default_ttl=600)\nModelRegistry.set_cache_backend(cache)\n\nclass ProductRepo(CachedBaseRepository[Product]):\n def __init__(self):\n super().__init__(\"products\", Product, cache, CacheConfig(enabled=True, backend=\"memory\"))\n\nasync def inspect_cache(repo_id: str) -> None:\n # After create()/get_by_id(), inspect what is actually stored:\n cached = await cache.get(f\"products:{repo_id}\")\n # b'{\"_id\":\"507f...\",\"name\":\"Widget\",\"price\":9.99,\"created_at\":\"2026-...\",\"updated_at\":\"2026-...\"}'\n print(cached)\n\n stats = await cache.get_stats()\n print(stats.hits, stats.misses, stats.sets, stats.deletes) # CacheStats dataclass\n</code></pre> ```</p>"},{"location":"03_use_cases/16_cached_repository_intricacies/#intricacy-3-lifecycle-sharing","title":"\u26a0\ufe0f Intricacy #3 \u2014 Lifecycle & Sharing","text":"<ul> <li>One shared backend instance. The repository needs it (<code>cache_backend=cache</code>) and the registry needs it (<code>ModelRegistry.set_cache_backend(cache)</code>) so <code>initialize_cache()</code> / <code>shutdown_cache()</code> manage the same object. Shutdown cancels the in-memory TTL cleanup task \u2014 forgetting it leaks an <code>asyncio.Task</code> at app exit.</li> <li><code>initialize_cache()</code> starts the backend; <code>initialize_all()</code> creates indexes. Both come after <code>connect()</code>.</li> <li><code>warm_cache([ids])</code> skips keys that already exist, fetches the rest from the DB, and returns how many it wrote \u2014 safe to call repeatedly.</li> <li><code>invalidate_cache(id)</code> deletes one key; <code>clear_pattern(\"products:*\")</code> wipes a collection.</li> </ul>"},{"location":"03_use_cases/16_cached_repository_intricacies/#redis-differences","title":"\ud83d\udd04 Redis Differences","text":"Behaviour In-memory Redis (<code>redis.asyncio</code>) Key prefixing prefix baked into the stored key <code>_full_key()</code> applied on every op TTL heap-based, lazy eviction + cleanup task Native <code>SETEX</code> Invalidation broadcast n/a <code>PUBLISH</code> on <code>mongo_ops:cache:invalidate</code> on delete <code>clear_pattern</code> prefix match on stored keys <code>SCAN MATCH</code> in batches"},{"location":"03_use_cases/16_cached_repository_intricacies/#choosing-the-right-layer","title":"\ud83c\udfaf Choosing the Right Layer","text":"Need Use Scalar docs, no refs \u2014 cache-first reads <code>CachedBaseRepository</code> (UC 07, UC 16) Refs resolved on read, no caching <code>PopulatingRepository</code> (UC 08, UC 15) Refs and cache-first reads composed subclass (UC 10) Bulk warm on startup / cache-then-database failover <code>warm_cache</code> + <code>CacheStats</code>"},{"location":"03_use_cases/16_cached_repository_intricacies/#related","title":"Related","text":"<ul> <li>07 \u2013 Caching \u00b7 10 \u2013 Cache + population \u00b7 11 \u2013 Cache lifecycle \u00b7 15 \u2013 PopulatingRepository internals \u00b7 17 \u2013 Populate & depopulate examples</li> </ul>"},{"location":"03_use_cases/17_populate_depopulate_examples/","title":"17. Populate & Depopulate \u2014 Worked Examples","text":"<p>Scenario: You want to see what populate/depopulate actually do to your documents \u2014 field by field, at every layer (Python model \u2192 depopulated \u2192 stored in Mongo \u2192 raw read back \u2192 populated).</p> <p>These examples all ran against the real <code>mongo_ops</code> engine (<code>mongo_ops/populate/engine.py</code>). Timestamps and <code>ObjectId</code>s are abbreviated with <code>\u2026</code> for readability.</p>"},{"location":"03_use_cases/17_populate_depopulate_examples/#models-rules-used-below","title":"Models & rules used below","text":"Python<pre><code>from bson import ObjectId\nfrom mongo_ops import BaseDocument, PopulatingRepository\nfrom mongo_ops.populate import PopulateRule, PopulationEngine\n\nclass Profile(BaseDocument):\n avatar_url: str = \"\"\n bio: str = \"\"\n\nclass Publisher(BaseDocument):\n name: str = \"\"\n\nclass Book(BaseDocument):\n title: str = \"\"\n publisher: Publisher | None = None # nested ref, reached via nested_rules\n\nclass User(BaseDocument):\n name: str = \"\"\n profile: Profile | None = None # scalar FK \u2014 DB stores an ObjectId\n books: list[Book] | None = None # list FK \u2014 DB stores list[ObjectId]\n\nengine = PopulationEngine({}) # repos keyed by collection name, see UC 15\n\n# A rule WITHOUT nested_rules \u2014 the write path is safe.\nprofile_rule = PopulateRule(field_name=\"profile\", collection_name=\"profiles\")\n\n# A rule WITH nested_rules \u2014 designed for READ-side deep population.\nbooks_rule = PopulateRule(\n field_name=\"books\",\n collection_name=\"books\",\n nested_rules=[PopulateRule(field_name=\"publisher\", collection_name=\"publishers\")],\n)\n</code></pre>"},{"location":"03_use_cases/17_populate_depopulate_examples/#1-write-path-depopulate-turns-models-into-references","title":"1. Write path \u2014 <code>depopulate()</code> turns models into references","text":"<p><code>engine.depopulate(document, rules)</code> walks each rule field and mutates the model in place before you <code>insert_one</code> / <code>find_one_and_update</code> it. What you get on the other side:</p>"},{"location":"03_use_cases/17_populate_depopulate_examples/#1a-scalar-fk-no-nested_rules-stored-as-objectid","title":"1a. Scalar FK, no <code>nested_rules</code> \u2192 stored as <code>ObjectId</code>","text":"Python<pre><code>async def demo():\n profile = Profile(id=ObjectId(\"656\u202604\"), avatar_url=\"a.png\", bio=\"hi\")\n user = User(id=ObjectId(\"656\u202605\"), name=\"Ada\", profile=profile)\n\n await engine.depopulate(user, [profile_rule])\n\n return user.profile # ObjectId('656\u202604') \u2014 was a Profile model\n</code></pre> <p>What Mongo receives (via <code>model_dump</code>, minus <code>_id</code> handling):</p> JSON<pre><code>{ \"name\": \"Ada\", \"profile\": { \"$oid\": \"656\u202604\" } }\n</code></pre>"},{"location":"03_use_cases/17_populate_depopulate_examples/#1b-scalar-fk-with-nested_rules-reference-lost-do-not-do-this","title":"1b. Scalar FK with <code>nested_rules</code> \u2192 reference lost (do not do this)","text":"Python<pre><code>profile_rule_deep = PopulateRule( # same field, but with nested_rules\n field_name=\"profile\",\n collection_name=\"profiles\",\n nested_rules=[PopulateRule(field_name=\"bio\", collection_name=\"profiles\")], # any child field\n)\n\nasync def demo():\n user = User(id=ObjectId(\"656\u202605\"), name=\"Ada\",\n profile=Profile(id=ObjectId(\"656\u202604\"), avatar_url=\"a.png\"))\n await engine.depopulate(user, [profile_rule_deep])\n return user.profile # None \u2014 reference silently dropped!\n</code></pre> <p>Because <code>depopulate</code> recurses into the child but then assigns the parent field <code>None</code> (see <code>engine.py</code> \u2014 <code>depopulated_value</code> is never set on that branch). On save you lose the reference entirely. There is no warning \u2014 the field is just gone.</p>"},{"location":"03_use_cases/17_populate_depopulate_examples/#1c-list-fk-no-nested_rules-stored-as-listobjectid-the-safe-pattern","title":"1c. List FK, no <code>nested_rules</code> \u2192 stored as <code>list[ObjectId]</code> (the safe pattern)","text":"Python<pre><code>async def demo():\n b1 = Book(id=ObjectId(\"656\u202606\"), title=\"MongoDB in Action\")\n b2 = Book(id=ObjectId(\"656\u202607\"), title=\"MongoDB: The Definitive Guide\")\n user = User(id=ObjectId(\"656\u202608\"), name=\"Ada\", books=[b1, b2])\n\n await engine.depopulate(user, [PopulateRule(field_name=\"books\", collection_name=\"books\")])\n\n return user.books # [ObjectId('656\u202606'), ObjectId('656\u202607')] \u2014 clean refs\n</code></pre>"},{"location":"03_use_cases/17_populate_depopulate_examples/#1d-list-fk-with-nested_rules-stored-as-embedded-documents-do-not-do-this","title":"1d. List FK with <code>nested_rules</code> \u2192 stored as embedded documents (do not do this)","text":"Python<pre><code>async def demo():\n publisher = Publisher(id=ObjectId(\"656\u202609\"), name=\"O'Reilly\")\n b1p = Book(id=ObjectId(\"656\u202606\"), title=\"MongoDB in Action\", publisher=publisher)\n user = User(id=ObjectId(\"656\u202608\"), name=\"Ada\", books=[b1p])\n\n await engine.depopulate(user, [books_rule]) # books_rule carries nested_rules\n return user.books[0]\n # <Book id=ObjectId('656\u202606') publisher=ObjectId('656\u202609')>\n</code></pre> <p><code>books</code> stays a list of Book models \u2014 the nested <code>publisher</code> collapsed to <code>ObjectId</code>, but the Book itself was left embedded. Serialized to Mongo you get a nested sub-document:</p> JSON<pre><code>{ \"name\": \"Ada\", \"books\": [ { \"title\": \"MongoDB in Action\", \"publisher\": { \"$oid\": \"656\u202609\" } } ] }\n</code></pre> <p>\u2026and reading that back through population raises <code>ValueError(\u2026contains embedded dict(s) \u2014 run repair script\u2026)</code>, because the engine expects <code>books</code> to hold references, not embedded docs.</p> <p>Why UC 09's <code>Author \u2192 books \u2192 publisher</code> graph can't be written as-is: a rule with <code>nested_rules</code> breaks the write path (1b logs silently, 1d stores embedded). The safe round-trip: save referenced documents first, store <code>list[ObjectId]</code>/<code>ObjectId</code> with a non-nested rule for reads, and keep <code>nested_rules</code> only on rules you never hand back to <code>depopulate</code>.</p>"},{"location":"03_use_cases/17_populate_depopulate_examples/#2-read-path-populate-turns-references-back-into-models","title":"2. Read path \u2014 <code>populate()</code> turns references back into models","text":"<p><code>PopulatingRepository.data_to_model(raw_dict)</code> resolves references before building the model. It first runs <code>_populate(dict)</code> \u2014 replacing the raw <code>ObjectId</code> references with full models via the registered repos \u2014 and only then constructs <code>self.model(**data)</code>, so every FK field is a real model by construction. The two <code>data_to_model</code> examples below (2a/2b) show the underlying resolution <code>_populate</code> performs per rule. (<code>repo</code> below is a <code>PopulatingRepository</code> wired per UC 15 \u2014 <code>books</code> and <code>publishers</code> repos registered in its <code>PopulationEngine</code>.)</p>"},{"location":"03_use_cases/17_populate_depopulate_examples/#2a-scalar-fk-objectid-profile-model","title":"2a. Scalar FK \u2014 <code>ObjectId</code> \u2192 <code>Profile</code> model","text":"Python<pre><code># what came back from Mongo (raw dict, profile is an ObjectId reference):\nraw = {\"_id\": ObjectId(\"656\u202605\"), \"name\": \"Ada\", \"profile\": ObjectId(\"656\u202604\")}\n\nasync def demo():\n user = await repo.data_to_model(raw) # _populate() \u2192 model(**data)\n return user.profile # Profile(id=ObjectId('656\u202604'), avatar_url='a.png')\n</code></pre>"},{"location":"03_use_cases/17_populate_depopulate_examples/#2b-list-fk-nested_rules-two-levels-deep","title":"2b. List FK + <code>nested_rules</code> \u2014 two levels deep","text":"Python<pre><code>raw = {\"_id\": ObjectId(\"656\u202608\"), \"name\": \"Ada\",\n \"books\": [ObjectId(\"656\u202606\"), ObjectId(\"656\u202607\")]}\n\nasync def demo():\n user = await repo.data_to_model(raw) # _populate() resolves both levels\n return (type(user.books[0]).__name__, # 'Book'\n type(user.books[0].publisher).__name__, # 'Publisher' \u2014 nested_rules reached it\n user.books[0].publisher.name) # \"O'Reilly\"\n</code></pre> <p>Walking through one book's resolution:</p> Text Only<pre><code>books_rule \u2192 field \"books\", collection \"books\"\n nested_rules[0].publisher rule \u2192 field \"publisher\", collection \"publishers\"\n1. for ObjectId('656\u202606') \u2192 repo(\"books\").get_by_id(...) \u2192 Book(\"MongoDB in Action\")\n2. book has nested rule \u2192 repo(\"publishers\").get_by_id(...) \u2192 Publisher(\"O'Reilly\")\n3. author.books[0] \u2192 Book(publisher=Publisher(\"O'Reilly\"))\n</code></pre> <p>Per-level cost: 1 query per reference (no <code>$lookup</code>). A hub with 20 hooks and a <code>publisher</code> nested rule is 40 queries \u2014 that is why the next section exists.</p>"},{"location":"03_use_cases/17_populate_depopulate_examples/#3-cache-round-trip-what-actually-goes-into-the-cache","title":"3. Cache round trip \u2014 what actually goes into the cache","text":"<p><code>CachedBaseRepository</code> stores <code>model_dump(by_alias=True)</code> JSON-encoded with <code>default=str</code> (the <code>encode_value</code> helper in <code>mongo_ops/cache/in_memory.py</code>). Crucially it snapshots the raw, just-out-of-Mongo shape \u2014 FK fields still hold <code>ObjectId</code>s \u2014 so the cache holds references, exactly like the DB row:</p> Layer <code>profile</code> field looks like Mongo doc (raw) <code>\u201cprofile\u201d: ObjectId(\"656\u202604\")</code> <code>model_dump(by_alias=True)</code> <code>\"profile\": ObjectId(\"656\u202604\")</code> <code>encode_value(...)</code> <code>\"...\\\"profile\\\": \\\"656\u202604\\\"...\"</code> \u2014 hex string <code>decode_value(...)</code> <code>\"profile\": \"656\u202604\"</code> \u2014 still a hex string <p>Verbatim (timestamps abbreviated):</p> Python<pre><code>from mongo_ops.cache.in_memory import decode_value, encode_value\n\nsnapshot = {\"id\": \"656\u202605\", \"profile\": ObjectId(\"656\u202604\")} # what a cached raw doc looks like\n\ncached = encode_value(snapshot) # json.dumps(snapshot, default=str).encode()\ndecoded = decode_value(cached)\n\ndecoded[\"profile\"] # '656\u202604' \u2014 hex STRING, not ObjectId, not Profile\n</code></pre> <p>Typed-FK caveat: the round trip above only works when the FK field's model type accepts the raw value (<code>str</code>/<code>ObjectId</code> typed). A field typed <code>Profile | None</code> cannot even be materialized from the raw Mongo doc \u2014 both hit and miss raise <code>ValidationError</code> (UC 16's intricacy). That is exactly why UC 10's composed repo overrides <code>get_by_id</code>/<code>create</code>/<code>update</code> to carry raw data through the cache and resolve refs with <code>_populate</code>.</p> <p>Two consequences (the \"intricacies\", detailed in UC 16):</p> <ol> <li>On a cache hit, <code>decode_value</code> gives you hex-string references. A field typed <code>Profile | None</code> receiving a hex string raises Pydantic <code>ValidationError</code> \u2014 so a plain <code>CachedBaseRepository</code> cannot serve model-typed FK fields, hit or miss. The composed pattern in UC 10 re-resolves them in its <code>get_by_id</code>: <code>data = decode_value(await cache.get(key))</code>, then <code>populated = await self._populate(data)</code>, then <code>self.model(**populated)</code>.</li> <li>On a miss, <code>BaseRepository.get_by_id</code> reads the raw Mongo doc whose <code>profile</code> is an <code>ObjectId</code> and immediately raises the same <code>ValidationError</code> \u2014 which is why UC 10 overrides <code>get_by_id</code> to fetch raw, cache raw, then populate. (Real ObjectIds are covered by UC 16's table.)</li> </ol>"},{"location":"03_use_cases/17_populate_depopulate_examples/#4-when-to-use-which","title":"4. When to use which","text":"You need\u2026 Use Scalar <code>ObjectId</code> store, model-struct reads <code>PopulatingRepository</code> (UC 08) Deep nested reads (2+ levels) <code>PopulatingRepository</code> + <code>nested_rules</code> (UC 09) \u2014 read side only Same shape, but cache-first reads UC 10 composed repo (engine + cache + <code>_populate</code>) Scalar docs, no refs, hot <code>get_by_id</code> <code>CachedBaseRepository</code> (UC 07, UC 16) Writing a reference graph back depopulate without <code>nested_rules</code> \u2014 save children first, then point at their ids"},{"location":"03_use_cases/17_populate_depopulate_examples/#related","title":"Related","text":"<ul> <li>08 \u2013 Population \u00b7 09 \u2013 Advanced population \u00b7 10 \u2013 Cache + Population \u00b7 15 \u2013 PopulatingRepository wiring \u00b7 16 \u2013 CachedRepository intricacies</li> </ul>"},{"location":"03_use_cases/18_aggregation_pipelines/","title":"Use Case 18: Aggregation Pipelines","text":"<p>Scenario: A reporting dashboard needs grouped statistics, computed fields, and joined data \u2014 operations that don't map to single-document CRUD.</p>"},{"location":"03_use_cases/18_aggregation_pipelines/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>collection.aggregate()</code> Raw Motor aggregation \u2014 no base-repo helper exists for pipelines. Repository wrapper Keeps pipeline logic behind a domain method so callers never touch the collection directly. <code>$group</code>, <code>$lookup</code>, <code>$project</code> Common pipeline stages demonstrated in a single repository."},{"location":"03_use_cases/18_aggregation_pipelines/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from fastapi import FastAPI\nfrom mongo_ops import BaseDocument, BaseRepository\n\n\n# ---------------------------\n# Models\n# ---------------------------\nclass Order(BaseDocument):\n user_id: str\n status: str = \"pending\"\n total: float = 0.0\n\n\nclass OrderRepository(BaseRepository[Order]):\n def __init__(self):\n super().__init__(\"orders\", Order)\n\n async def revenue_by_status(self) -> list[dict]:\n \"\"\"Group orders by status, sum totals.\"\"\"\n pipeline = [\n {\"$group\": {\"_id\": \"$status\", \"revenue\": {\"$sum\": \"$total\"}, \"count\": {\"$sum\": 1}}},\n {\"$sort\": {\"revenue\": -1}},\n ]\n return await self.collection.aggregate(pipeline).to_list(length=100)\n\n async def top_spenders(self, limit: int = 10) -> list[dict]:\n \"\"\"Top N users by total spend, with a lookup to resolve user names.\"\"\"\n pipeline = [\n {\"$group\": {\"_id\": \"$user_id\", \"total_spent\": {\"$sum\": \"$total\"}, \"order_count\": {\"$sum\": 1}}},\n {\"$sort\": {\"total_spent\": -1}},\n {\"$limit\": limit},\n {\"$lookup\": {\n \"from\": \"users\",\n \"localField\": \"_id\",\n \"foreignField\": \"_id\",\n \"as\": \"user\",\n }},\n {\"$unwind\": {\"path\": \"$user\", \"preserveNullAndEmptyArrays\": True}},\n {\"$project\": {\n \"total_spent\": 1,\n \"order_count\": 1,\n \"user_name\": {\"$ifNull\": [\"$user.username\", \"unknown\"]},\n }},\n ]\n return await self.collection.aggregate(pipeline).to_list(length=limit)\n\n async def daily_summary(self, days: int = 30) -> list[dict]:\n \"\"\"Order counts grouped by day for the last N days.\"\"\"\n from datetime import datetime, timedelta\n\n cutoff = datetime.utcnow() - timedelta(days=days)\n pipeline = [\n {\"$match\": {\"created_at\": {\"$gte\": cutoff}}},\n {\"$group\": {\n \"_id\": {\"$dateToString\": {\"format\": \"%Y-%m-%d\", \"date\": \"$created_at\"}},\n \"orders\": {\"$sum\": 1},\n \"revenue\": {\"$sum\": \"$total\"},\n }},\n {\"$sort\": {\"_id\": 1}},\n ]\n return await self.collection.aggregate(pipeline).to_list(length=days)\n\n\napp = FastAPI()\norder_repo = OrderRepository()\n\n\n@app.get(\"/reports/revenue-by-status\")\nasync def revenue_by_status():\n return await order_repo.revenue_by_status()\n\n\n@app.get(\"/reports/top-spenders\")\nasync def top_spenders(limit: int = 10):\n return await order_repo.top_spenders(limit)\n\n\n@app.get(\"/reports/daily-summary\")\nasync def daily_summary(days: int = 30):\n return await order_repo.daily_summary(days)\n</code></pre> <p>Note: lifespan wiring omitted \u2014 copy from use case 01.</p>"},{"location":"03_use_cases/18_aggregation_pipelines/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Aggregation methods bypass the caching and population layers \u2014 they read raw documents. Keep them deliberate and document the bypass in the method docstring.</li> <li>Use <code>$lookup</code> sparingly; for read-heavy workloads, prefer application-level population via use case 08 which benefits from the cache.</li> <li>Pipeline results are plain dicts, not model instances. Wrap them in a Pydantic model if you need validation or serialization in responses.</li> <li>Create indexes on fields used in <code>$match</code> and <code>$sort</code> stages \u2014 otherwise the aggregation runs a collection scan (see use case 13).</li> </ul>"},{"location":"03_use_cases/18_aggregation_pipelines/#related","title":"Related","text":"<ul> <li>02 \u2013 Custom repository \u00b7 13 \u2013 Index creation \u00b7 19 \u2013 Bulk operations \u00b7 12 \u2013 Transaction helper</li> </ul>"},{"location":"03_use_cases/19_bulk_operations/","title":"Use Case 19: Bulk Operations","text":"<p>Scenario: A background job needs to update or delete hundreds of documents in one call \u2014 doing it one-by-one would be prohibitively slow.</p>"},{"location":"03_use_cases/19_bulk_operations/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>update_many</code> / <code>delete_many</code> Motor's multi-document write operations. Repository wrapper Keeps bulk logic behind a domain method so callers pass IDs and business parameters, not raw dicts. <code>BulkWriteError</code> Raised when <code>ordered=False</code> and some operations fail \u2014 catch it for partial-success handling."},{"location":"03_use_cases/19_bulk_operations/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from datetime import datetime\n\nfrom bson import ObjectId, BulkWriteError\nfrom fastapi import FastAPI, HTTPException\nfrom mongo_ops import BaseDocument, BaseRepository\n\n\n# ---------------------------\n# Model\n# ---------------------------\nclass Task(BaseDocument):\n status: str = \"pending\"\n assignee_id: str | None = None\n priority: int = 0\n\n\n# ---------------------------\n# Repository\n# ---------------------------\nclass TaskRepository(BaseRepository[Task]):\n def __init__(self):\n super().__init__(\"tasks\", Task)\n\n async def bulk_update_status(self, ids: list[str], status: str) -> int:\n \"\"\"Set status on many tasks at once. Returns the count of modified documents.\"\"\"\n object_ids = [ObjectId(i) for i in ids]\n result = await self.collection.update_many(\n {\"_id\": {\"$in\": object_ids}},\n {\"$set\": {\"status\": status, \"updated_at\": datetime.utcnow()}},\n )\n return result.modified_count\n\n async def bulk_reassign(self, old_assignee: str, new_assignee: str) -> int:\n \"\"\"Move all pending tasks from one assignee to another.\"\"\"\n result = await self.collection.update_many(\n {\"assignee_id\": old_assignee, \"status\": \"pending\"},\n {\"$set\": {\"assignee_id\": new_assignee, \"updated_at\": datetime.utcnow()}},\n )\n return result.modified_count\n\n async def bulk_archive(self, ids: list[str]) -> int:\n \"\"\"Delete many tasks at once. Returns the count of deleted documents.\"\"\"\n object_ids = [ObjectId(i) for i in ids]\n result = await self.collection.delete_many({\"_id\": {\"$in\": object_ids}})\n return result.deleted_count\n\n async def bulk_upsert_users(self, records: list[dict]) -> dict:\n \"\"\"Insert-or-update many user records. Returns inserted/modified counts.\"\"\"\n from pymongo import UpdateOne\n\n ops = [\n UpdateOne(\n {\"email\": rec[\"email\"]},\n {\"$set\": rec},\n upsert=True,\n )\n for rec in records\n ]\n try:\n result = await self.collection.bulk_write(ops, ordered=False)\n return {\n \"inserted\": result.upserted_count,\n \"modified\": result.modified_count,\n }\n except BulkWriteError as exc:\n return {\n \"inserted\": exc.details.get(\"upsertedCount\", 0),\n \"modified\": exc.details.get(\"modifiedCount\", 0),\n \"errors\": exc.details.get(\"writeErrors\", []),\n }\n\n\napp = FastAPI()\ntask_repo = TaskRepository()\n\n\n@app.post(\"/tasks/bulk-status\")\nasync def bulk_status(ids: list[str], status: str):\n count = await task_repo.bulk_update_status(ids, status)\n return {\"modified\": count}\n\n\n@app.post(\"/tasks/bulk-reassign\")\nasync def reassign(old_assignee: str, new_assignee: str):\n count = await task_repo.bulk_reassign(old_assignee, new_assignee)\n return {\"modified\": count}\n\n\n@app.post(\"/tasks/bulk-archive\")\nasync def archive(ids: list[str]):\n count = await task_repo.bulk_archive(ids)\n return {\"deleted\": count}\n</code></pre> <p>Note: lifespan wiring omitted \u2014 copy from use case 01.</p>"},{"location":"03_use_cases/19_bulk_operations/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Bulk methods bypass the caching and population layers. If you need cache consistency after a bulk write, call <code>invalidate_cache(id)</code> for each affected document, or reinitialize the cache (see use case 11).</li> <li>Use <code>ordered=False</code> for upserts so one bad record doesn't block the rest. Catch <code>BulkWriteError</code> and surface the per-operation errors.</li> <li>For very large batches (10k+), chunk the operation into pages of 1 000 to avoid exceeding MongoDB's 16 MB <code>bulkWrite</code> payload limit.</li> <li><code>update_many</code> / <code>delete_many</code> require a <code>{\"_id\": {\"$in\": [...]}}</code> filter to scope the operation \u2014 never pass an unscoped filter.</li> </ul>"},{"location":"03_use_cases/19_bulk_operations/#related","title":"Related","text":"<ul> <li>02 \u2013 Custom repository \u00b7 18 \u2013 Aggregation pipelines \u00b7 03 \u2013 Transactions \u00b7 11 \u2013 Cache lifecycle</li> </ul>"}]} |