1 line
140 KiB
JSON
1 line
140 KiB
JSON
{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"\ud83e\udde9 mongo-ops \u2014 Async MongoDB Operations Layer for FastAPI","text":"<p><code>mongo-ops</code> is a modular, high-performance MongoDB operations library for FastAPI microservices. It standardizes connection lifecycle, Pydantic v2 document models, and repository-style async CRUD \u2014 with added layers for caching, reference population, and multi-document transactions. It is built on top of Motor.</p> <p>Doc model: this wiki is written for humans \u2014 how\u2011to guides, examples, and testing recipes. The authoritative API contracts live in the code (docstrings) and the machine\u2011readable bundle under <code>docs/mcp/</code>.</p>"},{"location":"#key-features","title":"\ud83d\ude80 Key Features","text":"<ul> <li>\ud83e\uddf1 Unified repository pattern \u2014 <code>BaseRepository</code> with generic CRUD</li> <li>\u26a1 Fully asynchronous (Motor-based)</li> <li>\ud83e\uddec Pydantic v2 data models with auto\u2011timestamps (<code>BaseDocument</code>)</li> <li>\ud83d\udd12 Model registry for multi-collection startup (<code>ModelRegistry</code>)</li> <li>\ud83d\uddc4\ufe0f ID-based caching \u2014 in-memory or Redis (<code>cache/</code>)</li> <li>\ud83d\udd17 Reference population with cycle detection (<code>populate/</code>)</li> <li>\ud83d\udd01 Transaction and session helpers (<code>transactions/</code>)</li> <li>\ud83e\uddea Mock-friendly architecture \u2014 the whole repo test suite runs without MongoDB</li> </ul>"},{"location":"#installation","title":"\ud83d\udce6 Installation","text":"<p>From your internal PyPI:</p> Bash<pre><code>pip install --extra-index-url https://$PYPI_USERNAME:$PYPI_PASSWORD@pip.aetoskia.com/simple mongo-ops\n</code></pre> <p>With the extra batteries your workload needs:</p> Bash<pre><code>pip install --extra-index-url https://$PYPI_USERNAME:$PYPI_PASSWORD@pip.aetoskia.com/simple \"mongo-ops[redis]\" # Redis cache backend\npip install --extra-index-url https://$PYPI_USERNAME:$PYPI_PASSWORD@pip.aetoskia.com/simple \"mongo-ops[all]\" # dev, redis, beanie, fastapi\n</code></pre> <p>From local source:</p> Bash<pre><code>pip install -e .\n</code></pre>"},{"location":"#documentation-structure","title":"\ud83d\udcc1 Documentation Structure","text":"Section Description Overview Architecture, lifecycle, and repository decision table Core Components Validated reference for the public API surface Use cases Step-by-step recipes \u00b7 01 \u2013 Basic CRUD API FastAPI user management app \u00b7 02 \u2013 Custom repository Product catalog with business logic \u00b7 03 \u2013 Transactions Atomic order + inventory updates \u00b7 04 \u2013 Pagination Filtering, sorting, page metadata \u00b7 05 \u2013 Soft deletes Recoverable delete pattern \u00b7 06 \u2013 Multi-model One service, many collections \u00b7 07 \u2013 Caching In-memory / Redis read cache \u00b7 08 \u2013 Population Resolve references on read \u00b7 09 \u2013 Advanced population Nested + circular references \u00b7 10 \u2013 Cache + population Reading through cache, populating on hit \u00b7 11 \u2013 Cache lifecycle Startup/shutdown hygiene \u00b7 12 \u2013 Transaction helper <code>execute_transaction</code> \u00b7 13 \u2013 Indexes Single, compound, and optioned indexes \u00b7 14 \u2013 Testing guide Mock Motor, engine, cache, transactions \u00b7 15 \u2013 PopulatingRepository internals Object \u21c4 ObjectId write/read lifecycle, wiring \u00b7 16 \u2013 Cached repository intricacies What's stored, JSON round-trip, typed-FK trap \u00b7 17 \u2013 Populate & depopulate examples Worked shapes for depopulate/populate + cached round trip Best Practices Layering, lifecycle, and gotchas Common Patterns Service layer, aggregation, bulk ops Error Handling Library exceptions and FastAPI mapping Testing Example Mock-based quickstart"},{"location":"#related-resources","title":"\ud83d\udd17 Related Resources","text":"<ul> <li>Source Code: Gitea Repository</li> <li>Internal PyPI: pip.aetoskia.com/simple/mongo-ops</li> <li>Drone CI: Auto-builds and publishes tagged releases, gated on black / ruff / mypy / pytest.</li> </ul> <p>\u00a9 Aetoskia Internal \u2014 <code>mongo-ops</code> 0.1.5</p>"},{"location":"01_overview/","title":"Library Overview","text":"<p><code>mongo-ops</code> gives FastAPI/microservice teams a small, opinionated toolkit for talking to MongoDB asynchronously. This page builds the mental model: what the layers are, in what order they must be wired up, and which repository to reach for.</p>"},{"location":"01_overview/#architecture","title":"\ud83c\udfd7\ufe0f Architecture","text":"Text Only<pre><code>\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 Application (FastAPI) \u2502\n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n\u2502 \u2502 CachedBase \u2502 Populating \u2502 BaseRepository\u2502 Raw \u2502 \u2502\n\u2502 \u2502 Repository \u2502 Repository \u2502 (CRUDMixin) \u2502 Motor \u2502 \u2502\n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n\u2502 \u2502 \u2502 \u2502 \u2502\n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n\u2502 \u2502 Cache backend\u2502 \u2502 Population\u2502 \u2502 TransactionManager \u2502 \u2502\n\u2502 \u2502 memory/redis \u2502 \u2502 Engine \u2502 \u2502 (session helpers) \u2502 \u2502\n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n\u2502 \u2502 ModelRegistry (models, indexes, cache lifecycle) \u2502 \u2502\n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n\u2502 \u2502 MongoConnectionManager (Motor client + database) \u2502 \u2502\n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u25bc\n MongoDB (Motor / PyMongo)\n</code></pre> <p>The layers build on each other:</p> <ol> <li>Connection \u2014 <code>MongoConnectionManager</code> owns the Motor client and active database.</li> <li>Models \u2014 <code>BaseDocument</code> (Pydantic v2) gives every document <code>id</code> (aliased <code>_id</code>) plus <code>created_at</code> / <code>updated_at</code>.</li> <li>CRUD \u2014 <code>CRUDMixin</code> / <code>BaseRepository</code> implement generic async CRUD against a collection.</li> <li>Startup \u2014 <code>ModelRegistry</code> centralizes model + index registration and cache lifecycle for multi-collection services.</li> <li>Advanced layers \u2014 caching and population compose around a repository:</li> <li><code>CachedBaseRepository</code> wraps <code>BaseRepository</code> with a <code>CacheBackend</code>.</li> <li><code>PopulatingRepository</code> wraps <code>BaseRepository</code> with a <code>PopulationEngine</code>.</li> <li>Transactions \u2014 <code>TransactionManager</code> runs a list of async operations atomically on a session.</li> </ol>"},{"location":"01_overview/#lifecycle-rules","title":"\ud83d\udd04 Lifecycle Rules","text":"<p><code>mongo-ops</code> has a strict startup order. Violating it raises fast, loud exceptions:</p> Step Call Why 1 <code>await MongoConnectionManager.connect(uri, db_name)</code> Without a connection, <code>get_database()</code>/<code>get_client()</code> raise <code>RuntimeError(\"Database not connected...\")</code>. 2 <code>await ModelRegistry.initialize_all()</code> Create registered indexes (idempotent via <code>create_index</code>). 3 <code>await ModelRegistry.set_cache_backend(...)</code> then <code>await ModelRegistry.initialize_cache()</code> Must happen after connection, before any cache-backed operation. Raises <code>RuntimeError(\"No cache backend registered...\")</code> if skipped. 4 Use repositories Constructed repos resolve the collection from the live database. 5 <code>await ModelRegistry.shutdown_cache()</code> + <code>await MongoConnectionManager.disconnect()</code> On shutdown (in-memory TTL task cancelled; Redis pub/sub closed). <p>The canonical wiring is the FastAPI lifespan context manager (see use case 01).</p>"},{"location":"01_overview/#which-repository-should-you-use","title":"\ud83e\udded Which Repository Should You Use?","text":"Repository Use when Adds <code>BaseRepository[T]</code> Plain CRUD \u2014 the default nothing extra <code>CachedBaseRepository[T]</code> Read\u2011heavy, low\u2011write fields (lookups by <code>_id</code>) cache\u2011first <code>get_by_id</code>, cache on <code>create</code>, invalidate on <code>update</code>/<code>delete</code>, <code>warm_cache</code> <code>PopulatingRepository[T]</code> You return related docs (FK references) denormalized <code>_populate</code> on read, <code>_depopulate</code> on write, FK\u2011guarded <code>patch</code> <code>CRUDMixin</code> Reusing CRUD inside an existing class raw CRUD against a collection you already have <p>There is no built\u2011in <code>CachedPopulatingRepository</code>. Use case 10 shows how to compose caching + population in a small subclass.</p>"},{"location":"01_overview/#read-next","title":"\u27a1\ufe0f Read Next","text":"<ul> <li>Core Components \u2014 the validated public API surface.</li> <li>Use case 01 \u2014 the minimal FastAPI app.</li> <li>Best Practices \u2014 team-wide conventions.</li> </ul>"},{"location":"02_components/","title":"Core Components","text":"<p>A validated reference to the public API. Signatures match the code in <code>mongo_ops/</code> exactly. See the use cases for runnable recipes.</p>"},{"location":"02_components/#1-mongoconnectionmanager","title":"1. MongoConnectionManager","text":"<p>Singleton-style owner of the Motor client and active database.</p> Method Signature Behavior <code>connect</code> <code>async (uri: str, db_name: str, **kwargs) -> AsyncIOMotorDatabase</code> Creates <code>AsyncIOMotorClient(uri, **kwargs)</code> and selects the database. Idempotent. <code>disconnect</code> <code>async () -> None</code> Closes the client and clears state. <code>get_database</code> <code>() -> AsyncIOMotorDatabase</code> Returns the active database; raises <code>RuntimeError(\"Database not connected. Call connect() first.\")</code>. <code>get_client</code> <code>() -> AsyncIOMotorClient</code> Returns the active client; raises <code>RuntimeError(\"Client not connected. Call connect() first.\")</code>. <code>lifespan</code> <code>async ctx manager (uri, db_name, **kwargs)</code> <code>connect()</code> on entry, yields the database, <code>disconnect()</code> on exit. Designed for FastAPI lifespan."},{"location":"02_components/#2-basedocument-pyobjectid","title":"2. BaseDocument & PyObjectId","text":"<p><code>BaseDocument(BaseModel)</code> \u2014 inherit for every Mongo entity.</p> <ul> <li><code>id: Optional[PyObjectId]</code> \u2014 aliased to <code>_id</code>, serialized to <code>str</code>.</li> <li><code>created_at: datetime</code> \u2014 defaults to <code>datetime.utcnow()</code>.</li> <li><code>updated_at: datetime</code> \u2014 defaults to <code>datetime.utcnow()</code>.</li> <li><code>Config.populate_by_name = True</code>, <code>arbitrary_types_allowed = True</code>, <code>json_encoders = {ObjectId: str}</code>.</li> </ul> <p><code>PyObjectId(ObjectId)</code> \u2014 Pydantic v2-compatible <code>ObjectId</code> that accepts <code>str</code> or <code>ObjectId</code> and validates with <code>ObjectId.is_valid</code>.</p>"},{"location":"02_components/#3-crudmixint","title":"3. CRUDMixin[T]","text":"<p>Generic CRUD over a Motor collection. The building block of all repositories.</p> Method Signature Notes <code>create</code> <code>async (data: T) -> T</code> Dumps model (excludes <code>id</code>, <code>None</code>), stamps <code>created_at</code>/<code>updated_at</code>, inserts, returns model with assigned <code>_id</code>. <code>get_by_id</code> <code>async (id: str \\| ObjectId) -> Optional[T]</code> <code>str</code> is accepted and converted to <code>ObjectId</code>. <code>get_many</code> <code>async (filter: dict \\| None = None, skip: int = 0, limit: int = 100, sort: list[tuple] \\| None = None) -> list[T]</code> Cursor <code>.skip().limit().sort(...)</code> then <code>to_list(limit)</code>. <code>limit=0</code> disables the limit clause. <code>update</code> <code>async (id, data: dict[str, Any]) -> Optional[T]</code> <code>$set</code> + refreshed <code>updated_at</code> via <code>find_one_and_update</code>. <code>patch</code> <code>async (id, data: dict[str, Any]) -> Optional[T]</code> Same as <code>update</code> but intended for REST PATCH semantics. <code>delete</code> <code>async (id) -> bool</code> True if a document was deleted. <code>count</code> <code>async (filter: dict \\| None = None) -> int</code> <code>count_documents</code>."},{"location":"02_components/#4-baserepositoryt","title":"4. BaseRepository[T]","text":"<p><code>BaseRepository(collection_name: str, model: type[T])</code> \u2014 resolves the collection from <code>MongoConnectionManager.get_database()[collection_name]</code>. Requires an active connection at construction time. Provides everything in <code>CRUDMixin</code> plus <code>collection_name</code>.</p>"},{"location":"02_components/#5-populatingrepositoryt","title":"5. PopulatingRepository[T]","text":"<p><code>PopulatingRepository(collection_name, model, population_engine: PopulationEngine | None = None, populate_rules: list[PopulateRule] | None = None)</code>.</p> <ul> <li><code>set_population_engine(engine)</code> / <code>set_populate_rules(rules)</code> \u2014 swap engine/rules at runtime.</li> <li><code>data_to_model</code> hooks <code>_populate</code> \u2014 <code>get_by_id</code>/<code>get_many</code> return fully populated models.</li> <li><code>_depopulate(document)</code> \u2014 collapses populated FK fields back to <code>ObjectId</code> before <code>create</code>/<code>update</code>.</li> <li><code>create</code> / <code>update</code> accept a model <code>T</code> (not a dict) so depopulation can run.</li> <li><code>patch</code> blocks FK fields \u2014 raises <code>ValueError(\"Cannot patch FK fields via patch(): ... Use update() to change FK fields.\")</code>.</li> </ul> <p>Populate semantics (important): a <code>PopulateRule</code> names a field that holds either an <code>ObjectId</code> or a <code>list[ObjectId]</code> and is the same field that gets replaced with the resolved document(s). There is no separate \"ref field\" vs \"target field\". See use case 08.</p>"},{"location":"02_components/#6-transactionmanager","title":"6. TransactionManager","text":"Method Signature Behavior <code>start_session</code> <code>async ctx manager (**kwargs) -> AsyncIOMotorClientSession</code> Yields a session with an active transaction. Pass <code>session=</code> to every collection call inside. <code>execute_transaction</code> <code>async (operations: list[Callable[[session], Awaitable[Any]]], **kwargs) -> list[Any]</code> Runs each op inside one transaction and returns results in order; any exception aborts the transaction and propagates."},{"location":"02_components/#7-modelregistry","title":"7. ModelRegistry","text":"<p>Centralized models, indexes, and cache lifecycle for multi-collection services.</p> Method Signature Behavior <code>register</code> <code>(collection_name: str, model: type[BaseDocument], indexes: list[Any] \\| None = None)</code> Records model + index specs. Index specs are passed as-is to pymongo <code>create_index</code> \u2014 single tuples, compound lists, or dicts with <code>keys</code>/<code>options</code>. <code>initialize_all</code> <code>async (db: AsyncIOMotorDatabase \\| None = None) -> None</code> <code>create_index</code> per registered spec (idempotent). Uses the manager database if <code>db</code> omitted. <code>get_model</code> <code>(collection_name) -> type[BaseDocument]</code> Raises <code>KeyError</code> if unregistered. <code>list_collections</code> <code>() -> list[str]</code> Registered collection names. <code>set_cache_backend</code> <code>(backend: CacheBackend) -> None</code> Register the single shared backend. <code>initialize_cache</code> <code>async () -> None</code> Starts the backend (background TTL cleanup) \u2014 raises <code>RuntimeError</code> if no backend registered. <code>shutdown_cache</code> <code>async () -> None</code> Stops the backend cleanly and clears it. <code>get_cache_backend</code> <code>() -> Optional[CacheBackend]</code> Current backend, if any."},{"location":"02_components/#8-cache-layer","title":"8. Cache Layer","text":""},{"location":"02_components/#81-cachebackend-abstract","title":"8.1 CacheBackend (abstract)","text":"<p>Async interface: <code>get(key) -> Optional[bytes]</code>, <code>set(key, value: bytes, ttl | None)</code>, <code>delete(key)</code>, <code>exists(key)</code>, <code>clear_pattern(pattern)</code>, <code>get_stats() -> CacheStats</code>, <code>initialize()</code>, <code>shutdown()</code>. Values are bytes (JSON-encoded).</p>"},{"location":"02_components/#82-cachestats","title":"8.2 CacheStats","text":"<p>Dataclass: <code>hits</code>, <code>misses</code>, <code>sets</code>, <code>deletes</code>, <code>current_size</code>, <code>max_size</code>.</p>"},{"location":"02_components/#83-cacheconfig","title":"8.3 CacheConfig","text":"<p>Dataclass: <code>enabled: bool = True</code>, <code>backend: Literal[\"memory\", \"redis\"] = \"memory\"</code>, <code>redis_client</code>, <code>default_ttl: int = 300</code>, <code>max_entries: int = 10000</code>, <code>key_prefix: str = \"\"</code>, <code>cleanup_interval: int = 60</code>. Raises <code>ValueError</code> if <code>backend=\"redis\"</code> without a client, and <code>ImportError</code> if <code>redis</code> is not installed.</p>"},{"location":"02_components/#84-inmemorycachebackend","title":"8.4 InMemoryCacheBackend","text":"<p><code>InMemoryCacheBackend(max_entries=10000, default_ttl=300, cleanup_interval=60)</code> \u2014 LRU <code>OrderedDict</code> + TTL heap; <code>initialize()</code> spawns the periodic cleanup task (TTL 0 expires immediately).</p>"},{"location":"02_components/#85-rediscachebackend","title":"8.5 RedisCacheBackend","text":"<p><code>RedisCacheBackend(redis_client, key_prefix=\"\", default_ttl=300)</code> \u2014 <code>setex</code> storage, SCAN-based <code>clear_pattern</code>, and <code>publish_invalidate(key)</code> for cross-service invalidation on <code>delete</code> via the <code>mongo_ops:cache:invalidate</code> channel.</p>"},{"location":"02_components/#86-cachedbaserepositoryt","title":"8.6 CachedBaseRepository[T]","text":"<p><code>CachedBaseRepository(collection_name, model, cache_backend: CacheBackend, config: CacheConfig | None = None)</code>.</p> <ul> <li>Cache keys are <code>\"{key_prefix}{id}\"</code> (prefix defaults to <code>\"{collection_name}:\"</code>).</li> <li><code>get_by_id</code> \u2014 cache-first; cache miss reads DB and stores <code>model_dump(by_alias=True)</code> (JSON-encoded) for <code>default_ttl</code>. Honors <code>config.enabled=False</code> (bypass).</li> <li><code>create</code> \u2014 inserts then caches the result.</li> <li><code>update</code>/<code>delete</code> \u2014 refresh or remove the cache entry.</li> <li><code>warm_cache(ids) -> int</code> \u2014 prefetch a list of IDs, returns count warmed.</li> <li><code>invalidate_cache(id)</code> \u2014 manual eviction.</li> </ul>"},{"location":"02_components/#9-population-layer","title":"9. Population Layer","text":""},{"location":"02_components/#91-populaterule","title":"9.1 PopulateRule","text":"<p>Dataclass:</p> Python<pre><code>@dataclass\nclass PopulateRule:\n field_name: str # field holding the ObjectId / list[ObjectId]; replaced in-place with the resolved doc(s)\n collection_name: str # collection the references point at\n nested_rules: list[PopulateRule] | None = None\n max_depth: int = 1\n filter: dict | None = None # DECLARED but NOT yet applied by the engine\n projection: dict | None = None # DECLARED but NOT yet applied by the engine\n</code></pre> <p>\u26a0\ufe0f <code>filter</code> and <code>projection</code> are accepted but currently ignored by <code>PopulationEngine</code> \u2014 do not rely on them yet.</p>"},{"location":"02_components/#92-populationengine","title":"9.2 PopulationEngine","text":"<p><code>PopulationEngine(repos: dict[str, Any], global_max_depth: int = 10)</code> \u2014 <code>repos</code> maps <code>collection_name</code> \u2192 repository.</p> <ul> <li><code>register_repo(collection_name, repo)</code> \u2014 add repositories at runtime.</li> <li><code>populate(document, rules, depth=0)</code> \u2014 resolves refs recursively, replacing <code>field_name</code> in place; raises <code>CircularReferenceError(collection, doc_id, path)</code> when a <code>(Class, id)</code> pair is revisited.</li> <li><code>depopulate(document, rules)</code> \u2014 collapse populated docs back to IDs (for storage).</li> <li><code>global_max_depth</code> caps recursion; per-rule <code>max_depth</code> bounds a rule's descent.</li> </ul>"},{"location":"02_components/#93-circularreferenceerrorvalueerror","title":"9.3 CircularReferenceError(ValueError)","text":"<p>Holds <code>collection</code>, <code>doc_id</code>, and the visited <code>path</code> for debugging cycle messages.</p>"},{"location":"02_components/#related","title":"Related","text":"<ul> <li>Use case 01 \u2013 Basic CRUD \u00b7 07 \u2013 Caching \u00b7 08 \u2013 Population</li> </ul>"},{"location":"04_best_practices/","title":"Best Practices","text":"<p>Team-wide conventions for building fast, testable MongoDB services with <code>mongo-ops</code>.</p>"},{"location":"04_best_practices/#layering","title":"\ud83c\udfd7\ufe0f Layering","text":"<ol> <li>One repository per collection. Encapsulate every query the domain needs behind repository methods; keep Mongo details (<code>$regex</code>, <code>$inc</code>, projections) inside the repository.</li> <li>Keep models thin. <code>BaseDocument</code> for the shape; use Pydantic <code>Field</code> constraints for validation; never put business rules in the model.</li> <li>Use services for cross-repository logic. A <code>Service</code> composes multiple repositories (and <code>TransactionManager</code>) \u2014 routes stay thin.</li> <li>Expose <code>get_many(filter=..., skip=..., limit=..., sort=...)</code> instead of raw <code>find</code> for list endpoints \u2014 you get controlled pagination for free.</li> </ol>"},{"location":"04_best_practices/#lifecycle","title":"\ud83d\udd04 Lifecycle","text":"<ol> <li>Connect once, in the lifespan. <code>MongoConnectionManager.lifespan(...)</code> (or explicit <code>connect</code>/<code>disconnect</code>) \u2014 never lazily per request.</li> <li>Construct repositories after <code>connect()</code>. Module-level <code>Repo()</code> before connection raises <code>RuntimeError(\"Database not connected...\")</code>. Use dependencies or construct inside the lifespan/request.</li> <li>Order the cache lifecycle strictly: <code>set_cache_backend(backend)</code> \u2192 <code>initialize_cache()</code> (after connect, before use) \u2192 <code>shutdown_cache()</code> on exit.</li> <li>Register all models up front via <code>ModelRegistry.register(...)</code> and let <code>initialize_all()</code> create indexes once at startup (idempotent).</li> </ol>"},{"location":"04_best_practices/#data-performance","title":"\ud83d\uddc4\ufe0f Data & Performance","text":"<ol> <li>Declare indexes for every hot query. Single-field, composite, and optioned (unique/TTL) specs all work via <code>ModelRegistry.register</code> \u2014 see use case 13.</li> <li>Cache only hot, low-write <code>_id</code> reads. Use <code>CachedBaseRepository</code> for lookups-by-id; invalidate (<code>update</code>/<code>delete</code> handle it) and pick a sensible <code>default_ttl</code>.</li> <li>Populate at the repository boundary. <code>PopulatingRepository</code> resolves refs on read and depopulates on write; do not hand-roll joins in endpoints.</li> <li>Respect the populate invariants: rules name the field that holds the reference and that is replaced; <code>patch()</code> cannot touch FK fields \u2014 use <code>update()</code>.</li> </ol>"},{"location":"04_best_practices/#transactions-errors","title":"\ud83d\udd01 Transactions & Errors","text":"<ol> <li>Use transactions for multi-document writes. <code>TransactionManager.start_session</code> (inline) or <code>execute_transaction</code> (list of ops) \u2014 and pass <code>session=</code> to every collection call inside.</li> <li>Handle the library's real exceptions at the edges: <code>DuplicateKeyError</code> \u2192 409, <code>InvalidId</code> \u2192 400, <code>CircularReferenceError</code> \u2192 409, <code>ValueError</code> guides \u2192 400/422 (see Error Handling).</li> </ol>"},{"location":"04_best_practices/#testing","title":"\ud83e\uddea Testing","text":"<ol> <li>Default to mock-based unit tests. Patch <code>MongoConnectionManager.get_database</code>, use <code>AsyncMock</code> collections and cursor chains \u2014 the whole suite runs without MongoDB (use case 14).</li> <li>Mirror the library tests. <code>tests/test_{repository,populating_repository,cache,registry,transactions}.py</code> are canonical examples of every pattern above.</li> <li>Use type hints end-to-end \u2014 mypy-gated CI (see pyproject) catches drift early.</li> </ol>"},{"location":"04_best_practices/#related","title":"Related","text":"<ul> <li>Overview \u00b7 Core Components \u00b7 Use Cases</li> </ul>"},{"location":"05_patterns/","title":"Common Patterns","text":"<p>Reusable code shapes built on <code>BaseRepository</code> and friends.</p>"},{"location":"05_patterns/#pattern-1-service-layer-with-repository","title":"Pattern 1: Service Layer with Repository","text":"<p>Keep orchestration out of routes. A service composes one or more repositories and owns the domain rules.</p> Python<pre><code>from mongo_ops import BaseRepository, BaseDocument\n\n\nclass User(BaseDocument):\n username: str = \"\"\n email: str = \"\"\n\n\nclass UserRepository(BaseRepository[User]):\n def __init__(self):\n super().__init__(\"users\", User)\n\n async def find_by_email(self, email: str) -> User | None:\n doc = await self.collection.find_one({\"email\": email})\n return self.model(**doc) if doc else None\n\n\nclass UserService:\n def __init__(self, user_repo: UserRepository):\n self.user_repo = user_repo\n\n async def register_user(self, username: str, email: str) -> User:\n if await self.user_repo.find_by_email(email):\n raise ValueError(\"Email already exists\")\n return await self.user_repo.create(User(username=username, email=email))\n</code></pre> <p>Register a unique index on <code>email</code> via <code>ModelRegistry</code> so the race is also caught by <code>DuplicateKeyError</code> \u2014 see use case 13.</p>"},{"location":"05_patterns/#pattern-2-aggregation-pipeline","title":"Pattern 2: Aggregation Pipeline","text":"<p>Aggregations hit the raw Motor collection \u2014 wrap them in a repository method so callers stay at the domain level.</p> Python<pre><code>from mongo_ops import BaseRepository, BaseDocument\n\n\nclass Post(BaseDocument):\n author_id: str = \"\"\n title: str = \"\"\n likes: int = 0\n\n\nclass PostRepository(BaseRepository[Post]):\n def __init__(self):\n super().__init__(\"posts\", Post)\n\n async def get_user_stats(self, user_id: str) -> dict:\n pipeline = [\n {\"$match\": {\"author_id\": user_id}},\n {\"$group\": {\"_id\": None, \"total_posts\": {\"$sum\": 1}, \"total_likes\": {\"$sum\": \"$likes\"}}},\n ]\n result = await self.collection.aggregate(pipeline).to_list(1)\n return result[0] if result else {\"total_posts\": 0, \"total_likes\": 0}\n</code></pre>"},{"location":"05_patterns/#pattern-3-bulk-operations","title":"Pattern 3: Bulk Operations","text":"<p>Multi-document mutation in one call \u2014 again wrapped at the repository boundary.</p> Python<pre><code>from datetime import datetime\n\nfrom bson import ObjectId\nfrom mongo_ops import BaseRepository, BaseDocument\n\n\nclass Task(BaseDocument):\n status: str = \"pending\"\n\n\nclass TaskRepository(BaseRepository[Task]):\n def __init__(self):\n super().__init__(\"tasks\", Task)\n\n async def bulk_update_status(self, ids: list[str], status: str) -> int:\n object_ids = [ObjectId(i) for i in ids]\n result = await self.collection.update_many(\n {\"_id\": {\"$in\": object_ids}},\n {\"$set\": {\"status\": status, \"updated_at\": datetime.utcnow()}},\n )\n return result.modified_count\n</code></pre> <p>Bulk/aggregation methods bypass the caching and population layers \u2014 keep them deliberate and documented.</p>"},{"location":"05_patterns/#pattern-4-transactional-multi-step-write","title":"Pattern 4: Transactional Multi-Step Write","text":"<p>Cross-collection atomicity with <code>execute_transaction</code> \u2014 see use case 12 for the full code.</p> Python<pre><code>results = await TransactionManager.execute_transaction([op1, op2, op3])\n# op1/op2/op3 each receive the shared session; one failure rolls back all.\n</code></pre>"},{"location":"05_patterns/#related","title":"Related","text":"<ul> <li>02 \u2013 Custom repository \u00b7 03 \u2013 Transactions \u00b7 Best Practices</li> </ul>"},{"location":"06_error_handling/","title":"Error Handling","text":"<p>What can raise, what it means, and how to map it in a FastAPI app.</p>"},{"location":"06_error_handling/#library-raised-exceptions","title":"\ud83d\udccb Library-Raised Exceptions","text":"Exception Source Meaning / fix <code>RuntimeError(\"Database not connected. Call connect() first.\")</code> <code>get_database()</code> / <code>get_client()</code> and any repository constructed first <code>MongoConnectionManager.connect()</code> hasn't run \u2014 wire the lifespan. <code>RuntimeError(\"No cache backend registered. Call set_cache_backend() first.\")</code> <code>ModelRegistry.initialize_cache()</code> Call <code>set_cache_backend(backend)</code> before <code>initialize_cache()</code>. <code>KeyError(\"Model for collection '...' not registered\")</code> <code>ModelRegistry.get_model()</code> Collection was never <code>register</code>ed (or typo). <code>ValueError(\"Cannot patch FK fields via patch(): ...\")</code> <code>PopulatingRepository.patch()</code> <code>patch</code> must not touch populated ref fields \u2014 use <code>update()</code> with a model. <code>ValueError(\"...contains embedded dict(s) \u2014 run repair script\")</code> <code>_populate</code> on read A FK field holds an embedded document instead of an <code>ObjectId</code> \u2014 migrate the data. <code>ValueError(\"...contains ObjectId \u2014 was populate skipped?\")</code> <code>_depopulate</code> on write A populate-ruled field is still an <code>ObjectId</code> at depopulate time \u2014 the read must have populated it first. <code>CircularReferenceError(collection, doc_id, path)</code> <code>PopulationEngine.populate</code> A <code>(Class, id)</code> pair was revisited \u2014 raise <code>max_depth</code> or fix the graph. <code>ImportError(\"redis package required ... mongo-ops[redis]\")</code> <code>CacheConfig</code> / <code>RedisCacheBackend</code> Missing <code>redis</code> extra. <code>ValueError(\"redis_client required when backend='redis'\")</code> <code>CacheConfig</code> <code>backend=\"redis\"</code> without a client. Pymongo <code>DuplicateKeyError</code> any insert/update Unique index violation (e.g., duplicate email). <code>bson.errors.InvalidId</code> <code>ObjectId(...)</code> on a bad string Wrapped by <code>PyObjectId</code> model validation on API inputs. <p><code>KeyError</code> for <code>ModelRegistry.get_model</code> and the <code>RuntimeError</code>/<code>ValueError</code> guards are by design \u2014 they fail loudly at startup or first call instead of misbehaving silently.</p>"},{"location":"06_error_handling/#fastapi-mapping-example","title":"\ud83d\ude80 FastAPI Mapping Example","text":"Python<pre><code>from fastapi import FastAPI, HTTPException, Request\nfrom fastapi.responses import JSONResponse\nfrom bson.errors import InvalidId\nfrom pymongo.errors import DuplicateKeyError\nfrom mongo_ops.cache import CircularReferenceError\n\n\napp = FastAPI()\n\n\n@app.exception_handler(DuplicateKeyError)\nasync def duplicate_key_handler(_: Request, __: DuplicateKeyError) -> JSONResponse:\n return JSONResponse(status_code=409, content={\"detail\": \"Resource already exists\"})\n\n\n@app.exception_handler(InvalidId)\nasync def invalid_id_handler(_: Request, __: InvalidId) -> JSONResponse:\n return JSONResponse(status_code=400, content={\"detail\": \"Invalid ID format\"})\n\n\n@app.exception_handler(CircularReferenceError)\nasync def circular_ref_handler(_: Request, exc: CircularReferenceError) -> JSONResponse:\n return JSONResponse(\n status_code=409,\n content={\"detail\": f\"Circular reference detected: {exc.path}\"},\n )\n\n\n# Or handle inline for route-specific responses:\n@app.post(\"/users/\")\nasync def create_user(user: User, repo=Depends(get_user_repository)):\n try:\n return await repo.create(user)\n except DuplicateKeyError:\n raise HTTPException(status_code=409, detail=\"User already exists\")\n except Exception as exc:\n raise HTTPException(status_code=500, detail=str(exc))\n\n\n@app.get(\"/users/{user_id}\")\nasync def get_user(user_id: str, repo=Depends(get_user_repository)):\n try:\n user = await repo.get_by_id(user_id)\n except InvalidId:\n raise HTTPException(status_code=400, detail=\"Invalid user ID\")\n if not user:\n raise HTTPException(status_code=404, detail=\"User not found\")\n return user\n</code></pre> <p>Prefer exception handlers for the library-level exceptions (409/400 above) and per-endpoint <code>try/except</code> for domain decisions (404).</p>"},{"location":"06_error_handling/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>The <code>ValueError</code> populate guards are your friends: they surface data-shape drift (embedded docs, skipped population) at the exact call site.</li> <li>In dev, log the <code>CircularReferenceError.path</code> \u2014 it prints the visited <code>Class:id</code> chain.</li> <li>Never swallow <code>RuntimeError</code>s at startup; let the app fail to load so the misconfiguration is obvious.</li> </ul>"},{"location":"06_error_handling/#related","title":"Related","text":"<ul> <li>03 \u2013 Transactions \u00b7 08 \u2013 Population \u00b7 11 \u2013 Cache lifecycle</li> </ul>"},{"location":"07_testing_example/","title":"Testing Example","text":"<p>A zero-MongoDB unit-test quickstart. The full guide is use case 14.</p>"},{"location":"07_testing_example/#mock-based-quickstart","title":"\ud83d\ude80 Mock-Based Quickstart","text":"<p>Patch the connection manager and stub the collection \u2014 no network, no docker.</p> Python<pre><code>import pytest\nfrom unittest.mock import AsyncMock, patch\n\nfrom bson import ObjectId\nfrom mongo_ops import BaseDocument, BaseRepository\nfrom mongo_ops.repository import MongoConnectionManager\n\n\nclass User(BaseDocument):\n username: str = \"\"\n email: str = \"\"\n\n\nclass UserRepository(BaseRepository[User]):\n def __init__(self):\n super().__init__(\"users\", User)\n\n\n@pytest.fixture\ndef repo():\n mock_collection = AsyncMock()\n with patch.object(MongoConnectionManager, \"get_database\") as mock_db:\n mock_db.return_value.__getitem__.return_value = mock_collection\n r = UserRepository()\n r.collection = mock_collection\n return r\n\n\n@pytest.mark.asyncio\nasync def test_create_user(repo):\n mock_collection = repo.collection\n oid = ObjectId()\n mock_collection.insert_one.return_value.inserted_id = oid\n\n user = await repo.create(User(username=\"test\", email=\"test@example.com\"))\n\n assert user.id == oid\n assert user.username == \"test\"\n\n\n@pytest.mark.asyncio\nasync def test_get_by_id(repo):\n oid = ObjectId()\n repo.collection.find_one.return_value = {\n \"_id\": oid,\n \"username\": \"test\",\n \"email\": \"test@example.com\",\n \"created_at\": \"2024-01-01T00:00:00\",\n \"updated_at\": \"2024-01-01T00:00:00\",\n }\n\n fetched = await repo.get_by_id(str(oid))\n\n assert fetched is not None\n assert fetched.username == \"test\"\n</code></pre>"},{"location":"07_testing_example/#notes","title":"\ud83d\udca1 Notes","text":"<ul> <li><code>from mongo_ops.repository import MongoConnectionManager</code> \u2014 patch where it is used (<code>mongo_ops.repository.MongoConnectionManager</code>), matching the library's own tests.</li> <li>pytest-asyncio runs as <code>auto</code> mode per <code>pyproject.toml</code>, so <code>@pytest.mark.asyncio</code> works without extra config.</li> <li>For population, cache, registry, and transaction mockups \u2014 see the full testing guide.</li> </ul>"},{"location":"07_testing_example/#related","title":"Related","text":"<ul> <li>14 \u2013 Testing guide \u00b7 Best Practices</li> </ul>"},{"location":"03_use_cases/01_basic_crud/","title":"Use Case 1: Basic FastAPI CRUD API","text":"<p>Scenario: A simple user management API with CRUD endpoints, index registration, and a correct connection lifecycle.</p>"},{"location":"03_use_cases/01_basic_crud/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>MongoConnectionManager.lifespan</code> Connect on startup, disconnect on shutdown. <code>BaseRepository[User]</code> Generic CRUD \u2014 no endpoint-level Mongo code. <code>ModelRegistry.register</code> Declares the <code>email</code> index; <code>initialize_all</code> creates it at startup."},{"location":"03_use_cases/01_basic_crud/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>import os\nfrom contextlib import asynccontextmanager\n\nfrom fastapi import Depends, FastAPI, HTTPException\nfrom pydantic import Field\n\nfrom mongo_ops import BaseDocument, BaseRepository, ModelRegistry, MongoConnectionManager\n\n# ---------------------------\n# Model\n# ---------------------------\nclass User(BaseDocument):\n username: str = Field(..., min_length=3, max_length=50)\n email: str = Field(...)\n is_active: bool = True\n\n\n# ---------------------------\n# Repository\n# ---------------------------\nclass UserRepository(BaseRepository[User]):\n def __init__(self):\n super().__init__(\"users\", User)\n\n\n# ---------------------------\n# Registration + index\n# ---------------------------\nModelRegistry.register(\"users\", User, indexes=[(\"email\", 1)])\n\n\n# ---------------------------\n# Lifecycle\n# ---------------------------\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=os.getenv(\"MONGO_URI\", \"mongodb://localhost:27017\"),\n db_name=\"mydb\",\n ):\n await ModelRegistry.initialize_all()\n yield\n\n\napp = FastAPI(lifespan=lifespan)\n\n\n# ---------------------------\n# Dependency \u2014 created per request, after connect()\n# ---------------------------\ndef get_user_repository() -> UserRepository:\n return UserRepository()\n\n\n# ---------------------------\n# Routes\n# ---------------------------\n@app.post(\"/users/\", response_model=User)\nasync def create_user(user: User, repo: UserRepository = Depends(get_user_repository)):\n return await repo.create(user)\n\n\n@app.get(\"/users/{user_id}\", response_model=User)\nasync def get_user(user_id: str, repo: UserRepository = Depends(get_user_repository)):\n user = await repo.get_by_id(user_id)\n if not user:\n raise HTTPException(status_code=404, detail=\"User not found\")\n return user\n\n\n@app.get(\"/users/\", response_model=list[User])\nasync def list_users(\n skip: int = 0,\n limit: int = 10,\n repo: UserRepository = Depends(get_user_repository),\n):\n return await repo.get_many(skip=skip, limit=limit)\n\n\n@app.put(\"/users/{user_id}\", response_model=User)\nasync def update_user(\n user_id: str,\n email: str,\n repo: UserRepository = Depends(get_user_repository),\n):\n user = await repo.update(user_id, {\"email\": email})\n if not user:\n raise HTTPException(status_code=404, detail=\"User not found\")\n return user\n\n\n@app.delete(\"/users/{user_id}\")\nasync def delete_user(user_id: str, repo: UserRepository = Depends(get_user_repository)):\n deleted = await repo.delete(user_id)\n if not deleted:\n raise HTTPException(status_code=404, detail=\"User not found\")\n return {\"message\": \"User deleted successfully\"}\n</code></pre>"},{"location":"03_use_cases/01_basic_crud/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li><code>UserRepository()</code> is created inside the dependency, i.e., only after the lifespan has connected. Instantiating a repository at module import time raises <code>RuntimeError</code> because the database isn't connected yet.</li> <li><code>BaseRepository.update(id, {...})</code> takes a dict; only <code>PopulatingRepository.update</code> takes a model.</li> <li>Use <code>patch()</code> for REST PATCH semantics \u2014 it accepts a partial dict like <code>update</code>, but <code>PopulatingRepository</code> rejects FK fields.</li> </ul>"},{"location":"03_use_cases/01_basic_crud/#related","title":"Related","text":"<ul> <li>02 \u2013 Custom repository \u00b7 04 \u2013 Pagination \u00b7 Overview</li> </ul>"},{"location":"03_use_cases/02_custom_repo/","title":"Use Case 2: Custom Repository with Business Logic","text":"<p>Scenario: An e-commerce product catalog needs search, filtering, and stock updates without Mongo leaking into routes.</p>"},{"location":"03_use_cases/02_custom_repo/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description Repository methods Encapsulate queries (<code>$regex</code>, filters, <code>$inc</code>) behind domain methods. <code>get_many</code> Filtering + default pagination via the base repository. Direct <code>collection</code> access For operations with no base-repo helper (regex search, atomic <code>$inc</code>)."},{"location":"03_use_cases/02_custom_repo/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from fastapi import FastAPI, HTTPException, Query\nfrom mongo_ops import BaseDocument, BaseRepository\n\n# ---------------------------\n# Model\n# ---------------------------\nclass Product(BaseDocument):\n name: str\n description: str = \"\"\n price: float = 0.0\n category: str = \"\"\n in_stock: bool = True\n quantity: int = 0\n tags: list[str] = []\n\n\n# ---------------------------\n# Repository\n# ---------------------------\nclass ProductRepository(BaseRepository[Product]):\n def __init__(self):\n super().__init__(\"products\", Product)\n\n async def search_by_name(self, query: str) -> list[Product]:\n \"\"\"Case-insensitive name search.\"\"\"\n docs = await self.collection.find(\n {\"name\": {\"$regex\": query, \"$options\": \"i\"}}\n ).to_list(length=100)\n return [self.model(**doc) for doc in docs]\n\n async def get_by_category(self, category: str, in_stock_only: bool = True) -> list[Product]:\n filter_query = {\"category\": category}\n if in_stock_only:\n filter_query[\"in_stock\"] = True\n return await self.get_many(filter=filter_query)\n\n async def get_low_stock(self, threshold: int = 10) -> list[Product]:\n return await self.get_many(filter={\"quantity\": {\"$lt\": threshold}, \"in_stock\": True})\n\n async def update_stock(self, product_id: str, quantity_delta: int) -> Product | None:\n \"\"\"Atomically increment/decrement stock.\"\"\"\n from bson import ObjectId\n from datetime import datetime\n\n result = await self.collection.find_one_and_update(\n {\"_id\": ObjectId(product_id)},\n {\"$inc\": {\"quantity\": quantity_delta}, \"$set\": {\"updated_at\": datetime.utcnow()}},\n return_document=True,\n )\n return self.model(**result) if result else None\n\n\napp = FastAPI()\nproduct_repo = ProductRepository()\n\n\n@app.get(\"/products/search\", response_model=list[Product])\nasync def search_products(q: str = Query(..., min_length=1)):\n return await product_repo.search_by_name(q)\n\n\n@app.get(\"/products/category/{category}\", response_model=list[Product])\nasync def products_by_category(category: str, in_stock: bool = True):\n return await product_repo.get_by_category(category, in_stock)\n\n\n@app.get(\"/products/low-stock\", response_model=list[Product])\nasync def low_stock_products(threshold: int = 10):\n return await product_repo.get_low_stock(threshold)\n\n\n@app.patch(\"/products/{product_id}/stock\")\nasync def update_product_stock(product_id: str, quantity_delta: int):\n product = await product_repo.update_stock(product_id, quantity_delta)\n if not product:\n raise HTTPException(status_code=404, detail=\"Product not found\")\n return product\n</code></pre> <p>Note: this snippet omits the FastAPI <code>lifespan</code> connection wiring for brevity \u2014 copy it from use case 01 so <code>ProductRepository()</code> is created only after <code>MongoConnectionManager.connect()</code>.</p>"},{"location":"03_use_cases/02_custom_repo/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Methods that hit <code>self.collection</code> directly (regex search, <code>$inc</code>) bypass the caching and population layers. If a feature composes them \u2014 extend <code>CachedBaseRepository</code> or <code>PopulatingRepository</code> instead and add the domain methods there.</li> <li>Prefer <code>get_many(filter=...)</code> over raw <code>find</code> when you want pagination/sort defaults for free.</li> <li>Reuse <code>self.model(**doc)</code> to convert raw dicts to model instances consistently.</li> </ul>"},{"location":"03_use_cases/02_custom_repo/#related","title":"Related","text":"<ul> <li>01 \u2013 Basic CRUD \u00b7 04 \u2013 Pagination \u00b7 05 \u2013 Soft deletes</li> </ul>"},{"location":"03_use_cases/03_transactions/","title":"Use Case 3: Transaction Support for Multi-Document Operations","text":"<p>Scenario: Order processing must update inventory and create an order atomically. Any failure rolls both back.</p>"},{"location":"03_use_cases/03_transactions/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>TransactionManager.start_session</code> Async context manager yielding a session with an active transaction. <code>session=</code> kwarg Pass to every <code>insert_one</code> / <code>update_one</code> / <code>find_one</code> inside the block."},{"location":"03_use_cases/03_transactions/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from contextlib import asynccontextmanager\nfrom datetime import datetime\n\nfrom bson import ObjectId\nfrom fastapi import FastAPI, HTTPException\nfrom mongo_ops import BaseDocument, BaseRepository, MongoConnectionManager, TransactionManager\n\n\nclass Order(BaseDocument):\n user_id: str\n items: list[dict] # [{\"product_id\": \"...\", \"quantity\": 2}]\n total_amount: float = 0.0\n status: str = \"pending\"\n\n\nclass Inventory(BaseDocument):\n product_id: ObjectId = None\n quantity: int = 0\n\n\nclass OrderRepository(BaseRepository[Order]):\n def __init__(self):\n super().__init__(\"orders\", Order)\n\n\nclass InventoryRepository(BaseRepository[Inventory]):\n def __init__(self):\n super().__init__(\"inventory\", Inventory)\n\n\nasync def create_order_with_inventory_update(\n order: Order,\n order_repo: OrderRepository,\n inv_repo: InventoryRepository,\n) -> Order:\n async def insert_order(session):\n doc = order.model_dump(exclude={\"id\"}, exclude_none=True)\n doc[\"created_at\"] = doc[\"updated_at\"] = datetime.utcnow()\n result = await order_repo.collection.insert_one(doc, session=session)\n return await order_repo.collection.find_one({\"_id\": result.inserted_id}, session=session)\n\n async def update_inventory(session):\n for item in order.items:\n await inv_repo.collection.update_one(\n {\"product_id\": ObjectId(item[\"product_id\"])},\n {\n \"$inc\": {\"quantity\": -item[\"quantity\"]},\n \"$set\": {\"updated_at\": datetime.utcnow()},\n },\n session=session,\n )\n return None\n\n async with TransactionManager.start_session() as session:\n created = await insert_order(session)\n await update_inventory(session)\n return Order(**created)\n\n\n# ---------------------------\n# FastAPI wiring\n# ---------------------------\napp = FastAPI()\n\norder_repo = OrderRepository()\ninv_repo = InventoryRepository()\n\n\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"shop\"\n ):\n yield\n\n\napp.lifespan = lifespan\n\n\n@app.post(\"/orders/\", response_model=Order)\nasync def create_order(order: Order):\n try:\n return await create_order_with_inventory_update(order, order_repo, inv_repo)\n except Exception as exc:\n raise HTTPException(status_code=400, detail=str(exc))\n</code></pre>"},{"location":"03_use_cases/03_transactions/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Repositories are created inside the lifespan (or a dependency) so the database is connected \u2014 see the note in use case 01.</li> <li>Every operation inside the transaction context must receive <code>session=<session></code> \u2014 a missing session silently runs outside the transaction.</li> <li>MongoDB transactions require a replica set (or the local standalone test server that emulates one).</li> <li>For a list-of-operations style, see <code>TransactionManager.execute_transaction</code> in use case 12.</li> </ul>"},{"location":"03_use_cases/03_transactions/#related","title":"Related","text":"<ul> <li>06 \u2013 Multi-model \u00b7 12 \u2013 Transaction helper \u00b7 Error Handling</li> </ul>"},{"location":"03_use_cases/04_pagination/","title":"Use Case 4: Pagination & Filtering","text":"<p>Scenario: A blog post API lists posts with page metadata, filtering, and sorting.</p>"},{"location":"03_use_cases/04_pagination/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>BaseRepository.get_many</code> <code>filter</code>, <code>skip</code>, <code>limit</code>, <code>sort</code> in one call. <code>BaseRepository.count</code> Total matching documents for page metadata. <code>PaginatedResponse</code> Reusable generic response wrapper."},{"location":"03_use_cases/04_pagination/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from typing import Generic, TypeVar\n\nfrom fastapi import FastAPI, Query\nfrom pydantic import BaseModel\nfrom mongo_ops import BaseDocument, BaseRepository\n\nT = TypeVar(\"T\")\n\n\nclass PaginatedResponse(BaseModel, Generic[T]):\n items: list[T]\n total: int\n page: int\n page_size: int\n total_pages: int\n has_next: bool\n has_prev: bool\n\n\nclass BlogPost(BaseDocument):\n title: str = \"\"\n content: str = \"\"\n author_id: str = \"\"\n published: bool = False\n tags: list[str] = []\n views: int = 0\n\n\nclass BlogPostRepository(BaseRepository[BlogPost]):\n def __init__(self):\n super().__init__(\"blog_posts\", BlogPost)\n\n async def paginate(\n self,\n page: int = 1,\n page_size: int = 10,\n filter_dict: dict | None = None,\n sort_by: str = \"created_at\",\n sort_order: int = -1,\n ) -> PaginatedResponse[BlogPost]:\n filter_dict = filter_dict or {}\n skip = (page - 1) * page_size\n total = await self.count(filter_dict)\n items = await self.get_many(\n filter=filter_dict,\n skip=skip,\n limit=page_size,\n sort=[(sort_by, sort_order)],\n )\n total_pages = (total + page_size - 1) // page_size\n return PaginatedResponse(\n items=items,\n total=total,\n page=page,\n page_size=page_size,\n total_pages=total_pages,\n has_next=page < total_pages,\n has_prev=page > 1,\n )\n\n async def get_by_author(self, author_id: str, published_only: bool = True) -> list[BlogPost]:\n filter_dict = {\"author_id\": author_id}\n if published_only:\n filter_dict[\"published\"] = True\n return await self.get_many(filter=filter_dict, sort=[(\"created_at\", -1)])\n\n async def search_by_tags(self, tags: list[str]) -> list[BlogPost]:\n return await self.get_many(filter={\"tags\": {\"$in\": tags}, \"published\": True})\n\n\napp = FastAPI()\nblog_repo = BlogPostRepository()\n\n\n@app.get(\"/posts/\", response_model=PaginatedResponse[BlogPost])\nasync def list_posts(\n page: int = 1,\n page_size: int = 10,\n published: bool | None = None,\n author_id: str | None = None,\n):\n filter_dict = {}\n if published is not None:\n filter_dict[\"published\"] = published\n if author_id:\n filter_dict[\"author_id\"] = author_id\n return await blog_repo.paginate(page, page_size, filter_dict)\n\n\n@app.get(\"/posts/author/{author_id}\", response_model=list[BlogPost])\nasync def posts_by_author(author_id: str, published: bool = True):\n return await blog_repo.get_by_author(author_id, published)\n\n\n@app.get(\"/posts/tags\", response_model=list[BlogPost])\nasync def posts_by_tags(tags: list[str] = Query(...)):\n return await blog_repo.search_by_tags(tags)\n</code></pre> <p>Note: add the <code>lifespan</code> wiring from use case 01 so <code>blog_repo</code> is created after connection.</p>"},{"location":"03_use_cases/04_pagination/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li><code>get_many</code> calls <code>cursor.to_list(limit)</code>; pass <code>limit=0</code> to skip the limit entirely, otherwise a large explicit limit is safer than unbounded reads.</li> <li>Always <code>count</code> first for stable metadata \u2014 it uses the same filter as the page query.</li> <li>Combine with <code>sort=[(field, -1|1)]</code> for deterministic ordering; create matching indexes to avoid full collection scans (see use case 13).</li> </ul>"},{"location":"03_use_cases/04_pagination/#related","title":"Related","text":"<ul> <li>02 \u2013 Custom repository \u00b7 13 \u2013 Indexes \u00b7 05 \u2013 Soft deletes</li> </ul>"},{"location":"03_use_cases/05_soft_deletes/","title":"Use Case 5: Soft Deletes Pattern","text":"<p>Scenario: Deleting a task should be recoverable. Instead of removing the document, set a tombstone flag and filter it from normal queries.</p>"},{"location":"03_use_cases/05_soft_deletes/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>SoftDeleteDocument</code> Base model carrying <code>is_deleted</code>, <code>deleted_at</code>, <code>deleted_by</code>. <code>SoftDeleteRepository[T]</code> Repository-level soft delete / restore / filtering using <code>update</code> + <code>get_many</code>. <code>permanent_delete</code> Escapes to the real <code>delete</code>."},{"location":"03_use_cases/05_soft_deletes/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from datetime import datetime\n\nfrom fastapi import FastAPI, HTTPException\nfrom mongo_ops import BaseDocument, BaseRepository\n\n\nclass SoftDeleteDocument(BaseDocument):\n is_deleted: bool = False\n deleted_at: datetime | None = None\n deleted_by: str | None = None\n\n\nclass Task(SoftDeleteDocument):\n title: str = \"\"\n description: str = \"\"\n assignee_id: str = \"\"\n status: str = \"pending\"\n priority: str = \"medium\"\n\n\nclass SoftDeleteRepository(BaseRepository[T]):\n async def soft_delete(self, id: str, deleted_by: str | None = None):\n return await self.update(\n id,\n {\"is_deleted\": True, \"deleted_at\": datetime.utcnow(), \"deleted_by\": deleted_by},\n )\n\n async def restore(self, id: str):\n return await self.update(\n id, {\"is_deleted\": False, \"deleted_at\": None, \"deleted_by\": None}\n )\n\n async def get_active(self, skip: int = 0, limit: int = 100):\n return await self.get_many(filter={\"is_deleted\": False}, skip=skip, limit=limit)\n\n async def get_deleted(self, skip: int = 0, limit: int = 100):\n return await self.get_many(filter={\"is_deleted\": True}, skip=skip, limit=limit)\n\n async def permanent_delete(self, id: str) -> bool:\n return await self.delete(id)\n\n\nclass TaskRepository(SoftDeleteRepository[Task]):\n def __init__(self):\n super().__init__(\"tasks\", Task)\n\n\napp = FastAPI()\ntask_repo = TaskRepository()\n\n\n@app.delete(\"/tasks/{task_id}\")\nasync def soft_delete_task(task_id: str, user_id: str):\n task = await task_repo.soft_delete(task_id, deleted_by=user_id)\n if not task:\n raise HTTPException(status_code=404, detail=\"Task not found\")\n return {\"message\": \"Task deleted\", \"task\": task}\n\n\n@app.post(\"/tasks/{task_id}/restore\")\nasync def restore_task(task_id: str):\n task = await task_repo.restore(task_id)\n if not task:\n raise HTTPException(status_code=404, detail=\"Task not found\")\n return {\"message\": \"Task restored\", \"task\": task}\n\n\n@app.get(\"/tasks/\", response_model=list[Task])\nasync def list_active_tasks(skip: int = 0, limit: int = 10):\n return await task_repo.get_active(skip, limit)\n\n\n@app.get(\"/tasks/deleted\", response_model=list[Task])\nasync def list_deleted_tasks(skip: int = 0, limit: int = 10):\n return await task_repo.get_deleted(skip, limit)\n</code></pre> <p>Note: add the <code>lifespan</code> wiring from use case 01 so <code>task_repo</code> is created after connection.</p>"},{"location":"03_use_cases/05_soft_deletes/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li><code>BaseRepository[T]</code> is generic \u2014 subclasses must parameterize it (see <code>SoftDeleteRepository[T]</code> above). <code>from mongo_ops import BaseRepository; BaseRepository[T]</code> works out of the box.</li> <li>Keep a compound index on <code>{\"is_deleted\": 1, \"status\": 1}</code> for active-list queries (see use case 13).</li> <li>Soft-deleted documents should be excluded at the repository boundary, never re-filtered ad hoc in endpoints.</li> </ul>"},{"location":"03_use_cases/05_soft_deletes/#related","title":"Related","text":"<ul> <li>01 \u2013 Basic CRUD \u00b7 13 \u2013 Indexes \u00b7 04 \u2013 Pagination</li> </ul>"},{"location":"03_use_cases/06_multi_model/","title":"Use Case 6: Multi-Model Service with Registration","text":"<p>Scenario: A social app manages <code>users</code>, <code>posts</code>, and <code>comments</code>. Each has its own model, repository, and indexes \u2014 registered centrally and initialized at startup.</p>"},{"location":"03_use_cases/06_multi_model/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>ModelRegistry.register</code> One call per collection \u2014 model + indexes together. <code>ModelRegistry.initialize_all</code> Creates every registered index at startup (idempotent). Repositories One repository class per collection, all sharing the same connection."},{"location":"03_use_cases/06_multi_model/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from contextlib import asynccontextmanager\n\nfrom fastapi import FastAPI\nfrom mongo_ops import BaseDocument, BaseRepository, ModelRegistry, MongoConnectionManager\n\n\nclass User(BaseDocument):\n username: str = \"\"\n email: str = \"\"\n role: str = \"user\"\n\n\nclass Post(BaseDocument):\n title: str = \"\"\n content: str = \"\"\n author_id: str = \"\"\n likes: int = 0\n\n\nclass Comment(BaseDocument):\n post_id: str = \"\"\n user_id: str = \"\"\n content: str = \"\"\n\n\nclass UserRepository(BaseRepository[User]):\n def __init__(self):\n super().__init__(\"users\", User)\n\n\nclass PostRepository(BaseRepository[Post]):\n def __init__(self):\n super().__init__(\"posts\", Post)\n\n\nclass CommentRepository(BaseRepository[Comment]):\n def __init__(self):\n super().__init__(\"comments\", Comment)\n\n\n# ---------------------------\n# Central registration + indexes\n# ---------------------------\nModelRegistry.register(\"users\", User, indexes=[(\"email\", 1), (\"username\", 1)])\nModelRegistry.register(\"posts\", Post, indexes=[(\"author_id\", 1), (\"created_at\", -1)])\nModelRegistry.register(\"comments\", Comment, indexes=[(\"post_id\", 1), (\"user_id\", 1)])\n\n\n# ---------------------------\n# Lifecycle\n# ---------------------------\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"social_app\"\n ):\n await ModelRegistry.initialize_all()\n yield\n\n\napp = FastAPI(lifespan=lifespan)\n\n\n# Created after connect() \u2014 inside the lifespan body is fine, or use a dependency.\nuser_repo = UserRepository()\npost_repo = PostRepository()\ncomment_repo = CommentRepository()\n\n\n@app.post(\"/users/\", response_model=User)\nasync def create_user(user: User):\n return await user_repo.create(user)\n\n\n@app.post(\"/posts/\", response_model=Post)\nasync def create_post(post: Post):\n return await post_repo.create(post)\n\n\n@app.post(\"/comments/\", response_model=Comment)\nasync def create_comment(comment: Comment):\n return await comment_repo.create(comment)\n</code></pre>"},{"location":"03_use_cases/06_multi_model/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Place <code>user_repo = ...</code> inside the lifespan/after connect. Module-level instantiation before <code>connect()</code> raises <code>RuntimeError(\"Database not connected...\")</code>.</li> <li>Model relationships here are plain <code>ObjectId</code> strings stored on the child docs. To resolve them on read, see use case 08 \u2013 Population.</li> <li>Registering indexes on <code>created_at</code>/<code>author_id</code>/<code>post_id</code> keeps the common queries indexed (see use case 13).</li> </ul>"},{"location":"03_use_cases/06_multi_model/#related","title":"Related","text":"<ul> <li>01 \u2013 Basic CRUD \u00b7 08 \u2013 Population \u00b7 13 \u2013 Indexes</li> </ul>"},{"location":"03_use_cases/07_caching/","title":"Use Case 7: Caching for High-Performance Reads","text":"<p>Scenario: A read-heavy API (product catalog) reduces DB load by caching documents by <code>_id</code> \u2014 in-memory locally, or shared via Redis.</p>"},{"location":"03_use_cases/07_caching/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>InMemoryCacheBackend</code> TTL + LRU cache with a background cleanup task. <code>RedisCacheBackend</code> Distributed cache on <code>redis.asyncio</code> with pub/sub invalidation. <code>CachedBaseRepository[T]</code> Extends <code>BaseRepository</code> \u2014 cache-first <code>get_by_id</code>, cache on <code>create</code>, invalidate on <code>update</code>/<code>delete</code>, <code>warm_cache(ids)</code>, <code>invalidate_cache(id)</code>. Backend lifecycle The same backend instance must be both passed to the repository AND registered via <code>ModelRegistry.set_cache_backend</code> so <code>initialize_cache()</code> starts its task."},{"location":"03_use_cases/07_caching/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from contextlib import asynccontextmanager\n\nfrom fastapi import FastAPI, HTTPException\nfrom mongo_ops import BaseDocument, CachedBaseRepository, ModelRegistry, MongoConnectionManager\nfrom mongo_ops.cache import CacheConfig, InMemoryCacheBackend\n\n\nclass Product(BaseDocument):\n name: str = \"\"\n price: float = 0.0\n\n\n# One shared backend \u2014 used by both the repository and the registry lifecycle.\ncache = InMemoryCacheBackend(max_entries=10_000, default_ttl=300)\nModelRegistry.set_cache_backend(cache)\n\n\nclass ProductRepo(CachedBaseRepository[Product]):\n def __init__(self):\n super().__init__(\n collection_name=\"products\",\n model=Product,\n cache_backend=cache,\n config=CacheConfig(enabled=True, backend=\"memory\"),\n )\n\n\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"shop\"\n ):\n await ModelRegistry.initialize_all()\n await ModelRegistry.initialize_cache() # starts the TTL cleanup task\n yield\n await ModelRegistry.shutdown_cache() # cancels it on exit\n\n\napp = FastAPI(lifespan=lifespan)\n\n\n@app.post(\"/products/\", response_model=Product)\nasync def create_product(product: Product):\n return await ProductRepo().create(product) # created after connect()\n\n\n@app.get(\"/products/{product_id}\", response_model=Product)\nasync def get_product(product_id: str):\n product = await ProductRepo().get_by_id(product_id)\n if not product:\n raise HTTPException(status_code=404, detail=\"Product not found\")\n return product\n\n\n@app.put(\"/products/{product_id}\", response_model=Product)\nasync def update_product(product_id: str, name: str | None = None, price: float | None = None):\n data = {}\n if name is not None:\n data[\"name\"] = name\n if price is not None:\n data[\"price\"] = price\n return await ProductRepo().update(product_id, data)\n\n\n# Outside request handlers:\n# await ProductRepo().warm_cache([object_id_1, object_id_2]) # pre-load n ids -> int\n# await ProductRepo().invalidate_cache(object_id_3) # manual eviction\n</code></pre>"},{"location":"03_use_cases/07_caching/#redis-backend","title":"\ud83d\udd04 Redis Backend","text":"<p>Swap the backend \u2014 the repository code stays identical:</p> Python<pre><code>from mongo_ops.cache.redis_backend import RedisCacheBackend\nfrom redis.asyncio import Redis\n\nredis_client = Redis(host=\"localhost\", port=6379)\nredis_backend = RedisCacheBackend(redis_client, key_prefix=\"prod:\")\n\nModelRegistry.set_cache_backend(redis_backend) # for the lifecycle in lifespan()\n\nclass ProductRepo(CachedBaseRepository[Product]):\n def __init__(self):\n super().__init__(\n collection_name=\"products\",\n model=Product,\n cache_backend=redis_backend,\n config=CacheConfig(enabled=True, backend=\"redis\"),\n )\n</code></pre>"},{"location":"03_use_cases/07_caching/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Cache keys are <code>\"{key_prefix}{id}\"</code> (default prefix <code>\"products:\"</code>). <code>clear_pattern(\"products:*\")</code> wipes a whole collection's entries.</li> <li>Values are JSON-encoded (<code>json.dumps(obj, default=str)</code>) \u2014 nested models inside a cached document are stored as dicts, not objects.</li> <li>Enable/disable per repository with <code>CacheConfig(enabled=False)</code>; a disabled repo bypasses the cache entirely.</li> <li>Distinguish the two <code>initialize*</code> calls: <code>initialize_cache()</code> starts the backend task; <code>initialize_all()</code> creates indexes. Both belong in the lifespan, after <code>connect()</code>.</li> </ul>"},{"location":"03_use_cases/07_caching/#related","title":"Related","text":"<ul> <li>11 \u2013 Cache lifecycle \u00b7 10 \u2013 Cache + population \u00b7 Components: cache</li> </ul>"},{"location":"03_use_cases/08_population/","title":"Use Case 8: Document Population","text":"<p>Scenario: An API returns a <code>User</code> with its related <code>Profile</code> embedded in one JSON payload \u2014 no second round-trip from the client, no joins.</p>"},{"location":"03_use_cases/08_population/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>PopulateRule</code> Declares which field to resolve. <code>field_name</code> holds the <code>ObjectId</code> (or <code>list[ObjectId]</code>) and is the same field the resolved document replaces in place. <code>collection_name</code> says where the referenced documents live. <code>PopulationEngine</code> Holds repositories by collection name and resolves rules recursively, detecting cycles (<code>CircularReferenceError</code>). <code>PopulatingRepository[T]</code> <code>get_by_id</code>/<code>get_many</code> populate on read; <code>create</code>/<code>update</code> depopulate on write; <code>patch</code> rejects FK fields. <p>\u26a0\ufe0f <code>PopulateRule</code> does not have a separate \"ref field\" vs \"target field\" \u2014 the ref field is the populated field. <code>filter</code>/<code>projection</code> on <code>PopulateRule</code> are declared but not yet applied by the engine.</p>"},{"location":"03_use_cases/08_population/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from contextlib import asynccontextmanager\n\nfrom fastapi import FastAPI, HTTPException\nfrom mongo_ops import BaseDocument, MongoConnectionManager, ModelRegistry, PopulatingRepository\nfrom mongo_ops.populate import PopulateRule, PopulationEngine\n\n\n# 1. Models \u2014 `profile` holds an ObjectId in the DB and becomes a Profile on read.\nclass Profile(BaseDocument):\n avatar_url: str = \"\"\n bio: str = \"\"\n\n\nclass User(BaseDocument):\n username: str = \"\"\n email: str = \"\"\n profile: Profile | None = None # ObjectId in MongoDB, Profile in memory\n\n\n# 2. Engine + rule \u2014 repositories are registered later (after connect()).\nengine = PopulationEngine({})\nprofile_rule = PopulateRule(\n field_name=\"profile\",\n collection_name=\"profiles\",\n)\n\n\n# 3. The repository the app uses.\nclass UserRepository(PopulatingRepository[User]):\n def __init__(self):\n super().__init__(\n collection_name=\"users\",\n model=User,\n population_engine=engine,\n populate_rules=[profile_rule],\n )\n\n\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"mydb\"\n ):\n # Tells the engine which repository resolves \"profiles\" refs.\n engine.register_repo(\"profiles\", PopulatingRepository[Profile](\"profiles\", Profile))\n await ModelRegistry.initialize_all()\n yield\n\n\napp = FastAPI(lifespan=lifespan)\n\n\n# 4. Writing \u2014 pass the model; the repository depopulates to an ObjectId.\n@app.post(\"/users/\", response_model=User)\nasync def create_user():\n user = User(\n username=\"alice\",\n email=\"alice@example.com\",\n profile=Profile(avatar_url=\"alice.png\", bio=\"hi\"),\n )\n return await UserRepository().create(user)\n # ^ stored as {\"profile\": <ObjectId>}, returned with profile populated\n\n\n# 5. Reading \u2014 get_by_id resolves the reference on the way back.\n@app.get(\"/users/{user_id}\", response_model=User)\nasync def get_user(user_id: str):\n user = await UserRepository().get_by_id(user_id)\n if not user:\n raise HTTPException(status_code=404, detail=\"User not found\")\n return user # user.profile is a Profile instance\n</code></pre>"},{"location":"03_use_cases/08_population/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li><code>Patch</code>ing an FK field raises <code>ValueError</code> \u2014 switch FK changes to <code>update(user_id, model)</code> instead. See use case 05 for the reasoning with soft deletes.</li> <li>Missing references resolve to <code>None</code>, not an error.</li> <li>A field holding an embedded dict (instead of an <code>ObjectId</code>) raises a <code>ValueError</code> (\"run repair script\") \u2014 migrate embedded docs to a separate collection first.</li> <li>If <code>create</code>/<code>update</code> receives a field that is already an <code>ObjectId</code> under a populate rule, <code>_depopulate</code> raises <code>ValueError(\"...was populate skipped?\")</code> \u2014 populate-then-depopulate pairs must be balanced.</li> </ul>"},{"location":"03_use_cases/08_population/#related","title":"Related","text":"<ul> <li>09 \u2013 Advanced population \u00b7 10 \u2013 Cache + population \u00b7 14 \u2013 Testing guide</li> </ul>"},{"location":"03_use_cases/09_advanced_population/","title":"Use Case 9: Nested Document Population & Circular-Ref Handling","text":"<p>Scenario: An <code>Author</code> has <code>books</code> (a list of references), each <code>Book</code> references a <code>publisher</code>, and an <code>Author</code> may reference a <code>mentor</code> \u2014 which is another <code>Author</code> (a potential cycle).</p>"},{"location":"03_use_cases/09_advanced_population/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>nested_rules</code> Populates deeper levels: resolve <code>books</code> on <code>Author</code>, then <code>publisher</code> inside each <code>Book</code>. <code>max_depth</code> Per-rule recursion bound \u2014 the safety net for cyclic graphs. <code>global_max_depth</code> <code>PopulationEngine(repos, global_max_depth=10)</code> global cap. <code>CircularReferenceError</code> Raised when a <code>(Class, id)</code> pair is revisited; carries the visited path."},{"location":"03_use_cases/09_advanced_population/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from contextlib import asynccontextmanager\n\nfrom bson import ObjectId\nfrom fastapi import FastAPI, HTTPException\nfrom mongo_ops import BaseDocument, ModelRegistry, MongoConnectionManager, PopulatingRepository\nfrom mongo_ops.cache import CircularReferenceError\nfrom mongo_ops.populate import PopulateRule, PopulationEngine\n\n\n# 1. Models \u2014 each ref field holds ObjectId(s) in the DB and becomes model(s) in memory.\nclass Publisher(BaseDocument):\n name: str = \"\"\n country: str = \"\"\n\n\nclass Book(BaseDocument):\n title: str = \"\"\n publisher: Publisher | None = None # ObjectId in DB, Publisher in memory\n\n\nclass Author(BaseDocument):\n name: str = \"\"\n books: list[Book] | None = None # list[ObjectId] in DB, list[Book] in memory\n mentor: \"Author\" | None = None # self-reference \u2014 potential cycle\n\n\n# 2. Engine + nested rules.\nengine = PopulationEngine({})\n\npublisher_rule = PopulateRule(\n field_name=\"publisher\",\n collection_name=\"publishers\",\n)\n\nbook_rule = PopulateRule(\n field_name=\"books\",\n collection_name=\"books\",\n nested_rules=[publisher_rule], # fetch each book, then its publisher\n max_depth=3,\n)\n\nmentor_rule = PopulateRule(\n field_name=\"mentor\",\n collection_name=\"authors\",\n max_depth=2, # stops mentor chains early \u2014 also avoids unbounded cycles\n)\n\n\nclass AuthorRepository(PopulatingRepository[Author]):\n def __init__(self):\n super().__init__(\n collection_name=\"authors\",\n model=Author,\n population_engine=engine,\n populate_rules=[book_rule, mentor_rule],\n )\n\n\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"library\"\n ):\n engine.register_repo(\"publishers\", PopulatingRepository[Publisher](\"publishers\", Publisher))\n engine.register_repo(\"books\", PopulatingRepository[Book](\"books\", Book))\n engine.register_repo(\"authors\", AuthorRepository())\n await ModelRegistry.initialize_all()\n yield\n\n\napp = FastAPI()\n\n\n@app.get(\"/authors/{author_id}\")\nasync def get_author(author_id: str):\n try:\n author = await AuthorRepository().get_by_id(author_id)\n except CircularReferenceError as exc:\n raise HTTPException(status_code=409, detail=f\"Circular reference: {exc}\")\n if not author:\n raise HTTPException(status_code=404, detail=\"Author not found\")\n return author\n</code></pre>"},{"location":"03_use_cases/09_advanced_population/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li><code>CircularReferenceError</code> is a <code>ValueError</code> subtype exposing <code>.collection</code>, <code>.doc_id</code>, and the visited <code>.path</code> \u2014 use it in error responses and logging.</li> <li>Keep <code>max_depth</code> conservative (2\u20133) for most graphs; combine with the global <code>global_max_depth=10</code> default.</li> <li>List refs depopulate back to <code>list[ObjectId]</code> on <code>create</code>/<code>update</code>, mirroring the scalar case.</li> </ul>"},{"location":"03_use_cases/09_advanced_population/#related","title":"Related","text":"<ul> <li>08 \u2013 Population \u00b7 10 \u2013 Cache + population \u00b7 Error Handling</li> </ul>"},{"location":"03_use_cases/10_cache_and_population/","title":"Use Case 10: Caching + Population (Read-Through, Populated on Read)","text":"<p>Scenario: A service needs fast reads of a <code>User</code> and its related <code>Profile</code>. We want the cache-hit path to skip the DB entirely while still returning a populated document.</p>"},{"location":"03_use_cases/10_cache_and_population/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>CachedBaseRepository</code> Provides the cache layer (<code>cache-first get_by_id</code>, invalidation on <code>update</code>/<code>delete</code>). <code>PopulationEngine</code> Provides the reference resolution. Composed subclass A small <code>CachedBaseRepository</code> subclass adds two <code>_populate</code> helpers so cache hits return populated models. <p>\u26a0\ufe0f Known limitation: there is no built-in <code>CachedPopulatingRepository</code> in the library. This use case documents the composition. We cache the raw (depopulated) document \u2014 references stay <code>ObjectId</code>s until read time, so <code>Profile</code> changes are reflected on the next fetch (within TTL) and <code>update</code>/<code>delete</code> invalidation remains correct.</p>"},{"location":"03_use_cases/10_cache_and_population/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from contextlib import asynccontextmanager\nfrom datetime import datetime\nfrom typing import Any, Union\n\nfrom bson import ObjectId\nfrom fastapi import FastAPI\nfrom mongo_ops import BaseDocument, CachedBaseRepository, ModelRegistry, MongoConnectionManager\nfrom mongo_ops.cache import CacheConfig, InMemoryCacheBackend\nfrom mongo_ops.cache.in_memory import decode_value, encode_value\nfrom mongo_ops.populate import PopulateRule, PopulationEngine\n\n\n# 1. Models \u2014 `profile` holds an ObjectId in DB, a Profile in memory.\nclass Profile(BaseDocument):\n avatar_url: str = \"\"\n bio: str = \"\"\n\n\nclass User(BaseDocument):\n username: str = \"\"\n email: str = \"\"\n profile: Profile | None = None\n\n\n# 2. Engine + rule.\nengine = PopulationEngine({})\nprofile_rule = PopulateRule(field_name=\"profile\", collection_name=\"profiles\")\n\n\n# 3. Composed repository.\nclass CachedUserRepository(CachedBaseRepository[User]):\n def __init__(\n self,\n cache_backend: InMemoryCacheBackend,\n population_engine: PopulationEngine,\n populate_rules: list[PopulateRule],\n config: CacheConfig | None = None,\n ):\n super().__init__(\"users\", User, cache_backend, config)\n self.population_engine = population_engine\n self._populate_rules = populate_rules\n\n async def _populate(self, data: dict[str, Any]) -> dict[str, Any]:\n \"\"\"Dict-level resolution (mirrors PopulatingRepository._populate).\"\"\"\n for rule in self._populate_rules:\n ref = data.get(rule.field_name)\n if ref is None:\n continue\n if isinstance(ref, str): # JSON round-trip turns ObjectId into hex str\n ref = ObjectId(ref)\n if isinstance(ref, list):\n resolved = []\n for item in ref:\n item = ObjectId(item) if isinstance(item, str) else item\n repo = self.population_engine._repos.get(rule.collection_name)\n doc = await repo.get_by_id(item) if repo else None\n resolved.append(doc)\n data[rule.field_name] = resolved\n elif isinstance(ref, ObjectId):\n repo = self.population_engine._repos.get(rule.collection_name)\n data[rule.field_name] = await repo.get_by_id(ref) if repo else None\n return data\n\n async def _depopulate(self, data: User) -> dict[str, Any]:\n \"\"\"Model -> raw dict, FK fields collapsed to ObjectId (mirrors _depopulate).\"\"\"\n doc = data.model_dump(exclude={\"id\"}, exclude_none=True)\n for rule in self._populate_rules:\n value = getattr(data, rule.field_name, None)\n if isinstance(value, list):\n doc[rule.field_name] = [item.id for item in value]\n elif isinstance(value, BaseDocument):\n doc[rule.field_name] = value.id\n return doc\n\n async def create(self, data: User) -> User:\n doc = await self._depopulate(data) # store FK refs as ObjectIds\n doc[\"created_at\"] = datetime.utcnow()\n doc[\"updated_at\"] = datetime.utcnow()\n result = await self.collection.insert_one(doc)\n doc[\"_id\"] = result.inserted_id\n if self._cache_config.enabled:\n await self._cache.set(self._cache_key(doc[\"_id\"]), encode_value(doc), self._cache_config.default_ttl)\n return self.model(**await self._populate(doc))\n\n async def update(self, id: Union[str, ObjectId], data: User) -> User | None:\n if isinstance(id, str):\n id = ObjectId(id)\n doc = await self._depopulate(data) # update() takes a full model here\n doc[\"updated_at\"] = datetime.utcnow()\n result = await self.collection.find_one_and_update(\n {\"_id\": id}, {\"$set\": doc}, return_document=True\n )\n key = self._cache_key(id)\n if result is None:\n if self._cache_config.enabled:\n await self._cache.delete(key)\n return None\n raw = dict(result)\n if self._cache_config.enabled:\n await self._cache.set(key, encode_value(raw), self._cache_config.default_ttl)\n return self.model(**await self._populate(raw))\n\n async def get_by_id(self, id: Union[str, ObjectId]) -> User | None:\n if not self._cache_config.enabled:\n # No cache: still a raw fetch + populate \u2014 the base get_by_id()\n # would build a typed model from the raw doc and fail on FK fields.\n raw = await self.collection.find_one({\"_id\": ObjectId(id) if isinstance(id, str) else id})\n return self.model(**await self._populate(raw)) if raw else None\n\n cached = await self._cache.get(self._cache_key(id))\n if cached is not None:\n # Cache hit: decode the RAW doc, then populate before returning.\n data = await self._populate(decode_value(cached))\n return self.model(**data)\n\n # Cache miss: one RAW DB read. The base get_by_id() would rebuild the\n # model from the raw doc, which fails for model-typed FK fields \u2014\n # so fetch the raw dict, cache it, and populate before materializing.\n key = self._cache_key(id)\n raw = await self.collection.find_one({\"_id\": ObjectId(id) if isinstance(id, str) else id})\n if raw is None:\n return None\n await self._cache.set(key, encode_value(raw), self._cache_config.default_ttl)\n return self.model(**await self._populate(raw))\n\n\n# 4. Wire-up \u2014 one backend for both the repo and the registry lifecycle.\ncache = InMemoryCacheBackend(max_entries=20_000, default_ttl=600)\nModelRegistry.set_cache_backend(cache)\n\n\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"app_db\"\n ):\n # Register the repositories the engine resolves refs against:\n # engine.register_repo(\"profiles\", ProfileRepo())\n engine.register_repo(\"users\", CachedUserRepository(cache, engine, [profile_rule]))\n await ModelRegistry.initialize_all()\n await ModelRegistry.initialize_cache()\n yield\n await ModelRegistry.shutdown_cache()\n\n\napp = FastAPI(lifespan=lifespan)\n</code></pre> <p>Note: replace the placeholder <code>engine.register_repo(\"profiles\", ...)</code> line with a repository for <code>Profile</code>. A matching <code>CachedBaseRepository[Profile](\"profiles\", Profile, cache)</code> keeps profile reads cached too \u2014 e.g. <code>engine.register_repo(\"profiles\", CachedBaseRepository[Profile](\"profiles\", Profile, cache))</code>.</p>"},{"location":"03_use_cases/10_cache_and_population/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Cache the raw doc, populate at read time. Ref fields stay <code>ObjectId</code>s in the cache; JSON round-trips them to hex strings, so <code>_populate</code> must normalize <code>str \u2192 ObjectId</code> (shown above).</li> <li><code>update</code>/<code>create</code> above are model-based (<code>update(id, User)</code>), mirroring <code>PopulatingRepository</code> \u2014 they depopulate before writing and cache a raw snapshot. The plain <code>CachedBaseRepository.update(id, dict)</code> and <code>delete(id)</code> keep working and invalidate the same key.</li> <li>Set a sensible <code>default_ttl</code> \u2014 cached <code>User</code> entries resolve <code>Profile</code> on each read, so profile edits show up within the TTL (or call <code>invalidate_cache(user_id)</code> explicitly).</li> <li>To reuse this compose logic across many collections, extract the <code>_populate</code> / <code>_depopulate</code> helpers plus the <code>create</code> / <code>update</code> / <code>get_by_id</code> overrides into a mixin and parameterize the rules per subclass.</li> </ul>"},{"location":"03_use_cases/10_cache_and_population/#related","title":"Related","text":"<ul> <li>07 \u2013 Caching \u00b7 08 \u2013 Population \u00b7 15 \u2013 PopulatingRepository internals \u00b7 16 \u2013 Cached repository intricacies</li> </ul>"},{"location":"03_use_cases/11_cache_lifecycle/","title":"Use Case 11: Proper Cache Lifecycle in FastAPI","text":"<p>Scenario: A microservice uses an in-memory (or Redis) cache backend and must start the cleanup task on app start, then shut it down cleanly on termination.</p>"},{"location":"03_use_cases/11_cache_lifecycle/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>ModelRegistry.set_cache_backend(backend)</code> Registers the single shared backend. <code>ModelRegistry.initialize_cache()</code> Starts the backend (spawns the in-memory TTL cleanup task). Raises <code>RuntimeError</code> if no backend was registered. <code>ModelRegistry.shutdown_cache()</code> Cancels the cleanup task, closes pub/sub, clears the backend. If a repo uses a backend directly Same instance must be registered so <code>initialize_cache</code> starts its task."},{"location":"03_use_cases/11_cache_lifecycle/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from contextlib import asynccontextmanager\n\nfrom fastapi import FastAPI\nfrom mongo_ops import ModelRegistry, MongoConnectionManager\nfrom mongo_ops.cache import InMemoryCacheBackend\n\ncache = InMemoryCacheBackend(max_entries=10_000, default_ttl=300)\nModelRegistry.set_cache_backend(cache) # before any cache-backed repo is used\n\n\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"mydb\"\n ):\n await ModelRegistry.initialize_all() # create indexes (idempotent)\n await ModelRegistry.initialize_cache() # start the TTL cleanup task\n yield\n await ModelRegistry.shutdown_cache() # cancel task + close cleanly\n\n\napp = FastAPI(lifespan=lifespan)\n</code></pre>"},{"location":"03_use_cases/11_cache_lifecycle/#what-actually-happens","title":"\ud83d\udd01 What Actually Happens","text":"<ul> <li><code>InMemoryCacheBackend.initialize()</code> spawns an <code>asyncio.Task</code> that evicts expired entries every <code>cleanup_interval</code> seconds. Without a shutdown, the event loop flags the dangling task on exit \u2014 <code>shutdown_cache()</code> cancels it and awaits it.</li> <li><code>shutdown_cache()</code> also NULs the registry cache backend and closes the Redis pub/sub handle (if Redis).</li> <li>If <code>initialize_cache()</code> is called before <code>set_cache_backend()</code>, it raises:</li> </ul> Text Only<pre><code>RuntimeError: No cache backend registered. Call set_cache_backend() first.\n</code></pre>"},{"location":"03_use_cases/11_cache_lifecycle/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Per-repository toggling is independent of the lifecycle: <code>CacheConfig(enabled=False)</code> bypasses the cache for that repo even after <code>initialize_cache()</code>.</li> <li>The cleanup task uses <code>cleanup_interval</code> seconds for scans; entries also expire on access via the TTL heap (<code>default_ttl=0</code> expires immediately).</li> <li>Register the backend before constructing any <code>CachedBaseRepository</code> that references it \u2014 otherwise a repo may hold an uninitialized backend (no cleanup task, no Redis pub/sub).</li> </ul>"},{"location":"03_use_cases/11_cache_lifecycle/#related","title":"Related","text":"<ul> <li>07 \u2013 Caching \u00b7 Components: ModelRegistry \u00b7 10 \u2013 Cache + population</li> </ul>"},{"location":"03_use_cases/12_transaction_helper/","title":"Use Case 12: Using <code>TransactionManager.execute_transaction</code>","text":"<p>Scenario: Perform several writes across different collections atomically \u2014 e.g., create an <code>Order</code> and decrement <code>Inventory</code>. The low-level <code>start_session</code> context works, but <code>execute_transaction</code> collects results from a list of async operations.</p>"},{"location":"03_use_cases/12_transaction_helper/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>TransactionManager.execute_transaction</code> Runs a list of async callables (each receives a session) inside one transaction; returns the result of each callable, in order. Automatic rollback Any raised exception aborts the transaction and propagates to the caller. <code>start_session</code> The underlying async context manager <code>async with TransactionManager.start_session() as session:</code>."},{"location":"03_use_cases/12_transaction_helper/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from contextlib import asynccontextmanager\nfrom datetime import datetime\n\nfrom bson import ObjectId\nfrom fastapi import FastAPI, HTTPException\nfrom mongo_ops import BaseDocument, ModelRegistry, MongoConnectionManager\nfrom mongo_ops.repository import BaseRepository\nfrom mongo_ops.transactions import TransactionManager\n\n\n# 1. Models\nclass Order(BaseDocument):\n user_id: str = \"\"\n items: list[dict] = [] # [{\"product_id\": ObjectId, \"qty\": int}]\n total: float = 0.0\n\n\nclass Inventory(BaseDocument):\n product_id: ObjectId = None\n quantity: int = 0\n\n\n# 2. Repositories (constructed inside the lifespan \u2014 after connect()).\nclass OrderRepo(BaseRepository[Order]):\n def __init__(self):\n super().__init__(\"orders\", Order)\n\n\nclass InventoryRepo(BaseRepository[Inventory]):\n def __init__(self):\n super().__init__(\"inventory\", Inventory)\n\n\n# 3. The atomic operation.\nasync def create_order_with_inventory(order: Order, order_repo: OrderRepo, inv_repo: InventoryRepo):\n async def insert_order(session):\n doc = order.model_dump(exclude={\"id\"}, exclude_none=True)\n doc[\"created_at\"] = doc[\"updated_at\"] = datetime.utcnow()\n result = await order_repo.collection.insert_one(doc, session=session)\n return await order_repo.collection.find_one({\"_id\": result.inserted_id}, session=session)\n\n async def update_inventory(session):\n for item in order.items:\n await inv_repo.collection.update_one(\n {\"product_id\": ObjectId(item[\"product_id\"])},\n {\"$inc\": {\"quantity\": -item[\"qty\"]}},\n session=session,\n )\n return \"inventory-updated\"\n\n results = await TransactionManager.execute_transaction(\n [insert_order, update_inventory]\n )\n return results[0] # created order document; results[1] == \"inventory-updated\"\n\n\n# 4. FastAPI wiring.\napp = FastAPI()\n\n\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"shop\"\n ):\n await ModelRegistry.initialize_all()\n yield\n\n\napp.lifespan = lifespan\n\n\n@app.post(\"/orders/\", response_model=Order)\nasync def create_order(order: Order):\n created = await create_order_with_inventory(order, OrderRepo(), InventoryRepo())\n if created is None:\n raise HTTPException(status_code=400, detail=\"Transaction failed\")\n return created\n</code></pre>"},{"location":"03_use_cases/12_transaction_helper/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Return values: each callable can return whatever you need; results are collected in the same order.</li> <li>Errors: raise inside any callable \u2192 the whole transaction aborts (session rolls back) and the exception propagates.</li> <li>Reads inside a transaction: pass <code>session=session</code> to <code>find_one</code>/<code>find</code> too.</li> <li>Testing (no Mongo): <code>monkeypatch</code> a fake client on <code>mongo_ops.transactions.MongoConnectionManager.get_client</code> and stub <code>start_session</code> \u2014 see <code>tests/test_transactions.py</code>.</li> <li>Repo models still carry <code>created_at</code>/<code>updated_at</code>; for raw collection inserts inside the transaction you set them manually (as shown).</li> </ul>"},{"location":"03_use_cases/12_transaction_helper/#related","title":"Related","text":"<ul> <li>03 \u2013 Transactions \u00b7 14 \u2013 Testing guide \u00b7 Error Handling</li> </ul>"},{"location":"03_use_cases/13_index_creation/","title":"Use Case 13: Declaring Indexes (Single-Field, Composite, Unique)","text":"<p>Scenario: Ensure each collection has the right indexes for fast queries and data integrity \u2014 declared in one place and created at startup.</p>"},{"location":"03_use_cases/13_index_creation/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>ModelRegistry.register(indexes=...)</code> Each spec is passed as-is to pymongo <code>create_index</code>. Supported forms: tuple <code>(\"field\", direction)</code>, compound list <code>[(\"a\", 1), (\"b\", -1)]</code>, or a dict with <code>keys</code> + <code>options</code>. <code>ModelRegistry.initialize_all</code> Creates every registered index during startup (idempotent \u2014 <code>create_index</code> skips existing indexes)."},{"location":"03_use_cases/13_index_creation/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from contextlib import asynccontextmanager\n\nfrom pymongo import ASCENDING, DESCENDING\nfrom mongo_ops import BaseDocument, ModelRegistry, MongoConnectionManager\n\n\nclass User(BaseDocument):\n username: str = \"\"\n email: str = \"\"\n\n\nclass BlogPost(BaseDocument):\n author_id: str = \"\"\n created_at: str = \"\"\n title: str = \"\"\n\n\nclass Passenger(BaseDocument):\n email: str = \"\"\n seat: str = \"\"\n\n\n# 1\ufe0f\u20e3 Single-field index \u2014 email lookups.\nModelRegistry.register(\n collection_name=\"users\",\n model=User,\n indexes=[(\"email\", ASCENDING)],\n)\n\n# 2\ufe0f\u20e3 Composite index \u2014 queries filtering by author + creation date.\nModelRegistry.register(\n collection_name=\"posts\",\n model=BlogPost,\n indexes=[[(\"author_id\", ASCENDING), (\"created_at\", DESCENDING)]],\n)\n\n# 3\ufe0f\u20e3 Unique index with a custom name \u2014 enforce unique emails.\nModelRegistry.register(\n collection_name=\"passengers\",\n model=Passenger,\n indexes=[\n {\n \"keys\": [(\"email\", ASCENDING)],\n \"options\": {\"unique\": True, \"name\": \"uq_passenger_email\"},\n }\n ],\n)\n\n\n# 4\ufe0f\u20e3 Everything is created on startup.\n@asynccontextmanager\nasync def lifespan(_app):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\",\n db_name=\"mydb\",\n ):\n await ModelRegistry.initialize_all()\n yield\n</code></pre>"},{"location":"03_use_cases/13_index_creation/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Every spec passes through to <code>collection.create_index(spec)</code> \u2014 so MongoDB options like <code>unique</code>, <code>sparse</code>, and <code>expireAfterSeconds</code> (TTL) belong in the <code>options</code> dict.</li> <li>Idempotent by construction: <code>create_index</code> is a no-op when a same-shape index already exists.</li> <li>Verify with the Mongo shell:</li> </ul> JavaScript<pre><code>db.<collection>.getIndexes()\n</code></pre> <ul> <li>A unique index on an already-duplicated field will fail with <code>DuplicateKeyError</code> on startup \u2014 clean the data first.</li> </ul>"},{"location":"03_use_cases/13_index_creation/#related","title":"Related","text":"<ul> <li>01 \u2013 Basic CRUD \u00b7 06 \u2013 Multi-model \u00b7 Components: ModelRegistry</li> </ul>"},{"location":"03_use_cases/14_testing_guide/","title":"Use Case 14: Testing Guide \u2014 Mocking Motor, Cache, Registry & Transactions","text":"<p>Scenario: Write unit tests that never touch a real MongoDB \u2014 mock the collection, the cache backend, and the client, exactly like the library's own test suite (<code>tests/</code>).</p>"},{"location":"03_use_cases/14_testing_guide/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description <code>AsyncMock</code> collections Stub <code>find_one</code>, <code>insert_one</code>, <code>find_one_and_update</code>, cursor chains. Patching <code>MongoConnectionManager.get_database</code> Gives repositories a mock collection without a live connection. <code>monkeypatch</code> on <code>get_client</code> Fakes sessions for <code>TransactionManager</code> tests. Reference tests <code>tests/test_populating_repository.py</code>, <code>tests/test_cache.py</code>, <code>tests/test_registry.py</code>, <code>tests/test_transactions.py</code>."},{"location":"03_use_cases/14_testing_guide/#example-boilerplate","title":"\ud83d\ude80 Example Boilerplate","text":"Python<pre><code>import pytest\nfrom unittest.mock import AsyncMock, MagicMock, patch\n\nfrom bson import ObjectId\nfrom mongo_ops.cache import CacheConfig, InMemoryCacheBackend\nfrom mongo_ops.cache.repository import CachedBaseRepository\nfrom mongo_ops.models import BaseDocument\nfrom mongo_ops.registry import ModelRegistry\nfrom mongo_ops.populate import PopulateRule, PopulationEngine\nfrom mongo_ops.repository import PopulatingRepository\n\n\n# ----------------------------------------------------------------------\n# 1. Models (same shape as the library tests)\n# ----------------------------------------------------------------------\nclass Profile(BaseDocument):\n avatar_url: str = \"\"\n\n\nclass User(BaseDocument):\n name: str = \"\"\n profile: Profile | None = None # ObjectId in DB, Profile in memory\n\n\n# ----------------------------------------------------------------------\n# 2. Fixtures\n# ----------------------------------------------------------------------\n@pytest.fixture\ndef mock_collection():\n return AsyncMock()\n\n\n@pytest.fixture\ndef engine():\n profile_repo = AsyncMock()\n return PopulationEngine({\"profiles\": profile_repo})\n\n\n@pytest.fixture\ndef repo(mock_collection, engine):\n with patch(\"mongo_ops.repository.MongoConnectionManager.get_database\") as mock_db:\n mock_db.return_value.__getitem__.return_value = mock_collection\n r = PopulatingRepository(\n \"users\",\n User,\n population_engine=engine,\n populate_rules=[PopulateRule(field_name=\"profile\", collection_name=\"profiles\")],\n )\n r.collection = mock_collection\n return r\n\n\n# ----------------------------------------------------------------------\n# 3. Population \u2014 get_by_id resolves the reference\n# ----------------------------------------------------------------------\n@pytest.mark.asyncio\nasync def test_get_by_id_populates(repo, mock_collection, engine):\n uid, pid = ObjectId(), ObjectId()\n mock_collection.find_one.return_value = {\n \"_id\": uid,\n \"name\": \"Alice\",\n \"profile\": pid, # ObjectId stored in DB\n \"created_at\": \"2024-01-01T00:00:00\",\n \"updated_at\": \"2024-01-01T00:00:00\",\n }\n engine._repos[\"profiles\"].get_by_id.return_value = Profile(id=pid, avatar_url=\"pic.png\")\n\n result = await repo.get_by_id(uid)\n\n assert result is not None\n assert result.name == \"Alice\"\n assert isinstance(result.profile, Profile)\n assert result.profile.avatar_url == \"pic.png\"\n\n\n# ----------------------------------------------------------------------\n# 4. Patch FK guard\n# ----------------------------------------------------------------------\n@pytest.mark.asyncio\nasync def test_patch_rejects_fk_field(repo):\n with pytest.raises(ValueError, match=\"Cannot patch FK fields\"):\n await repo.patch(ObjectId(), {\"profile\": ObjectId()})\n\n\n# ----------------------------------------------------------------------\n# 5. Registry \u2014 index specs pass through to create_index\n# ----------------------------------------------------------------------\n@pytest.mark.asyncio\nasync def test_initialize_all_creates_indexes():\n ModelRegistry.register(\"users\", User, indexes=[(\"email\", 1)])\n\n fake_collection = AsyncMock()\n await ModelRegistry.initialize_all(db={\"users\": fake_collection})\n\n fake_collection.create_index.assert_awaited_once_with((\"email\", 1))\n\n\n# ----------------------------------------------------------------------\n# 6. Cached repository \u2014 cache-first reads + invalidation\n# ----------------------------------------------------------------------\n@pytest.mark.asyncio\nasync def test_cached_get_by_id_populates_cache():\n backend = InMemoryCacheBackend(\n max_entries=100, default_ttl=300, cleanup_interval=9999\n )\n await backend.initialize()\n try:\n with patch(\"mongo_ops.repository.MongoConnectionManager.get_database\") as mock_db:\n mock_collection = AsyncMock()\n mock_db.return_value.__getitem__.return_value = mock_collection\n repo = CachedBaseRepository(\n \"users\", User, backend, CacheConfig(enabled=True)\n )\n repo.collection = mock_collection\n\n oid = ObjectId()\n mock_collection.find_one.return_value = {\n \"_id\": oid,\n \"name\": \"cached\",\n \"created_at\": \"2024-01-01T00:00:00\",\n \"updated_at\": \"2024-01-01T00:00:00\",\n }\n\n first = await repo.get_by_id(oid)\n assert first is not None\n\n mock_collection.find_one.return_value = None # DB now \"empty\"\n second = await repo.get_by_id(oid) # served from cache\n\n assert second is not None\n assert second.name == \"cached\"\n mock_collection.find_one.assert_awaited_once() # only one DB read\n finally:\n await backend.shutdown()\n\n\n# ----------------------------------------------------------------------\n# 7. Transactions \u2014 fake the client's start_session\n# ----------------------------------------------------------------------\nfrom mongo_ops.transactions import TransactionManager\n\n\n@pytest.mark.asyncio\nasync def test_execute_transaction(monkeypatch):\n session_ctx = AsyncMock()\n session_ctx.__aenter__.return_value = AsyncMock()\n session_ctx.__aexit__.return_value = None\n\n client = MagicMock()\n client.start_session = AsyncMock(return_value=session_ctx)\n\n # NOTE: start_transaction must return a context manager, not a coroutine.\n async_session = session_ctx.__aenter__.return_value\n async_session.start_transaction = lambda **_: session_ctx\n\n monkeypatch.setattr(\n \"mongo_ops.transactions.MongoConnectionManager.get_client\",\n lambda: client,\n )\n\n async def fake_op(session):\n return \"ok\"\n\n results = await TransactionManager.execute_transaction([fake_op])\n assert results == [\"ok\"]\n</code></pre>"},{"location":"03_use_cases/14_testing_guide/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>pytest-asyncio is already configured in <code>pyproject.toml</code> (<code>asyncio_mode = \"auto\"</code>), so <code>@pytest.mark.asyncio</code> tests work out of the box. Run with <code>pytest</code> (coverage reports are enabled there too).</li> <li>Never hit the network. Keep the patches in fixtures (or a <code>conftest.py</code>) and reuse them.</li> <li>Mock <code>cursor</code> chains with <code>MagicMock()</code> + <code>.to_list = AsyncMock(...)</code>, exactly like <code>tests/test_repository.py</code>.</li> <li>For an optional integration check (real Mongo), use <code>MongoConnectionManager.lifespan</code> against a local replica set and drop the test database in teardown \u2014 keep it separate from the unit suite.</li> <li>The pattern works symmetrically for Redis: mock the <code>RedisCacheBackend</code> methods (<code>get</code>, <code>set</code>, <code>delete</code>) \u2014 no Redis process required.</li> </ul>"},{"location":"03_use_cases/14_testing_guide/#related","title":"Related","text":"<ul> <li>08 \u2013 Population \u00b7 07 \u2013 Caching \u00b7 12 \u2013 Transaction helper</li> </ul>"},{"location":"03_use_cases/15_populating_repository_wiring/","title":"Use Case 15: Inside <code>PopulatingRepository</code> \u2014 the Object \u21c4 ObjectId Lifecycle","text":"<p>Scenario: You want to see what the repository actually does, in which order, before you trust it with your data \u2014 how a <code>Profile</code> becomes an <code>ObjectId</code> for storage and comes back as a <code>Profile</code> on read, and where every piece is wired.</p>"},{"location":"03_use_cases/15_populating_repository_wiring/#two-representations-one-field","title":"\ud83d\udce6 Two Representations, One Field","text":"<p>A populate-ruled field is a shape shifter \u2014 the same name holds different things depending on where you look:</p> Place Value held in the field In the MongoDB doc <code>ObjectId</code> (or <code>list[ObjectId]</code>) In the cache/JSON hex string (see use case 16) In the app model the referenced model (or <code>None</code>) <p><code>PopulatingRepository.create</code> / <code>update</code> depopulate (model \u2192 <code>ObjectId</code>) before writing; <code>get_by_id</code> / <code>get_many</code> populate (<code>ObjectId</code> \u2192 model) after reading:</p> Text Only<pre><code>App model MongoDB document App model\nUser(profile=Profile) \u2500\u2500depopulate\u2500\u2500\u25b6 { profile: <ObjectId> } \u2500\u2500populate\u2500\u2500\u25b6 User(profile=Profile)\n (write path) (storage shape) (read path)\n</code></pre>"},{"location":"03_use_cases/15_populating_repository_wiring/#write-path-create-step-by-step","title":"\ud83d\ude80 Write Path \u2014 <code>create()</code> Step by Step","text":"<p>Given <code>user = User(username=\"alice\", profile=saved_profile)</code> where <code>saved_profile.id</code> exists:</p> Python<pre><code>class UserRepository(PopulatingRepository[User]):\n def __init__(self):\n super().__init__(\n collection_name=\"users\",\n model=User,\n population_engine=engine,\n populate_rules=[profile_rule],\n )\n</code></pre> <ol> <li>Guard check (<code>_depopulate</code>, <code>repository.py</code>): for each rule field, if the value is already an <code>ObjectId</code> (or <code>list[ObjectId]</code>) an error is raised \u2014 see _depopulate guards below. Saved models pass.</li> <li><code>engine.depopulate(user, rules)</code> collapses the graph in place:</li> <li><code>saved_profile</code> (a <code>BaseDocument</code> with <code>.id</code>) \u2192 becomes <code>saved_profile.id</code> \u2192 <code>ObjectId</code></li> <li>a <code>Profile</code> without <code>.id</code> (unsaved) \u2192 is left as a model object, which <code>model_dump</code> then embeds as a dict \u2014 see intricacy #3</li> <li>when <code>nested_rules</code> are present, the nested model is depopulated recursively first, then collapsed</li> <li><code>model_dump(exclude={\"id\"}, exclude_none=True)</code> produces the raw insert dict; timestamps are added.</li> <li><code>insert_one(doc)</code> writes <code>{..., \"profile\": ObjectId(\"...\"), ...}</code> to MongoDB.</li> <li>On the way out, <code>data_to_model</code> runs <code>_populate</code> (the read path below) so <code>create</code> returns a fully populated model, not the raw one.</li> </ol>"},{"location":"03_use_cases/15_populating_repository_wiring/#the-_depopulate-guards-loud-failures-silent-corruption","title":"The <code>_depopulate</code> guards (loud failures > silent corruption)","text":"Stored/held value under a rule field Behaviour <code>ObjectId</code> / <code>list[ObjectId]</code> <code>ValueError(\"...contains ObjectId \u2014 was populate skipped?\")</code> \u2014 the doc was loaded raw (e.g. via <code>BaseRepository</code> or an unpopulated read) and handed back to <code>create</code>/<code>update</code> a saved <code>BaseDocument</code> (has <code>.id</code>) collapse to <code>.id</code> (<code>ObjectId</code>) an unsaved <code>BaseDocument</code> (no <code>.id</code>) kept as a model \u2192 embedded dict below a <code>dict</code> not <code>BaseDocument</code> \u2192 left as-is \u2192 embedded non-<code>BaseDocument</code>, non-dict value <code>AttributeError</code> raised by <code>engine.depopulate</code> no engine / no rules passthrough \u2014 <code>_depopulate</code> just does <code>model_dump</code>, whatever shape you gave is stored"},{"location":"03_use_cases/15_populating_repository_wiring/#read-path-get_by_id-step-by-step","title":"\ud83d\ude80 Read Path \u2014 <code>get_by_id()</code> Step by Step","text":"<p><code>get_by_id</code> inherits CRUD and only changes <code>data_to_model</code> (<code>repository.py:247</code>):</p> Python<pre><code>doc = await self.collection.find_one({\"_id\": id}) # {\"profile\": ObjectId, ...}\nreturn await self.data_to_model(doc) # data_to_model \u2192 self._populate(doc)\n</code></pre> <p><code>_populate(data)</code> walks each rule field in the raw dict:</p> <ol> <li><code>ref = data.get(\"profile\")</code> \u2192 <code>ObjectId</code>. If <code>ref is None</code> \u2192 field untouched (stays absent).</li> <li>Look up the repo: <code>repo = engine._repos.get(\"profiles\")</code>. If it's not registered, <code>doc = None</code>.</li> <li><code>doc = await repo.get_by_id(ref)</code> \u2014 a single read on the referenced collection.</li> <li>If <code>doc</code> is <code>None</code> \u2192 <code>data[\"profile\"] = None</code> (missing refs resolve to <code>None</code>, never raise).</li> <li>If the rule has <code>nested_rules</code> \u2192 <code>engine.populate(doc, nested_rules, depth=1)</code> deepens the result.</li> <li><code>data[\"profile\"] = doc</code> \u2192 <code>self.model(**data)</code> builds the <code>User</code> with a real <code>Profile</code>.</li> </ol> <p>Lists behave the same per item; a <code>ref</code> that is a dict (an embedded document) raises <code>ValueError(\"...contains embedded dict(s) \u2014 run repair script\")</code>.</p> <p>Engine is optional. Set <code>population_engine=None</code> and <code>populate_rules=[]</code>, and <code>PopulatingRepository</code> is just a <code>BaseRepository</code> \u2014 FK fields come back as raw <code>ObjectId</code>, and your model field type must agree (that is the whole point of the guard in the WRITE path: a <code>Profile | None</code>-typed field populated with a raw <code>ObjectId</code> is a broken round-trip waiting to happen).</p>"},{"location":"03_use_cases/15_populating_repository_wiring/#how-its-wired","title":"\ud83d\udd0c How It's Wired","text":"Python<pre><code>from mongo_ops import BaseDocument, PopulatingRepository\nfrom mongo_ops.populate import PopulateRule, PopulationEngine\n\nclass Profile(BaseDocument):\n avatar_url: str = \"\"\n\nclass User(BaseDocument):\n username: str = \"\"\n profile: Profile | None = None # ObjectId in DB, Profile in memory\n\nengine = PopulationEngine({}) # repositories live here, keyed by collection name\n\nprofile_rule = PopulateRule(field_name=\"profile\", collection_name=\"profiles\")\n\nclass UserRepository(PopulatingRepository[User]):\n def __init__(self):\n super().__init__(\"users\", User, population_engine=engine, populate_rules=[profile_rule])\n\ndef wire() -> None:\n \"\"\"Call AFTER connect() \u2014 repositories need a live database.\"\"\"\n engine.register_repo(\"profiles\", PopulatingRepository[Profile](\"profiles\", Profile))\n engine.register_repo(\"users\", UserRepository())\n</code></pre> <p>Wiring rules:</p> <ul> <li><code>engine._repos</code> is keyed by <code>collection_name</code> as written in the rule \u2014 typo \u2192 silent <code>None</code> refs.</li> <li><code>register_repo</code> needs an already-constructed repository \u2192 call it inside the lifespan (after <code>connect()</code> / <code>MongoConnectionManager.lifespan</code>), not at module import.</li> <li>The referenced repository only needs a <code>get_by_id</code> that returns a <code>BaseDocument</code> \u2014 it can be a plain <code>BaseRepository</code>, another <code>PopulatingRepository</code>, or even a cached repo (see use case 10). Population requires no extra DB index on the referenced collection's <code>_id</code>.</li> <li>Swap at runtime: <code>repo.set_population_engine(new_engine)</code> and <code>repo.set_populate_rules(new_rules)</code> \u2014 the tests exercise both.</li> </ul>"},{"location":"03_use_cases/15_populating_repository_wiring/#intricacy-nested_rules-do-not-round-trip-through-depopulate","title":"\u26a0\ufe0f Intricacy \u2014 <code>nested_rules</code> Do NOT Round-Trip Through <code>depopulate</code>","text":"<p><code>nested_rules</code> are designed for read-side deep population. If the same rules run through the write path (<code>create</code> / <code>update</code>), <code>engine.depopulate</code> behaves differently from plain refs \u2014 verified against the engine:</p> Rule shape What <code>depopulate</code> does to it Effect on the stored doc scalar FK, no <code>nested_rules</code> collapse to <code>.id</code> stored as <code>ObjectId</code> \u2713 <code>list[ObjectId]</code> FK, no <code>nested_rules</code> collapse each item to <code>.id</code> stored as <code>list[ObjectId]</code> \u2713 <code>list</code> FK with <code>nested_rules</code> items are kept as models stored as embedded dicts \u2717 \u2014 reading back raises <code>\"contains embedded dict(s) \u2014 run repair script\"</code> scalar FK with <code>nested_rules</code> the recursion runs, then the field is assigned <code>None</code> reference lost \u2717 \u2014 written as absent/null <p>In other words: a document carrying deep <code>nested_rules</code> (like UC 09's <code>Author \u2192 books \u2192 publisher</code>) cannot be <code>create</code>d / <code>update</code>d as-is. If the graph must be written back, materialize references separately (save each <code>Book</code> to its collection first, then store <code>list[ObjectId]</code> without <code>nested_rules</code>), and keep <code>nested_rules</code> only on read rules you never hand back to <code>depopulate</code>.</p>"},{"location":"03_use_cases/15_populating_repository_wiring/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Never pass a raw <code>ObjectId</code>-holding model to <code>create</code>/<code>update</code>: the \"was populate skipped?\" <code>ValueError</code> is the guard. Read through the repo so the read path can populate first.</li> <li>Unsaved references are a write-once trap: they embed as dicts, and re-reading raises the \"run repair script\" <code>ValueError</code>. Save referenced docs to their collection first, then reference their <code>id</code>.</li> <li><code>filter</code> / <code>projection</code> on <code>PopulateRule</code> are declared but not applied by the engine \u2014 don't rely on them.</li> <li>Reads cost 1 query per reference (no <code>$lookup</code> yet); batch-heavy endpoints should add caching (next use case).</li> </ul>"},{"location":"03_use_cases/15_populating_repository_wiring/#related","title":"Related","text":"<ul> <li>08 \u2013 Population \u00b7 09 \u2013 Advanced population \u00b7 16 \u2013 Cached repository intricacies \u00b7 Components</li> </ul>"},{"location":"03_use_cases/16_cached_repository_intricacies/","title":"Use Case 16: Inside the Cached Repository \u2014 What's Actually Stored & Returned","text":"<p>Scenario: You want the exact contract of <code>CachedBaseRepository</code> \u2014 what goes into the cache, in what shape, and why a cached read occasionally looks \"wrong\" for populated models \u2014 before wiring it into a service.</p>"},{"location":"03_use_cases/16_cached_repository_intricacies/#the-cache-contract","title":"\ud83d\udce6 The Cache Contract","text":"Aspect Value Key <code>\"{key_prefix}{id}\"</code> \u2014 default prefix <code>\"{collection_name}:\"</code> Value <code>json.dumps(model_dump(by_alias=True), default=str)</code> \u2014 a byte string of JSON <code>get_by_id</code> hit <code>decode_value(cached)</code> \u2192 <code>self.model(**data)</code> \u2014 no DB hit <code>get_by_id</code> miss DB read, then the raw doc is cached (<code>model_dump(by_alias=True)</code>) <code>create</code> creates in DB, then caches the result (<code>model_dump</code>) <code>update</code> DB update, then set new value or delete the key if the doc vanished <code>delete</code> DB delete, then removes the key TTL <code>config.default_ttl</code> (default 300 s); in-memory eviction is LRU + expiry <p>So the cache stores JSON snapshots of whole documents \u2014 it is a read-through cache keyed by document id, not a query cache.</p>"},{"location":"03_use_cases/16_cached_repository_intricacies/#intricacy-1-objectids-become-hex-strings","title":"\u26a0\ufe0f Intricacy #1 \u2014 ObjectIds Become Hex Strings","text":"<p><code>json.dumps(..., default=str)</code> stringifies every non-JSON value \u2014 most importantly an <code>ObjectId</code> in a FK field:</p> Python<pre><code># model in memory: User(id=..., profile=ObjectId(\"507f1f77bcf86cd799439011\"))\n# cached bytes: b'{\"_id\":\"507f1f77bcf86cd799439011\",\"profile\":\"507f1f77bcf86cd799439011\", ...}'\n</code></pre> <p>On a cache hit, <code>self.model(**data)</code> must therefore accept a hex string where the raw doc held an <code>ObjectId</code>. This is fine for: - <code>id</code> (typed <code>PyObjectId</code>, which accepts both <code>str</code> and <code>ObjectId</code>), and - <code>created_at</code> / <code>updated_at</code> (ISO strings coerce to <code>datetime</code>).</p> <p>It is not fine for a field typed as a model.</p>"},{"location":"03_use_cases/16_cached_repository_intricacies/#intricacy-2-model-typed-fk-fields-fail-on-a-cache-hit","title":"\u26a0\ufe0f Intricacy #2 \u2014 Model-Typed FK Fields Fail on a Cache Hit","text":"<p>If your document has a populated field, e.g.:</p> Python<pre><code>class User(BaseDocument):\n profile: Profile | None = None # populate-ruled\n</code></pre> <p>then the base-class hit path <code>self.model(**data)</code> receives <code>profile=\"507f\u2026\"</code> and Pydantic raises a <code>ValidationError</code> \u2014 a string cannot coerce into a <code>Profile</code>. The miss path fails the same way: <code>BaseRepository.get_by_id</code> materializes the model from the raw doc whose <code>profile</code> is an <code>ObjectId</code> \u2014 also a <code>ValidationError</code>. So a plain <code>CachedBaseRepository</code> cannot materialize a model-typed FK field at all, hit or miss \u2014 the problem is the base classes build a typed model straight from raw docs.</p> <p>The tension is structural: <code>PopulatingRepository</code> types the field as the model; <code>CachedBaseRepository</code> stores the raw (depopulated) shape. You cannot have a single typed model serve both at once. The two resolutions:</p> <ol> <li>Compose \u2014 keep <code>profile: Profile | None</code> and cache raw, populating on read. Exact recipe in use case 10: its miss path fetches the raw doc (bypassing the base model-build) and both paths normalize <code>str \u2192 ObjectId</code> before populate, because the JSON round-trip hands you strings.</li> <li>Type it as an id \u2014 <code>profile: PyObjectId | None</code> and no populate rules; then cached hits validate cleanly, but you've given up population entirely.</li> </ol> <p>Do not cache a populated model object through the base class: <code>create</code> caches <code>result.model_dump</code>, so the first cache write stores the embedded-dict shape while later <code>get_by_id</code> misses would repopulate \u2014 inconsistent shapes for the same key, and <code>update</code> overwrites with yet another. Pick one canonical raw shape and stick to it.</p>"},{"location":"03_use_cases/16_cached_repository_intricacies/#example-inspecting-what-gets-stored","title":"\ud83d\ude80 Example \u2014 Inspecting What Gets Stored","text":"<p>Python<pre><code>from mongo_ops import BaseDocument, CachedBaseRepository, ModelRegistry, MongoConnectionManager\nfrom mongo_ops.cache import CacheConfig, InMemoryCacheBackend\n\nclass Product(BaseDocument):\n name: str = \"\"\n price: float = 0.0\n\ncache = InMemoryCacheBackend(max_entries=10_000, default_ttl=600)\nModelRegistry.set_cache_backend(cache)\n\nclass ProductRepo(CachedBaseRepository[Product]):\n def __init__(self):\n super().__init__(\"products\", Product, cache, CacheConfig(enabled=True, backend=\"memory\"))\n\nasync def inspect_cache(repo_id: str) -> None:\n # After create()/get_by_id(), inspect what is actually stored:\n cached = await cache.get(f\"products:{repo_id}\")\n # b'{\"_id\":\"507f...\",\"name\":\"Widget\",\"price\":9.99,\"created_at\":\"2026-...\",\"updated_at\":\"2026-...\"}'\n print(cached)\n\n stats = await cache.get_stats()\n print(stats.hits, stats.misses, stats.sets, stats.deletes) # CacheStats dataclass\n</code></pre> ```</p>"},{"location":"03_use_cases/16_cached_repository_intricacies/#intricacy-3-lifecycle-sharing","title":"\u26a0\ufe0f Intricacy #3 \u2014 Lifecycle & Sharing","text":"<ul> <li>One shared backend instance. The repository needs it (<code>cache_backend=cache</code>) and the registry needs it (<code>ModelRegistry.set_cache_backend(cache)</code>) so <code>initialize_cache()</code> / <code>shutdown_cache()</code> manage the same object. Shutdown cancels the in-memory TTL cleanup task \u2014 forgetting it leaks an <code>asyncio.Task</code> at app exit.</li> <li><code>initialize_cache()</code> starts the backend; <code>initialize_all()</code> creates indexes. Both come after <code>connect()</code>.</li> <li><code>warm_cache([ids])</code> skips keys that already exist, fetches the rest from the DB, and returns how many it wrote \u2014 safe to call repeatedly.</li> <li><code>invalidate_cache(id)</code> deletes one key; <code>clear_pattern(\"products:*\")</code> wipes a collection.</li> </ul>"},{"location":"03_use_cases/16_cached_repository_intricacies/#redis-differences","title":"\ud83d\udd04 Redis Differences","text":"Behaviour In-memory Redis (<code>redis.asyncio</code>) Key prefixing prefix baked into the stored key <code>_full_key()</code> applied on every op TTL heap-based, lazy eviction + cleanup task Native <code>SETEX</code> Invalidation broadcast n/a <code>PUBLISH</code> on <code>mongo_ops:cache:invalidate</code> on delete <code>clear_pattern</code> prefix match on stored keys <code>SCAN MATCH</code> in batches"},{"location":"03_use_cases/16_cached_repository_intricacies/#choosing-the-right-layer","title":"\ud83c\udfaf Choosing the Right Layer","text":"Need Use Scalar docs, no refs \u2014 cache-first reads <code>CachedBaseRepository</code> (UC 07, UC 16) Refs resolved on read, no caching <code>PopulatingRepository</code> (UC 08, UC 15) Refs and cache-first reads composed subclass (UC 10) Bulk warm on startup / cache-then-database failover <code>warm_cache</code> + <code>CacheStats</code>"},{"location":"03_use_cases/16_cached_repository_intricacies/#related","title":"Related","text":"<ul> <li>07 \u2013 Caching \u00b7 10 \u2013 Cache + population \u00b7 11 \u2013 Cache lifecycle \u00b7 15 \u2013 PopulatingRepository internals</li> </ul>"},{"location":"03_use_cases/17_populate_depopulate_examples/","title":"17. Populate & Depopulate \u2014 Worked Examples","text":"<p>Scenario: You want to see what populate/depopulate actually do to your documents \u2014 field by field, at every layer (Python model \u2192 depopulated \u2192 stored in Mongo \u2192 raw read back \u2192 populated).</p> <p>These examples all ran against the real <code>mongo_ops</code> engine (<code>mongo_ops/populate/engine.py</code>). Timestamps and <code>ObjectId</code>s are abbreviated with <code>\u2026</code> for readability.</p>"},{"location":"03_use_cases/17_populate_depopulate_examples/#models-rules-used-below","title":"Models & rules used below","text":"Python<pre><code>from bson import ObjectId\nfrom mongo_ops import BaseDocument, PopulatingRepository\nfrom mongo_ops.populate import PopulateRule, PopulationEngine\n\nclass Profile(BaseDocument):\n avatar_url: str = \"\"\n bio: str = \"\"\n\nclass Publisher(BaseDocument):\n name: str = \"\"\n\nclass Book(BaseDocument):\n title: str = \"\"\n publisher: Publisher | None = None # nested ref, reached via nested_rules\n\nclass User(BaseDocument):\n name: str = \"\"\n profile: Profile | None = None # scalar FK \u2014 DB stores an ObjectId\n books: list[Book] | None = None # list FK \u2014 DB stores list[ObjectId]\n\nengine = PopulationEngine({}) # repos keyed by collection name, see UC 15\n\n# A rule WITHOUT nested_rules \u2014 the write path is safe.\nprofile_rule = PopulateRule(field_name=\"profile\", collection_name=\"profiles\")\n\n# A rule WITH nested_rules \u2014 designed for READ-side deep population.\nbooks_rule = PopulateRule(\n field_name=\"books\",\n collection_name=\"books\",\n nested_rules=[PopulateRule(field_name=\"publisher\", collection_name=\"publishers\")],\n)\n</code></pre>"},{"location":"03_use_cases/17_populate_depopulate_examples/#1-write-path-depopulate-turns-models-into-references","title":"1. Write path \u2014 <code>depopulate()</code> turns models into references","text":"<p><code>engine.depopulate(document, rules)</code> walks each rule field and mutates the model in place before you <code>insert_one</code> / <code>find_one_and_update</code> it. What you get on the other side:</p>"},{"location":"03_use_cases/17_populate_depopulate_examples/#1a-scalar-fk-no-nested_rules-stored-as-objectid","title":"1a. Scalar FK, no <code>nested_rules</code> \u2192 stored as <code>ObjectId</code>","text":"Python<pre><code>async def demo():\n profile = Profile(id=ObjectId(\"656\u202604\"), avatar_url=\"a.png\", bio=\"hi\")\n user = User(id=ObjectId(\"656\u202605\"), name=\"Ada\", profile=profile)\n\n await engine.depopulate(user, [profile_rule])\n\n return user.profile # ObjectId('656\u202604') \u2014 was a Profile model\n</code></pre> <p>What Mongo receives (via <code>model_dump</code>, minus <code>_id</code> handling):</p> JSON<pre><code>{ \"name\": \"Ada\", \"profile\": { \"$oid\": \"656\u202604\" } }\n</code></pre>"},{"location":"03_use_cases/17_populate_depopulate_examples/#1b-scalar-fk-with-nested_rules-reference-lost-do-not-do-this","title":"1b. Scalar FK with <code>nested_rules</code> \u2192 reference lost (do not do this)","text":"Python<pre><code>profile_rule_deep = PopulateRule( # same field, but with nested_rules\n field_name=\"profile\",\n collection_name=\"profiles\",\n nested_rules=[PopulateRule(field_name=\"bio\", collection_name=\"profiles\")], # any child field\n)\n\nasync def demo():\n user = User(id=ObjectId(\"656\u202605\"), name=\"Ada\",\n profile=Profile(id=ObjectId(\"656\u202604\"), avatar_url=\"a.png\"))\n await engine.depopulate(user, [profile_rule_deep])\n return user.profile # None \u2014 reference silently dropped!\n</code></pre> <p>Because <code>depopulate</code> recurses into the child but then assigns the parent field <code>None</code> (see <code>engine.py</code> \u2014 <code>depopulated_value</code> is never set on that branch). On save you lose the reference entirely. There is no warning \u2014 the field is just gone.</p>"},{"location":"03_use_cases/17_populate_depopulate_examples/#1c-list-fk-no-nested_rules-stored-as-listobjectid-the-safe-pattern","title":"1c. List FK, no <code>nested_rules</code> \u2192 stored as <code>list[ObjectId]</code> (the safe pattern)","text":"Python<pre><code>async def demo():\n b1 = Book(id=ObjectId(\"656\u202606\"), title=\"MongoDB in Action\")\n b2 = Book(id=ObjectId(\"656\u202607\"), title=\"MongoDB: The Definitive Guide\")\n user = User(id=ObjectId(\"656\u202608\"), name=\"Ada\", books=[b1, b2])\n\n await engine.depopulate(user, [PopulateRule(field_name=\"books\", collection_name=\"books\")])\n\n return user.books # [ObjectId('656\u202606'), ObjectId('656\u202607')] \u2014 clean refs\n</code></pre>"},{"location":"03_use_cases/17_populate_depopulate_examples/#1d-list-fk-with-nested_rules-stored-as-embedded-documents-do-not-do-this","title":"1d. List FK with <code>nested_rules</code> \u2192 stored as embedded documents (do not do this)","text":"Python<pre><code>async def demo():\n publisher = Publisher(id=ObjectId(\"656\u202609\"), name=\"O'Reilly\")\n b1p = Book(id=ObjectId(\"656\u202606\"), title=\"MongoDB in Action\", publisher=publisher)\n user = User(id=ObjectId(\"656\u202608\"), name=\"Ada\", books=[b1p])\n\n await engine.depopulate(user, [books_rule]) # books_rule carries nested_rules\n return user.books[0]\n # <Book id=ObjectId('656\u202606') publisher=ObjectId('656\u202609')>\n</code></pre> <p><code>books</code> stays a list of Book models \u2014 the nested <code>publisher</code> collapsed to <code>ObjectId</code>, but the Book itself was left embedded. Serialized to Mongo you get a nested sub-document:</p> JSON<pre><code>{ \"name\": \"Ada\", \"books\": [ { \"title\": \"MongoDB in Action\", \"publisher\": { \"$oid\": \"656\u202609\" } } ] }\n</code></pre> <p>\u2026and reading that back through population raises <code>ValueError(\u2026contains embedded dict(s) \u2014 run repair script\u2026)</code>, because the engine expects <code>books</code> to hold references, not embedded docs.</p> <p>Why UC 09's <code>Author \u2192 books \u2192 publisher</code> graph can't be written as-is: a rule with <code>nested_rules</code> breaks the write path (1b logs silently, 1d stores embedded). The safe round-trip: save referenced documents first, store <code>list[ObjectId]</code>/<code>ObjectId</code> with a non-nested rule for reads, and keep <code>nested_rules</code> only on rules you never hand back to <code>depopulate</code>.</p>"},{"location":"03_use_cases/17_populate_depopulate_examples/#2-read-path-populate-turns-references-back-into-models","title":"2. Read path \u2014 <code>populate()</code> turns references back into models","text":"<p><code>PopulatingRepository.data_to_model(raw_dict)</code> resolves references before building the model. It first runs <code>_populate(dict)</code> \u2014 replacing the raw <code>ObjectId</code> references with full models via the registered repos \u2014 and only then constructs <code>self.model(**data)</code>, so every FK field is a real model by construction. The two <code>data_to_model</code> examples below (2a/2b) show the underlying resolution <code>_populate</code> performs per rule. (<code>repo</code> below is a <code>PopulatingRepository</code> wired per UC 15 \u2014 <code>books</code> and <code>publishers</code> repos registered in its <code>PopulationEngine</code>.)</p>"},{"location":"03_use_cases/17_populate_depopulate_examples/#2a-scalar-fk-objectid-profile-model","title":"2a. Scalar FK \u2014 <code>ObjectId</code> \u2192 <code>Profile</code> model","text":"Python<pre><code># what came back from Mongo (raw dict, profile is an ObjectId reference):\nraw = {\"_id\": ObjectId(\"656\u202605\"), \"name\": \"Ada\", \"profile\": ObjectId(\"656\u202604\")}\n\nasync def demo():\n user = await repo.data_to_model(raw) # _populate() \u2192 model(**data)\n return user.profile # Profile(id=ObjectId('656\u202604'), avatar_url='a.png')\n</code></pre>"},{"location":"03_use_cases/17_populate_depopulate_examples/#2b-list-fk-nested_rules-two-levels-deep","title":"2b. List FK + <code>nested_rules</code> \u2014 two levels deep","text":"Python<pre><code>raw = {\"_id\": ObjectId(\"656\u202608\"), \"name\": \"Ada\",\n \"books\": [ObjectId(\"656\u202606\"), ObjectId(\"656\u202607\")]}\n\nasync def demo():\n user = await repo.data_to_model(raw) # _populate() resolves both levels\n return (type(user.books[0]).__name__, # 'Book'\n type(user.books[0].publisher).__name__, # 'Publisher' \u2014 nested_rules reached it\n user.books[0].publisher.name) # \"O'Reilly\"\n</code></pre> <p>Walking through one book's resolution:</p> Text Only<pre><code>books_rule \u2192 field \"books\", collection \"books\"\n nested_rules[0].publisher rule \u2192 field \"publisher\", collection \"publishers\"\n1. for ObjectId('656\u202606') \u2192 repo(\"books\").get_by_id(...) \u2192 Book(\"MongoDB in Action\")\n2. book has nested rule \u2192 repo(\"publishers\").get_by_id(...) \u2192 Publisher(\"O'Reilly\")\n3. author.books[0] \u2192 Book(publisher=Publisher(\"O'Reilly\"))\n</code></pre> <p>Per-level cost: 1 query per reference (no <code>$lookup</code>). A hub with 20 hooks and a <code>publisher</code> nested rule is 40 queries \u2014 that is why the next section exists.</p>"},{"location":"03_use_cases/17_populate_depopulate_examples/#3-cache-round-trip-what-actually-goes-into-the-cache","title":"3. Cache round trip \u2014 what actually goes into the cache","text":"<p><code>CachedBaseRepository</code> stores <code>model_dump(by_alias=True)</code> JSON-encoded with <code>default=str</code> (the <code>encode_value</code> helper in <code>mongo_ops/cache/in_memory.py</code>). Crucially it snapshots the raw, just-out-of-Mongo shape \u2014 FK fields still hold <code>ObjectId</code>s \u2014 so the cache holds references, exactly like the DB row:</p> Layer <code>profile</code> field looks like Mongo doc (raw) <code>\u201cprofile\u201d: ObjectId(\"656\u202604\")</code> <code>model_dump(by_alias=True)</code> <code>\"profile\": ObjectId(\"656\u202604\")</code> <code>encode_value(...)</code> <code>\"...\\\"profile\\\": \\\"656\u202604\\\"...\"</code> \u2014 hex string <code>decode_value(...)</code> <code>\"profile\": \"656\u202604\"</code> \u2014 still a hex string <p>Verbatim (timestamps abbreviated):</p> Python<pre><code>from mongo_ops.cache.in_memory import decode_value, encode_value\n\nsnapshot = {\"id\": \"656\u202605\", \"profile\": ObjectId(\"656\u202604\")} # what a cached raw doc looks like\n\ncached = encode_value(snapshot) # json.dumps(snapshot, default=str).encode()\ndecoded = decode_value(cached)\n\ndecoded[\"profile\"] # '656\u202604' \u2014 hex STRING, not ObjectId, not Profile\n</code></pre> <p>Typed-FK caveat: the round trip above only works when the FK field's model type accepts the raw value (<code>str</code>/<code>ObjectId</code> typed). A field typed <code>Profile | None</code> cannot even be materialized from the raw Mongo doc \u2014 both hit and miss raise <code>ValidationError</code> (UC 16's intricacy). That is exactly why UC 10's composed repo overrides <code>get_by_id</code>/<code>create</code>/<code>update</code> to carry raw data through the cache and resolve refs with <code>_populate</code>.</p> <p>Two consequences (the \"intricacies\", detailed in UC 16):</p> <ol> <li>On a cache hit, <code>decode_value</code> gives you hex-string references. A field typed <code>Profile | None</code> receiving a hex string raises Pydantic <code>ValidationError</code> \u2014 so a plain <code>CachedBaseRepository</code> cannot serve model-typed FK fields, hit or miss. The composed pattern in UC 10 re-resolves them in its <code>get_by_id</code>: <code>data = decode_value(await cache.get(key))</code>, then <code>populated = await self._populate(data)</code>, then <code>self.model(**populated)</code>.</li> <li>On a miss, <code>BaseRepository.get_by_id</code> reads the raw Mongo doc whose <code>profile</code> is an <code>ObjectId</code> and immediately raises the same <code>ValidationError</code> \u2014 which is why UC 10 overrides <code>get_by_id</code> to fetch raw, cache raw, then populate. (Real ObjectIds are covered by UC 16's table.)</li> </ol>"},{"location":"03_use_cases/17_populate_depopulate_examples/#4-when-to-use-which","title":"4. When to use which","text":"You need\u2026 Use Scalar <code>ObjectId</code> store, model-struct reads <code>PopulatingRepository</code> (UC 08) Deep nested reads (2+ levels) <code>PopulatingRepository</code> + <code>nested_rules</code> (UC 09) \u2014 read side only Same shape, but cache-first reads UC 10 composed repo (engine + cache + <code>_populate</code>) Scalar docs, no refs, hot <code>get_by_id</code> <code>CachedBaseRepository</code> (UC 07, UC 16) Writing a reference graph back depopulate without <code>nested_rules</code> \u2014 save children first, then point at their ids"},{"location":"03_use_cases/17_populate_depopulate_examples/#related","title":"Related","text":"<ul> <li>08 \u2013 Population \u00b7 09 \u2013 Advanced population \u00b7 10 \u2013 Cache + Population \u00b7 15 \u2013 PopulatingRepository wiring \u00b7 16 \u2013 CachedRepository intricacies</li> </ul>"}]} |