mongo-ops is a modular, high-performance MongoDB operations library for FastAPI microservices. It standardizes connection lifecycle, Pydantic v2 document models, and repository-style async CRUD \u2014 with added layers for caching, reference population, and multi-document transactions. It is built on top of Motor.
Doc model: this wiki is written for humans \u2014 how\u2011to guides, examples, and testing recipes. The authoritative API contracts live in the code (docstrings) and the machine\u2011readable bundle under docs/mcp/.
BaseRepository with generic CRUDBaseDocument)ModelRegistry)cache/)populate/)transactions/)From your internal PyPI:
Bashpip install --extra-index-url https://$PYPI_USERNAME:$PYPI_PASSWORD@pip.aetoskia.com/simple mongo-ops\nWith the extra batteries your workload needs:
Bashpip install --extra-index-url https://$PYPI_USERNAME:$PYPI_PASSWORD@pip.aetoskia.com/simple \"mongo-ops[redis]\" # Redis cache backend\npip install --extra-index-url https://$PYPI_USERNAME:$PYPI_PASSWORD@pip.aetoskia.com/simple \"mongo-ops[all]\" # dev, redis, beanie, fastapi\nFrom local source:
Bashpip install -e .\nexecute_transaction \u00b7 18 \u2013 Aggregation pipelines $group, $lookup, reporting \u00b7 19 \u2013 Bulk operations update_many / delete_many / bulk_write Best Practices Layering, lifecycle, and gotchas Error Handling Library exceptions and FastAPI mapping Testing Mock Motor, engine, cache, transactions \u00b7 Testing example Mock-based quickstart"},{"location":"#related-resources","title":"\ud83d\udd17 Related Resources","text":"\u00a9 Aetoskia Internal \u2014 mongo-ops 0.1.5
mongo-ops gives FastAPI/microservice teams a small, opinionated toolkit for talking to MongoDB asynchronously. This page builds the mental model: what the layers are, in what order they must be wired up, and which repository to reach for.
\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 Application (FastAPI) \u2502\n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n\u2502 \u2502 CachedBase \u2502 Populating \u2502 BaseRepository\u2502 Raw \u2502 \u2502\n\u2502 \u2502 Repository \u2502 Repository \u2502 (CRUDMixin) \u2502 Motor \u2502 \u2502\n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n\u2502 \u2502 \u2502 \u2502 \u2502\n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n\u2502 \u2502 Cache backend\u2502 \u2502 Population\u2502 \u2502 TransactionManager \u2502 \u2502\n\u2502 \u2502 memory/redis \u2502 \u2502 Engine \u2502 \u2502 (session helpers) \u2502 \u2502\n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n\u2502 \u2502 ModelRegistry (models, indexes, cache lifecycle) \u2502 \u2502\n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n\u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n\u2502 \u2502 MongoConnectionManager (Motor client + database) \u2502 \u2502\n\u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u25bc\n MongoDB (Motor / PyMongo)\nThe layers build on each other:
MongoConnectionManager owns the Motor client and active database.BaseDocument (Pydantic v2) gives every document id (aliased _id) plus created_at / updated_at.CRUDMixin / BaseRepository implement generic async CRUD against a collection.ModelRegistry centralizes model + index registration and cache lifecycle for multi-collection services.CachedBaseRepository wraps BaseRepository with a CacheBackend.PopulatingRepository wraps BaseRepository with a PopulationEngine.TransactionManager runs a list of async operations atomically on a session.mongo-ops has a strict startup order. Violating it raises fast, loud exceptions:
await MongoConnectionManager.connect(uri, db_name) Without a connection, get_database()/get_client() raise RuntimeError(\"Database not connected...\"). 2 await ModelRegistry.initialize_all() Create registered indexes (idempotent via create_index). 3 await ModelRegistry.set_cache_backend(...) then await ModelRegistry.initialize_cache() Must happen after connection, before any cache-backed operation. Raises RuntimeError(\"No cache backend registered...\") if skipped. 4 Use repositories Constructed repos resolve the collection from the live database. 5 await ModelRegistry.shutdown_cache() + await MongoConnectionManager.disconnect() On shutdown (in-memory TTL task cancelled; Redis pub/sub closed). The canonical wiring is the FastAPI lifespan context manager (see use case 01).
"},{"location":"01_overview/#which-repository-should-you-use","title":"\ud83e\udded Which Repository Should You Use?","text":"Repository Use when AddsBaseRepository[T] Plain CRUD \u2014 the default nothing extra CachedBaseRepository[T] Read\u2011heavy, low\u2011write fields (lookups by _id) cache\u2011first get_by_id, cache on create, invalidate on update/delete, warm_cache PopulatingRepository[T] You return related docs (FK references) denormalized _populate on read, _depopulate on write, FK\u2011guarded patch CRUDMixin Reusing CRUD inside an existing class raw CRUD against a collection you already have There is no built\u2011in CachedPopulatingRepository. Use case 10 shows how to compose caching + population in a small subclass.
A validated reference to the public API. Signatures match the code in mongo_ops/ exactly. See the use cases for runnable recipes.
Singleton-style owner of the Motor client and active database.
Method Signature Behaviorconnect async (uri: str, db_name: str, **kwargs) -> AsyncIOMotorDatabase Creates AsyncIOMotorClient(uri, **kwargs) and selects the database. Idempotent. disconnect async () -> None Closes the client and clears state. get_database () -> AsyncIOMotorDatabase Returns the active database; raises RuntimeError(\"Database not connected. Call connect() first.\"). get_client () -> AsyncIOMotorClient Returns the active client; raises RuntimeError(\"Client not connected. Call connect() first.\"). lifespan async ctx manager (uri, db_name, **kwargs) connect() on entry, yields the database, disconnect() on exit. Designed for FastAPI lifespan."},{"location":"02_components/#2-basedocument-pyobjectid","title":"2. BaseDocument & PyObjectId","text":"BaseDocument(BaseModel) \u2014 inherit for every Mongo entity.
id: Optional[PyObjectId] \u2014 aliased to _id, serialized to str.created_at: datetime \u2014 defaults to datetime.utcnow().updated_at: datetime \u2014 defaults to datetime.utcnow().Config.populate_by_name = True, arbitrary_types_allowed = True, json_encoders = {ObjectId: str}.PyObjectId(ObjectId) \u2014 Pydantic v2-compatible ObjectId that accepts str or ObjectId and validates with ObjectId.is_valid.
Generic CRUD over a Motor collection. The building block of all repositories.
Method Signature Notescreate async (data: T) -> T Dumps model (excludes id, None), stamps created_at/updated_at, inserts, returns model with assigned _id. get_by_id async (id: str \\| ObjectId) -> Optional[T] str is accepted and converted to ObjectId. get_many async (filter: dict \\| None = None, skip: int = 0, limit: int = 100, sort: list[tuple] \\| None = None) -> list[T] Cursor .skip().limit().sort(...) then to_list(limit). limit=0 disables the limit clause. update async (id, data: dict[str, Any]) -> Optional[T] $set + refreshed updated_at via find_one_and_update. patch async (id, data: dict[str, Any]) -> Optional[T] Same as update but intended for REST PATCH semantics. delete async (id) -> bool True if a document was deleted. count async (filter: dict \\| None = None) -> int count_documents."},{"location":"02_components/#4-baserepositoryt","title":"4. BaseRepository[T]","text":"BaseRepository(collection_name: str, model: type[T]) \u2014 resolves the collection from MongoConnectionManager.get_database()[collection_name]. Requires an active connection at construction time. Provides everything in CRUDMixin plus collection_name.
PopulatingRepository(collection_name, model, population_engine: PopulationEngine | None = None, populate_rules: list[PopulateRule] | None = None).
set_population_engine(engine) / set_populate_rules(rules) \u2014 swap engine/rules at runtime.data_to_model hooks _populate \u2014 get_by_id/get_many return fully populated models._depopulate(document) \u2014 collapses populated FK fields back to ObjectId before create/update.create / update accept a model T (not a dict) so depopulation can run.patch blocks FK fields \u2014 raises ValueError(\"Cannot patch FK fields via patch(): ... Use update() to change FK fields.\").Populate semantics (important): a PopulateRule names a field that holds either an ObjectId or a list[ObjectId] and is the same field that gets replaced with the resolved document(s). There is no separate \"ref field\" vs \"target field\". See use case 08.
start_session async ctx manager (**kwargs) -> AsyncIOMotorClientSession Yields a session with an active transaction. Pass session= to every collection call inside. execute_transaction async (operations: list[Callable[[session], Awaitable[Any]]], **kwargs) -> list[Any] Runs each op inside one transaction and returns results in order; any exception aborts the transaction and propagates."},{"location":"02_components/#7-modelregistry","title":"7. ModelRegistry","text":"Centralized models, indexes, and cache lifecycle for multi-collection services.
Method Signature Behaviorregister (collection_name: str, model: type[BaseDocument], indexes: list[Any] \\| None = None) Records model + index specs. Index specs are passed as-is to pymongo create_index \u2014 single tuples, compound lists, or dicts with keys/options. initialize_all async (db: AsyncIOMotorDatabase \\| None = None) -> None create_index per registered spec (idempotent). Uses the manager database if db omitted. get_model (collection_name) -> type[BaseDocument] Raises KeyError if unregistered. list_collections () -> list[str] Registered collection names. set_cache_backend (backend: CacheBackend) -> None Register the single shared backend. initialize_cache async () -> None Starts the backend (background TTL cleanup) \u2014 raises RuntimeError if no backend registered. shutdown_cache async () -> None Stops the backend cleanly and clears it. get_cache_backend () -> Optional[CacheBackend] Current backend, if any."},{"location":"02_components/#8-cache-layer","title":"8. Cache Layer","text":""},{"location":"02_components/#81-cachebackend-abstract","title":"8.1 CacheBackend (abstract)","text":"Async interface: get(key) -> Optional[bytes], set(key, value: bytes, ttl | None), delete(key), exists(key), clear_pattern(pattern), get_stats() -> CacheStats, initialize(), shutdown(). Values are bytes (JSON-encoded).
Dataclass: hits, misses, sets, deletes, current_size, max_size.
Dataclass: enabled: bool = True, backend: Literal[\"memory\", \"redis\"] = \"memory\", redis_client, default_ttl: int = 300, max_entries: int = 10000, key_prefix: str = \"\", cleanup_interval: int = 60. Raises ValueError if backend=\"redis\" without a client, and ImportError if redis is not installed.
InMemoryCacheBackend(max_entries=10000, default_ttl=300, cleanup_interval=60) \u2014 LRU OrderedDict + TTL heap; initialize() spawns the periodic cleanup task (TTL 0 expires immediately).
RedisCacheBackend(redis_client, key_prefix=\"\", default_ttl=300) \u2014 setex storage, SCAN-based clear_pattern, and publish_invalidate(key) for cross-service invalidation on delete via the mongo_ops:cache:invalidate channel.
CachedBaseRepository(collection_name, model, cache_backend: CacheBackend, config: CacheConfig | None = None).
\"{key_prefix}{id}\" (prefix defaults to \"{collection_name}:\").get_by_id \u2014 cache-first; cache miss reads DB and stores model_dump(by_alias=True) (JSON-encoded) for default_ttl. Honors config.enabled=False (bypass).create \u2014 inserts then caches the result.update/delete \u2014 refresh or remove the cache entry.warm_cache(ids) -> int \u2014 prefetch a list of IDs, returns count warmed.invalidate_cache(id) \u2014 manual eviction.Dataclass:
Python@dataclass\nclass PopulateRule:\n field_name: str # field holding the ObjectId / list[ObjectId]; replaced in-place with the resolved doc(s)\n collection_name: str # collection the references point at\n nested_rules: list[PopulateRule] | None = None\n max_depth: int = 1\n filter: dict | None = None # DECLARED but NOT yet applied by the engine\n projection: dict | None = None # DECLARED but NOT yet applied by the engine\n\u26a0\ufe0f filter and projection are accepted but currently ignored by PopulationEngine \u2014 do not rely on them yet.
PopulationEngine(repos: dict[str, Any], global_max_depth: int = 10) \u2014 repos maps collection_name \u2192 repository.
register_repo(collection_name, repo) \u2014 add repositories at runtime.populate(document, rules, depth=0) \u2014 resolves refs recursively, replacing field_name in place; raises CircularReferenceError(collection, doc_id, path) when a (Class, id) pair is revisited.depopulate(document, rules) \u2014 collapse populated docs back to IDs (for storage).global_max_depth caps recursion; per-rule max_depth bounds a rule's descent.Holds collection, doc_id, and the visited path for debugging cycle messages.
Team-wide conventions for building fast, testable MongoDB services with mongo-ops.
$regex, $inc, projections) inside the repository.BaseDocument for the shape; use Pydantic Field constraints for validation; never put business rules in the model.Service composes multiple repositories (and TransactionManager) \u2014 routes stay thin.get_many(filter=..., skip=..., limit=..., sort=...) instead of raw find for list endpoints \u2014 you get controlled pagination for free.MongoConnectionManager.lifespan(...) (or explicit connect/disconnect) \u2014 never lazily per request.connect(). Module-level Repo() before connection raises RuntimeError(\"Database not connected...\"). Use dependencies or construct inside the lifespan/request.set_cache_backend(backend) \u2192 initialize_cache() (after connect, before use) \u2192 shutdown_cache() on exit.ModelRegistry.register(...) and let initialize_all() create indexes once at startup (idempotent).ModelRegistry.register \u2014 see use case 13._id reads. Use CachedBaseRepository for lookups-by-id; invalidate (update/delete handle it) and pick a sensible default_ttl.PopulatingRepository resolves refs on read and depopulates on write; do not hand-roll joins in endpoints.patch() cannot touch FK fields \u2014 use update().TransactionManager.start_session (inline) or execute_transaction (list of ops) \u2014 and pass session= to every collection call inside.DuplicateKeyError \u2192 409, InvalidId \u2192 400, CircularReferenceError \u2192 409, ValueError guides \u2192 400/422 (see Error Handling).MongoConnectionManager.get_database, use AsyncMock collections and cursor chains \u2014 the whole suite runs without MongoDB (use case 14).tests/test_{repository,populating_repository,cache,registry,transactions}.py are canonical examples of every pattern above.What can raise, what it means, and how to map it in a FastAPI app.
"},{"location":"06_error_handling/#library-raised-exceptions","title":"\ud83d\udccb Library-Raised Exceptions","text":"Exception Source Meaning / fixRuntimeError(\"Database not connected. Call connect() first.\") get_database() / get_client() and any repository constructed first MongoConnectionManager.connect() hasn't run \u2014 wire the lifespan. RuntimeError(\"No cache backend registered. Call set_cache_backend() first.\") ModelRegistry.initialize_cache() Call set_cache_backend(backend) before initialize_cache(). KeyError(\"Model for collection '...' not registered\") ModelRegistry.get_model() Collection was never registered (or typo). ValueError(\"Cannot patch FK fields via patch(): ...\") PopulatingRepository.patch() patch must not touch populated ref fields \u2014 use update() with a model. ValueError(\"...contains embedded dict(s) \u2014 run repair script\") _populate on read A FK field holds an embedded document instead of an ObjectId \u2014 migrate the data. ValueError(\"...contains ObjectId \u2014 was populate skipped?\") _depopulate on write A populate-ruled field is still an ObjectId at depopulate time \u2014 the read must have populated it first. CircularReferenceError(collection, doc_id, path) PopulationEngine.populate A (Class, id) pair was revisited \u2014 raise max_depth or fix the graph. ImportError(\"redis package required ... mongo-ops[redis]\") CacheConfig / RedisCacheBackend Missing redis extra. ValueError(\"redis_client required when backend='redis'\") CacheConfig backend=\"redis\" without a client. Pymongo DuplicateKeyError any insert/update Unique index violation (e.g., duplicate email). bson.errors.InvalidId ObjectId(...) on a bad string Wrapped by PyObjectId model validation on API inputs. KeyError for ModelRegistry.get_model and the RuntimeError/ValueError guards are by design \u2014 they fail loudly at startup or first call instead of misbehaving silently.
from fastapi import FastAPI, HTTPException, Request\nfrom fastapi.responses import JSONResponse\nfrom bson.errors import InvalidId\nfrom pymongo.errors import DuplicateKeyError\nfrom mongo_ops.cache import CircularReferenceError\n\n\napp = FastAPI()\n\n\n@app.exception_handler(DuplicateKeyError)\nasync def duplicate_key_handler(_: Request, __: DuplicateKeyError) -> JSONResponse:\n return JSONResponse(status_code=409, content={\"detail\": \"Resource already exists\"})\n\n\n@app.exception_handler(InvalidId)\nasync def invalid_id_handler(_: Request, __: InvalidId) -> JSONResponse:\n return JSONResponse(status_code=400, content={\"detail\": \"Invalid ID format\"})\n\n\n@app.exception_handler(CircularReferenceError)\nasync def circular_ref_handler(_: Request, exc: CircularReferenceError) -> JSONResponse:\n return JSONResponse(\n status_code=409,\n content={\"detail\": f\"Circular reference detected: {exc.path}\"},\n )\n\n\n# Or handle inline for route-specific responses:\n@app.post(\"/users/\")\nasync def create_user(user: User, repo=Depends(get_user_repository)):\n try:\n return await repo.create(user)\n except DuplicateKeyError:\n raise HTTPException(status_code=409, detail=\"User already exists\")\n except Exception as exc:\n raise HTTPException(status_code=500, detail=str(exc))\n\n\n@app.get(\"/users/{user_id}\")\nasync def get_user(user_id: str, repo=Depends(get_user_repository)):\n try:\n user = await repo.get_by_id(user_id)\n except InvalidId:\n raise HTTPException(status_code=400, detail=\"Invalid user ID\")\n if not user:\n raise HTTPException(status_code=404, detail=\"User not found\")\n return user\nPrefer exception handlers for the library-level exceptions (409/400 above) and per-endpoint try/except for domain decisions (404).
ValueError populate guards are your friends: they surface data-shape drift (embedded docs, skipped population) at the exact call site.CircularReferenceError.path \u2014 it prints the visited Class:id chain.RuntimeErrors at startup; let the app fail to load so the misconfiguration is obvious.A zero-MongoDB unit-test quickstart. The full guide is use case 14.
"},{"location":"07_testing_example/#mock-based-quickstart","title":"\ud83d\ude80 Mock-Based Quickstart","text":"Patch the connection manager and stub the collection \u2014 no network, no docker.
Pythonimport pytest\nfrom unittest.mock import AsyncMock, patch\n\nfrom bson import ObjectId\nfrom mongo_ops import BaseDocument, BaseRepository\nfrom mongo_ops.repository import MongoConnectionManager\n\n\nclass User(BaseDocument):\n username: str = \"\"\n email: str = \"\"\n\n\nclass UserRepository(BaseRepository[User]):\n def __init__(self):\n super().__init__(\"users\", User)\n\n\n@pytest.fixture\ndef repo():\n mock_collection = AsyncMock()\n with patch.object(MongoConnectionManager, \"get_database\") as mock_db:\n mock_db.return_value.__getitem__.return_value = mock_collection\n r = UserRepository()\n r.collection = mock_collection\n return r\n\n\n@pytest.mark.asyncio\nasync def test_create_user(repo):\n mock_collection = repo.collection\n oid = ObjectId()\n mock_collection.insert_one.return_value.inserted_id = oid\n\n user = await repo.create(User(username=\"test\", email=\"test@example.com\"))\n\n assert user.id == oid\n assert user.username == \"test\"\n\n\n@pytest.mark.asyncio\nasync def test_get_by_id(repo):\n oid = ObjectId()\n repo.collection.find_one.return_value = {\n \"_id\": oid,\n \"username\": \"test\",\n \"email\": \"test@example.com\",\n \"created_at\": \"2024-01-01T00:00:00\",\n \"updated_at\": \"2024-01-01T00:00:00\",\n }\n\n fetched = await repo.get_by_id(str(oid))\n\n assert fetched is not None\n assert fetched.username == \"test\"\nfrom mongo_ops.repository import MongoConnectionManager \u2014 patch where it is used (mongo_ops.repository.MongoConnectionManager), matching the library's own tests.auto mode per pyproject.toml, so @pytest.mark.asyncio works without extra config.Scenario: A simple user management API with CRUD endpoints, index registration, and a correct connection lifecycle.
"},{"location":"03_use_cases/01_basic_crud/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component DescriptionMongoConnectionManager.lifespan Connect on startup, disconnect on shutdown. BaseRepository[User] Generic CRUD \u2014 no endpoint-level Mongo code. ModelRegistry.register Declares the email index; initialize_all creates it at startup."},{"location":"03_use_cases/01_basic_crud/#example","title":"\ud83d\ude80 Example","text":"Pythonimport os\nfrom contextlib import asynccontextmanager\n\nfrom fastapi import Depends, FastAPI, HTTPException\nfrom pydantic import Field\n\nfrom mongo_ops import BaseDocument, BaseRepository, ModelRegistry, MongoConnectionManager\n\n# ---------------------------\n# Model\n# ---------------------------\nclass User(BaseDocument):\n username: str = Field(..., min_length=3, max_length=50)\n email: str = Field(...)\n is_active: bool = True\n\n\n# ---------------------------\n# Repository\n# ---------------------------\nclass UserRepository(BaseRepository[User]):\n def __init__(self):\n super().__init__(\"users\", User)\n\n\n# ---------------------------\n# Registration + index\n# ---------------------------\nModelRegistry.register(\"users\", User, indexes=[(\"email\", 1)])\n\n\n# ---------------------------\n# Lifecycle\n# ---------------------------\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=os.getenv(\"MONGO_URI\", \"mongodb://localhost:27017\"),\n db_name=\"mydb\",\n ):\n await ModelRegistry.initialize_all()\n yield\n\n\napp = FastAPI(lifespan=lifespan)\n\n\n# ---------------------------\n# Dependency \u2014 created per request, after connect()\n# ---------------------------\ndef get_user_repository() -> UserRepository:\n return UserRepository()\n\n\n# ---------------------------\n# Routes\n# ---------------------------\n@app.post(\"/users/\", response_model=User)\nasync def create_user(user: User, repo: UserRepository = Depends(get_user_repository)):\n return await repo.create(user)\n\n\n@app.get(\"/users/{user_id}\", response_model=User)\nasync def get_user(user_id: str, repo: UserRepository = Depends(get_user_repository)):\n user = await repo.get_by_id(user_id)\n if not user:\n raise HTTPException(status_code=404, detail=\"User not found\")\n return user\n\n\n@app.get(\"/users/\", response_model=list[User])\nasync def list_users(\n skip: int = 0,\n limit: int = 10,\n repo: UserRepository = Depends(get_user_repository),\n):\n return await repo.get_many(skip=skip, limit=limit)\n\n\n@app.put(\"/users/{user_id}\", response_model=User)\nasync def update_user(\n user_id: str,\n email: str,\n repo: UserRepository = Depends(get_user_repository),\n):\n user = await repo.update(user_id, {\"email\": email})\n if not user:\n raise HTTPException(status_code=404, detail=\"User not found\")\n return user\n\n\n@app.delete(\"/users/{user_id}\")\nasync def delete_user(user_id: str, repo: UserRepository = Depends(get_user_repository)):\n deleted = await repo.delete(user_id)\n if not deleted:\n raise HTTPException(status_code=404, detail=\"User not found\")\n return {\"message\": \"User deleted successfully\"}\nUserRepository() is created inside the dependency, i.e., only after the lifespan has connected. Instantiating a repository at module import time raises RuntimeError because the database isn't connected yet.BaseRepository.update(id, {...}) takes a dict; only PopulatingRepository.update takes a model.patch() for REST PATCH semantics \u2014 it accepts a partial dict like update, but PopulatingRepository rejects FK fields.Scenario: An e-commerce product catalog needs search, filtering, and stock updates without Mongo leaking into routes.
"},{"location":"03_use_cases/02_custom_repo/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Description Repository methods Encapsulate queries ($regex, filters, $inc) behind domain methods. get_many Filtering + default pagination via the base repository. Direct collection access For operations with no base-repo helper (regex search, atomic $inc)."},{"location":"03_use_cases/02_custom_repo/#example","title":"\ud83d\ude80 Example","text":"Pythonfrom fastapi import FastAPI, HTTPException, Query\nfrom mongo_ops import BaseDocument, BaseRepository\n\n# ---------------------------\n# Model\n# ---------------------------\nclass Product(BaseDocument):\n name: str\n description: str = \"\"\n price: float = 0.0\n category: str = \"\"\n in_stock: bool = True\n quantity: int = 0\n tags: list[str] = []\n\n\n# ---------------------------\n# Repository\n# ---------------------------\nclass ProductRepository(BaseRepository[Product]):\n def __init__(self):\n super().__init__(\"products\", Product)\n\n async def search_by_name(self, query: str) -> list[Product]:\n \"\"\"Case-insensitive name search.\"\"\"\n docs = await self.collection.find(\n {\"name\": {\"$regex\": query, \"$options\": \"i\"}}\n ).to_list(length=100)\n return [self.model(**doc) for doc in docs]\n\n async def get_by_category(self, category: str, in_stock_only: bool = True) -> list[Product]:\n filter_query = {\"category\": category}\n if in_stock_only:\n filter_query[\"in_stock\"] = True\n return await self.get_many(filter=filter_query)\n\n async def get_low_stock(self, threshold: int = 10) -> list[Product]:\n return await self.get_many(filter={\"quantity\": {\"$lt\": threshold}, \"in_stock\": True})\n\n async def update_stock(self, product_id: str, quantity_delta: int) -> Product | None:\n \"\"\"Atomically increment/decrement stock.\"\"\"\n from bson import ObjectId\n from datetime import datetime\n\n result = await self.collection.find_one_and_update(\n {\"_id\": ObjectId(product_id)},\n {\"$inc\": {\"quantity\": quantity_delta}, \"$set\": {\"updated_at\": datetime.utcnow()}},\n return_document=True,\n )\n return self.model(**result) if result else None\n\n\napp = FastAPI()\nproduct_repo = ProductRepository()\n\n\n@app.get(\"/products/search\", response_model=list[Product])\nasync def search_products(q: str = Query(..., min_length=1)):\n return await product_repo.search_by_name(q)\n\n\n@app.get(\"/products/category/{category}\", response_model=list[Product])\nasync def products_by_category(category: str, in_stock: bool = True):\n return await product_repo.get_by_category(category, in_stock)\n\n\n@app.get(\"/products/low-stock\", response_model=list[Product])\nasync def low_stock_products(threshold: int = 10):\n return await product_repo.get_low_stock(threshold)\n\n\n@app.patch(\"/products/{product_id}/stock\")\nasync def update_product_stock(product_id: str, quantity_delta: int):\n product = await product_repo.update_stock(product_id, quantity_delta)\n if not product:\n raise HTTPException(status_code=404, detail=\"Product not found\")\n return product\nNote: this snippet omits the FastAPI lifespan connection wiring for brevity \u2014 copy it from use case 01 so ProductRepository() is created only after MongoConnectionManager.connect().
self.collection directly (regex search, $inc) bypass the caching and population layers. If a feature composes them \u2014 extend CachedBaseRepository or PopulatingRepository instead and add the domain methods there.get_many(filter=...) over raw find when you want pagination/sort defaults for free.self.model(**doc) to convert raw dicts to model instances consistently.Scenario: Order processing must update inventory and create an order atomically. Any failure rolls both back.
"},{"location":"03_use_cases/03_transactions/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component DescriptionTransactionManager.start_session Async context manager yielding a session with an active transaction. session= kwarg Pass to every insert_one / update_one / find_one inside the block."},{"location":"03_use_cases/03_transactions/#example","title":"\ud83d\ude80 Example","text":"Pythonfrom contextlib import asynccontextmanager\nfrom datetime import datetime\n\nfrom bson import ObjectId\nfrom fastapi import FastAPI, HTTPException\nfrom mongo_ops import BaseDocument, BaseRepository, MongoConnectionManager, TransactionManager\n\n\nclass Order(BaseDocument):\n user_id: str\n items: list[dict] # [{\"product_id\": \"...\", \"quantity\": 2}]\n total_amount: float = 0.0\n status: str = \"pending\"\n\n\nclass Inventory(BaseDocument):\n product_id: ObjectId = None\n quantity: int = 0\n\n\nclass OrderRepository(BaseRepository[Order]):\n def __init__(self):\n super().__init__(\"orders\", Order)\n\n\nclass InventoryRepository(BaseRepository[Inventory]):\n def __init__(self):\n super().__init__(\"inventory\", Inventory)\n\n\nasync def create_order_with_inventory_update(\n order: Order,\n order_repo: OrderRepository,\n inv_repo: InventoryRepository,\n) -> Order:\n async def insert_order(session):\n doc = order.model_dump(exclude={\"id\"}, exclude_none=True)\n doc[\"created_at\"] = doc[\"updated_at\"] = datetime.utcnow()\n result = await order_repo.collection.insert_one(doc, session=session)\n return await order_repo.collection.find_one({\"_id\": result.inserted_id}, session=session)\n\n async def update_inventory(session):\n for item in order.items:\n await inv_repo.collection.update_one(\n {\"product_id\": ObjectId(item[\"product_id\"])},\n {\n \"$inc\": {\"quantity\": -item[\"quantity\"]},\n \"$set\": {\"updated_at\": datetime.utcnow()},\n },\n session=session,\n )\n return None\n\n async with TransactionManager.start_session() as session:\n created = await insert_order(session)\n await update_inventory(session)\n return Order(**created)\n\n\n# ---------------------------\n# FastAPI wiring\n# ---------------------------\napp = FastAPI()\n\norder_repo = OrderRepository()\ninv_repo = InventoryRepository()\n\n\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"shop\"\n ):\n yield\n\n\napp.lifespan = lifespan\n\n\n@app.post(\"/orders/\", response_model=Order)\nasync def create_order(order: Order):\n try:\n return await create_order_with_inventory_update(order, order_repo, inv_repo)\n except Exception as exc:\n raise HTTPException(status_code=400, detail=str(exc))\nsession=<session> \u2014 a missing session silently runs outside the transaction.TransactionManager.execute_transaction in use case 12.Scenario: A blog post API lists posts with page metadata, filtering, and sorting.
"},{"location":"03_use_cases/04_pagination/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component DescriptionBaseRepository.get_many filter, skip, limit, sort in one call. BaseRepository.count Total matching documents for page metadata. PaginatedResponse Reusable generic response wrapper."},{"location":"03_use_cases/04_pagination/#example","title":"\ud83d\ude80 Example","text":"Pythonfrom typing import Generic, TypeVar\n\nfrom fastapi import FastAPI, Query\nfrom pydantic import BaseModel\nfrom mongo_ops import BaseDocument, BaseRepository\n\nT = TypeVar(\"T\")\n\n\nclass PaginatedResponse(BaseModel, Generic[T]):\n items: list[T]\n total: int\n page: int\n page_size: int\n total_pages: int\n has_next: bool\n has_prev: bool\n\n\nclass BlogPost(BaseDocument):\n title: str = \"\"\n content: str = \"\"\n author_id: str = \"\"\n published: bool = False\n tags: list[str] = []\n views: int = 0\n\n\nclass BlogPostRepository(BaseRepository[BlogPost]):\n def __init__(self):\n super().__init__(\"blog_posts\", BlogPost)\n\n async def paginate(\n self,\n page: int = 1,\n page_size: int = 10,\n filter_dict: dict | None = None,\n sort_by: str = \"created_at\",\n sort_order: int = -1,\n ) -> PaginatedResponse[BlogPost]:\n filter_dict = filter_dict or {}\n skip = (page - 1) * page_size\n total = await self.count(filter_dict)\n items = await self.get_many(\n filter=filter_dict,\n skip=skip,\n limit=page_size,\n sort=[(sort_by, sort_order)],\n )\n total_pages = (total + page_size - 1) // page_size\n return PaginatedResponse(\n items=items,\n total=total,\n page=page,\n page_size=page_size,\n total_pages=total_pages,\n has_next=page < total_pages,\n has_prev=page > 1,\n )\n\n async def get_by_author(self, author_id: str, published_only: bool = True) -> list[BlogPost]:\n filter_dict = {\"author_id\": author_id}\n if published_only:\n filter_dict[\"published\"] = True\n return await self.get_many(filter=filter_dict, sort=[(\"created_at\", -1)])\n\n async def search_by_tags(self, tags: list[str]) -> list[BlogPost]:\n return await self.get_many(filter={\"tags\": {\"$in\": tags}, \"published\": True})\n\n\napp = FastAPI()\nblog_repo = BlogPostRepository()\n\n\n@app.get(\"/posts/\", response_model=PaginatedResponse[BlogPost])\nasync def list_posts(\n page: int = 1,\n page_size: int = 10,\n published: bool | None = None,\n author_id: str | None = None,\n):\n filter_dict = {}\n if published is not None:\n filter_dict[\"published\"] = published\n if author_id:\n filter_dict[\"author_id\"] = author_id\n return await blog_repo.paginate(page, page_size, filter_dict)\n\n\n@app.get(\"/posts/author/{author_id}\", response_model=list[BlogPost])\nasync def posts_by_author(author_id: str, published: bool = True):\n return await blog_repo.get_by_author(author_id, published)\n\n\n@app.get(\"/posts/tags\", response_model=list[BlogPost])\nasync def posts_by_tags(tags: list[str] = Query(...)):\n return await blog_repo.search_by_tags(tags)\nNote: add the lifespan wiring from use case 01 so blog_repo is created after connection.
get_many calls cursor.to_list(limit); pass limit=0 to skip the limit entirely, otherwise a large explicit limit is safer than unbounded reads.count first for stable metadata \u2014 it uses the same filter as the page query.sort=[(field, -1|1)] for deterministic ordering; create matching indexes to avoid full collection scans (see use case 13).Scenario: Deleting a task should be recoverable. Instead of removing the document, set a tombstone flag and filter it from normal queries.
"},{"location":"03_use_cases/05_soft_deletes/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component DescriptionSoftDeleteDocument Base model carrying is_deleted, deleted_at, deleted_by. SoftDeleteRepository[T] Repository-level soft delete / restore / filtering using update + get_many. permanent_delete Escapes to the real delete."},{"location":"03_use_cases/05_soft_deletes/#example","title":"\ud83d\ude80 Example","text":"Pythonfrom datetime import datetime\n\nfrom fastapi import FastAPI, HTTPException\nfrom mongo_ops import BaseDocument, BaseRepository\n\n\nclass SoftDeleteDocument(BaseDocument):\n is_deleted: bool = False\n deleted_at: datetime | None = None\n deleted_by: str | None = None\n\n\nclass Task(SoftDeleteDocument):\n title: str = \"\"\n description: str = \"\"\n assignee_id: str = \"\"\n status: str = \"pending\"\n priority: str = \"medium\"\n\n\nclass SoftDeleteRepository(BaseRepository[T]):\n async def soft_delete(self, id: str, deleted_by: str | None = None):\n return await self.update(\n id,\n {\"is_deleted\": True, \"deleted_at\": datetime.utcnow(), \"deleted_by\": deleted_by},\n )\n\n async def restore(self, id: str):\n return await self.update(\n id, {\"is_deleted\": False, \"deleted_at\": None, \"deleted_by\": None}\n )\n\n async def get_active(self, skip: int = 0, limit: int = 100):\n return await self.get_many(filter={\"is_deleted\": False}, skip=skip, limit=limit)\n\n async def get_deleted(self, skip: int = 0, limit: int = 100):\n return await self.get_many(filter={\"is_deleted\": True}, skip=skip, limit=limit)\n\n async def permanent_delete(self, id: str) -> bool:\n return await self.delete(id)\n\n\nclass TaskRepository(SoftDeleteRepository[Task]):\n def __init__(self):\n super().__init__(\"tasks\", Task)\n\n\napp = FastAPI()\ntask_repo = TaskRepository()\n\n\n@app.delete(\"/tasks/{task_id}\")\nasync def soft_delete_task(task_id: str, user_id: str):\n task = await task_repo.soft_delete(task_id, deleted_by=user_id)\n if not task:\n raise HTTPException(status_code=404, detail=\"Task not found\")\n return {\"message\": \"Task deleted\", \"task\": task}\n\n\n@app.post(\"/tasks/{task_id}/restore\")\nasync def restore_task(task_id: str):\n task = await task_repo.restore(task_id)\n if not task:\n raise HTTPException(status_code=404, detail=\"Task not found\")\n return {\"message\": \"Task restored\", \"task\": task}\n\n\n@app.get(\"/tasks/\", response_model=list[Task])\nasync def list_active_tasks(skip: int = 0, limit: int = 10):\n return await task_repo.get_active(skip, limit)\n\n\n@app.get(\"/tasks/deleted\", response_model=list[Task])\nasync def list_deleted_tasks(skip: int = 0, limit: int = 10):\n return await task_repo.get_deleted(skip, limit)\nNote: add the lifespan wiring from use case 01 so task_repo is created after connection.
BaseRepository[T] is generic \u2014 subclasses must parameterize it (see SoftDeleteRepository[T] above). from mongo_ops import BaseRepository; BaseRepository[T] works out of the box.{\"is_deleted\": 1, \"status\": 1} for active-list queries (see use case 13).Scenario: A social app manages users, posts, and comments. Each has its own model, repository, and indexes \u2014 registered centrally and initialized at startup.
ModelRegistry.register One call per collection \u2014 model + indexes together. ModelRegistry.initialize_all Creates every registered index at startup (idempotent). Repositories One repository class per collection, all sharing the same connection."},{"location":"03_use_cases/06_multi_model/#example","title":"\ud83d\ude80 Example","text":"Pythonfrom contextlib import asynccontextmanager\n\nfrom fastapi import FastAPI\nfrom mongo_ops import BaseDocument, BaseRepository, ModelRegistry, MongoConnectionManager\n\n\nclass User(BaseDocument):\n username: str = \"\"\n email: str = \"\"\n role: str = \"user\"\n\n\nclass Post(BaseDocument):\n title: str = \"\"\n content: str = \"\"\n author_id: str = \"\"\n likes: int = 0\n\n\nclass Comment(BaseDocument):\n post_id: str = \"\"\n user_id: str = \"\"\n content: str = \"\"\n\n\nclass UserRepository(BaseRepository[User]):\n def __init__(self):\n super().__init__(\"users\", User)\n\n\nclass PostRepository(BaseRepository[Post]):\n def __init__(self):\n super().__init__(\"posts\", Post)\n\n\nclass CommentRepository(BaseRepository[Comment]):\n def __init__(self):\n super().__init__(\"comments\", Comment)\n\n\n# ---------------------------\n# Central registration + indexes\n# ---------------------------\nModelRegistry.register(\"users\", User, indexes=[(\"email\", 1), (\"username\", 1)])\nModelRegistry.register(\"posts\", Post, indexes=[(\"author_id\", 1), (\"created_at\", -1)])\nModelRegistry.register(\"comments\", Comment, indexes=[(\"post_id\", 1), (\"user_id\", 1)])\n\n\n# ---------------------------\n# Lifecycle\n# ---------------------------\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"social_app\"\n ):\n await ModelRegistry.initialize_all()\n yield\n\n\napp = FastAPI(lifespan=lifespan)\n\n\n# Created after connect() \u2014 inside the lifespan body is fine, or use a dependency.\nuser_repo = UserRepository()\npost_repo = PostRepository()\ncomment_repo = CommentRepository()\n\n\n@app.post(\"/users/\", response_model=User)\nasync def create_user(user: User):\n return await user_repo.create(user)\n\n\n@app.post(\"/posts/\", response_model=Post)\nasync def create_post(post: Post):\n return await post_repo.create(post)\n\n\n@app.post(\"/comments/\", response_model=Comment)\nasync def create_comment(comment: Comment):\n return await comment_repo.create(comment)\nuser_repo = ... inside the lifespan/after connect. Module-level instantiation before connect() raises RuntimeError(\"Database not connected...\").ObjectId strings stored on the child docs. To resolve them on read, see use case 08 \u2013 Population.created_at/author_id/post_id keeps the common queries indexed (see use case 13).Scenario: A read-heavy API (product catalog) reduces DB load by caching documents by _id \u2014 in-memory locally, or shared via Redis.
InMemoryCacheBackend TTL + LRU cache with a background cleanup task. RedisCacheBackend Distributed cache on redis.asyncio with pub/sub invalidation. CachedBaseRepository[T] Extends BaseRepository \u2014 cache-first get_by_id, cache on create, invalidate on update/delete, warm_cache(ids), invalidate_cache(id). Backend lifecycle The same backend instance must be both passed to the repository AND registered via ModelRegistry.set_cache_backend so initialize_cache() starts its task."},{"location":"03_use_cases/07_caching/#example","title":"\ud83d\ude80 Example","text":"Pythonfrom contextlib import asynccontextmanager\n\nfrom fastapi import FastAPI, HTTPException\nfrom mongo_ops import BaseDocument, CachedBaseRepository, ModelRegistry, MongoConnectionManager\nfrom mongo_ops.cache import CacheConfig, InMemoryCacheBackend\n\n\nclass Product(BaseDocument):\n name: str = \"\"\n price: float = 0.0\n\n\n# One shared backend \u2014 used by both the repository and the registry lifecycle.\ncache = InMemoryCacheBackend(max_entries=10_000, default_ttl=300)\nModelRegistry.set_cache_backend(cache)\n\n\nclass ProductRepo(CachedBaseRepository[Product]):\n def __init__(self):\n super().__init__(\n collection_name=\"products\",\n model=Product,\n cache_backend=cache,\n config=CacheConfig(enabled=True, backend=\"memory\"),\n )\n\n\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"shop\"\n ):\n await ModelRegistry.initialize_all()\n await ModelRegistry.initialize_cache() # starts the TTL cleanup task\n yield\n await ModelRegistry.shutdown_cache() # cancels it on exit\n\n\napp = FastAPI(lifespan=lifespan)\n\n\n@app.post(\"/products/\", response_model=Product)\nasync def create_product(product: Product):\n return await ProductRepo().create(product) # created after connect()\n\n\n@app.get(\"/products/{product_id}\", response_model=Product)\nasync def get_product(product_id: str):\n product = await ProductRepo().get_by_id(product_id)\n if not product:\n raise HTTPException(status_code=404, detail=\"Product not found\")\n return product\n\n\n@app.put(\"/products/{product_id}\", response_model=Product)\nasync def update_product(product_id: str, name: str | None = None, price: float | None = None):\n data = {}\n if name is not None:\n data[\"name\"] = name\n if price is not None:\n data[\"price\"] = price\n return await ProductRepo().update(product_id, data)\n\n\n# Outside request handlers:\n# await ProductRepo().warm_cache([object_id_1, object_id_2]) # pre-load n ids -> int\n# await ProductRepo().invalidate_cache(object_id_3) # manual eviction\nSwap the backend \u2014 the repository code stays identical:
Pythonfrom mongo_ops.cache.redis_backend import RedisCacheBackend\nfrom redis.asyncio import Redis\n\nredis_client = Redis(host=\"localhost\", port=6379)\nredis_backend = RedisCacheBackend(redis_client, key_prefix=\"prod:\")\n\nModelRegistry.set_cache_backend(redis_backend) # for the lifecycle in lifespan()\n\nclass ProductRepo(CachedBaseRepository[Product]):\n def __init__(self):\n super().__init__(\n collection_name=\"products\",\n model=Product,\n cache_backend=redis_backend,\n config=CacheConfig(enabled=True, backend=\"redis\"),\n )\n\"{key_prefix}{id}\" (default prefix \"products:\"). clear_pattern(\"products:*\") wipes a whole collection's entries.json.dumps(obj, default=str)) \u2014 nested models inside a cached document are stored as dicts, not objects.CacheConfig(enabled=False); a disabled repo bypasses the cache entirely.initialize* calls: initialize_cache() starts the backend task; initialize_all() creates indexes. Both belong in the lifespan, after connect().Scenario: An API returns a User with its related Profile embedded in one JSON payload \u2014 no second round-trip from the client, no joins.
PopulateRule Declares which field to resolve. field_name holds the ObjectId (or list[ObjectId]) and is the same field the resolved document replaces in place. collection_name says where the referenced documents live. PopulationEngine Holds repositories by collection name and resolves rules recursively, detecting cycles (CircularReferenceError). PopulatingRepository[T] get_by_id/get_many populate on read; create/update depopulate on write; patch rejects FK fields. \u26a0\ufe0f PopulateRule does not have a separate \"ref field\" vs \"target field\" \u2014 the ref field is the populated field. filter/projection on PopulateRule are declared but not yet applied by the engine.
from contextlib import asynccontextmanager\n\nfrom fastapi import FastAPI, HTTPException\nfrom mongo_ops import BaseDocument, MongoConnectionManager, ModelRegistry, PopulatingRepository\nfrom mongo_ops.populate import PopulateRule, PopulationEngine\n\n\n# 1. Models \u2014 `profile` holds an ObjectId in the DB and becomes a Profile on read.\nclass Profile(BaseDocument):\n avatar_url: str = \"\"\n bio: str = \"\"\n\n\nclass User(BaseDocument):\n username: str = \"\"\n email: str = \"\"\n profile: Profile | None = None # ObjectId in MongoDB, Profile in memory\n\n\n# 2. Engine + rule \u2014 repositories are registered later (after connect()).\nengine = PopulationEngine({})\nprofile_rule = PopulateRule(\n field_name=\"profile\",\n collection_name=\"profiles\",\n)\n\n\n# 3. The repository the app uses.\nclass UserRepository(PopulatingRepository[User]):\n def __init__(self):\n super().__init__(\n collection_name=\"users\",\n model=User,\n population_engine=engine,\n populate_rules=[profile_rule],\n )\n\n\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"mydb\"\n ):\n # Tells the engine which repository resolves \"profiles\" refs.\n engine.register_repo(\"profiles\", PopulatingRepository[Profile](\"profiles\", Profile))\n await ModelRegistry.initialize_all()\n yield\n\n\napp = FastAPI(lifespan=lifespan)\n\n\n# 4. Writing \u2014 pass the model; the repository depopulates to an ObjectId.\n@app.post(\"/users/\", response_model=User)\nasync def create_user():\n user = User(\n username=\"alice\",\n email=\"alice@example.com\",\n profile=Profile(avatar_url=\"alice.png\", bio=\"hi\"),\n )\n return await UserRepository().create(user)\n # ^ stored as {\"profile\": <ObjectId>}, returned with profile populated\n\n\n# 5. Reading \u2014 get_by_id resolves the reference on the way back.\n@app.get(\"/users/{user_id}\", response_model=User)\nasync def get_user(user_id: str):\n user = await UserRepository().get_by_id(user_id)\n if not user:\n raise HTTPException(status_code=404, detail=\"User not found\")\n return user # user.profile is a Profile instance\nPatching an FK field raises ValueError \u2014 switch FK changes to update(user_id, model) instead. See use case 05 for the reasoning with soft deletes.None, not an error.ObjectId) raises a ValueError (\"run repair script\") \u2014 migrate embedded docs to a separate collection first.create/update receives a field that is already an ObjectId under a populate rule, _depopulate raises ValueError(\"...was populate skipped?\") \u2014 populate-then-depopulate pairs must be balanced.Scenario: An Author has books (a list of references), each Book references a publisher, and an Author may reference a mentor \u2014 which is another Author (a potential cycle).
nested_rules Populates deeper levels: resolve books on Author, then publisher inside each Book. max_depth Per-rule recursion bound \u2014 the safety net for cyclic graphs. global_max_depth PopulationEngine(repos, global_max_depth=10) global cap. CircularReferenceError Raised when a (Class, id) pair is revisited; carries the visited path."},{"location":"03_use_cases/09_advanced_population/#example","title":"\ud83d\ude80 Example","text":"Pythonfrom contextlib import asynccontextmanager\n\nfrom bson import ObjectId\nfrom fastapi import FastAPI, HTTPException\nfrom mongo_ops import BaseDocument, ModelRegistry, MongoConnectionManager, PopulatingRepository\nfrom mongo_ops.cache import CircularReferenceError\nfrom mongo_ops.populate import PopulateRule, PopulationEngine\n\n\n# 1. Models \u2014 each ref field holds ObjectId(s) in the DB and becomes model(s) in memory.\nclass Publisher(BaseDocument):\n name: str = \"\"\n country: str = \"\"\n\n\nclass Book(BaseDocument):\n title: str = \"\"\n publisher: Publisher | None = None # ObjectId in DB, Publisher in memory\n\n\nclass Author(BaseDocument):\n name: str = \"\"\n books: list[Book] | None = None # list[ObjectId] in DB, list[Book] in memory\n mentor: \"Author\" | None = None # self-reference \u2014 potential cycle\n\n\n# 2. Engine + nested rules.\nengine = PopulationEngine({})\n\npublisher_rule = PopulateRule(\n field_name=\"publisher\",\n collection_name=\"publishers\",\n)\n\nbook_rule = PopulateRule(\n field_name=\"books\",\n collection_name=\"books\",\n nested_rules=[publisher_rule], # fetch each book, then its publisher\n max_depth=3,\n)\n\nmentor_rule = PopulateRule(\n field_name=\"mentor\",\n collection_name=\"authors\",\n max_depth=2, # stops mentor chains early \u2014 also avoids unbounded cycles\n)\n\n\nclass AuthorRepository(PopulatingRepository[Author]):\n def __init__(self):\n super().__init__(\n collection_name=\"authors\",\n model=Author,\n population_engine=engine,\n populate_rules=[book_rule, mentor_rule],\n )\n\n\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"library\"\n ):\n engine.register_repo(\"publishers\", PopulatingRepository[Publisher](\"publishers\", Publisher))\n engine.register_repo(\"books\", PopulatingRepository[Book](\"books\", Book))\n engine.register_repo(\"authors\", AuthorRepository())\n await ModelRegistry.initialize_all()\n yield\n\n\napp = FastAPI()\n\n\n@app.get(\"/authors/{author_id}\")\nasync def get_author(author_id: str):\n try:\n author = await AuthorRepository().get_by_id(author_id)\n except CircularReferenceError as exc:\n raise HTTPException(status_code=409, detail=f\"Circular reference: {exc}\")\n if not author:\n raise HTTPException(status_code=404, detail=\"Author not found\")\n return author\nCircularReferenceError is a ValueError subtype exposing .collection, .doc_id, and the visited .path \u2014 use it in error responses and logging.max_depth conservative (2\u20133) for most graphs; combine with the global global_max_depth=10 default.list[ObjectId] on create/update, mirroring the scalar case.Scenario: A service needs fast reads of a User and its related Profile. We want the cache-hit path to skip the DB entirely while still returning a populated document.
CachedBaseRepository Provides the cache layer (cache-first get_by_id, invalidation on update/delete). PopulationEngine Provides the reference resolution. Composed subclass A small CachedBaseRepository subclass adds two _populate helpers so cache hits return populated models. \u26a0\ufe0f Known limitation: there is no built-in CachedPopulatingRepository in the library. This use case documents the composition. We cache the raw (depopulated) document \u2014 references stay ObjectIds until read time, so Profile changes are reflected on the next fetch (within TTL) and update/delete invalidation remains correct.
from contextlib import asynccontextmanager\nfrom datetime import datetime\nfrom typing import Any, Union\n\nfrom bson import ObjectId\nfrom fastapi import FastAPI\nfrom mongo_ops import BaseDocument, CachedBaseRepository, ModelRegistry, MongoConnectionManager\nfrom mongo_ops.cache import CacheConfig, InMemoryCacheBackend\nfrom mongo_ops.cache.in_memory import decode_value, encode_value\nfrom mongo_ops.populate import PopulateRule, PopulationEngine\n\n\n# 1. Models \u2014 `profile` holds an ObjectId in DB, a Profile in memory.\nclass Profile(BaseDocument):\n avatar_url: str = \"\"\n bio: str = \"\"\n\n\nclass User(BaseDocument):\n username: str = \"\"\n email: str = \"\"\n profile: Profile | None = None\n\n\n# 2. Engine + rule.\nengine = PopulationEngine({})\nprofile_rule = PopulateRule(field_name=\"profile\", collection_name=\"profiles\")\n\n\n# 3. Composed repository.\nclass CachedUserRepository(CachedBaseRepository[User]):\n def __init__(\n self,\n cache_backend: InMemoryCacheBackend,\n population_engine: PopulationEngine,\n populate_rules: list[PopulateRule],\n config: CacheConfig | None = None,\n ):\n super().__init__(\"users\", User, cache_backend, config)\n self.population_engine = population_engine\n self._populate_rules = populate_rules\n\n async def _populate(self, data: dict[str, Any]) -> dict[str, Any]:\n \"\"\"Dict-level resolution (mirrors PopulatingRepository._populate).\"\"\"\n for rule in self._populate_rules:\n ref = data.get(rule.field_name)\n if ref is None:\n continue\n if isinstance(ref, str): # JSON round-trip turns ObjectId into hex str\n ref = ObjectId(ref)\n if isinstance(ref, list):\n resolved = []\n for item in ref:\n item = ObjectId(item) if isinstance(item, str) else item\n repo = self.population_engine._repos.get(rule.collection_name)\n doc = await repo.get_by_id(item) if repo else None\n resolved.append(doc)\n data[rule.field_name] = resolved\n elif isinstance(ref, ObjectId):\n repo = self.population_engine._repos.get(rule.collection_name)\n data[rule.field_name] = await repo.get_by_id(ref) if repo else None\n return data\n\n async def _depopulate(self, data: User) -> dict[str, Any]:\n \"\"\"Model -> raw dict, FK fields collapsed to ObjectId (mirrors _depopulate).\"\"\"\n doc = data.model_dump(exclude={\"id\"}, exclude_none=True)\n for rule in self._populate_rules:\n value = getattr(data, rule.field_name, None)\n if isinstance(value, list):\n doc[rule.field_name] = [item.id for item in value]\n elif isinstance(value, BaseDocument):\n doc[rule.field_name] = value.id\n return doc\n\n async def create(self, data: User) -> User:\n doc = await self._depopulate(data) # store FK refs as ObjectIds\n doc[\"created_at\"] = datetime.utcnow()\n doc[\"updated_at\"] = datetime.utcnow()\n result = await self.collection.insert_one(doc)\n doc[\"_id\"] = result.inserted_id\n if self._cache_config.enabled:\n await self._cache.set(self._cache_key(doc[\"_id\"]), encode_value(doc), self._cache_config.default_ttl)\n return self.model(**await self._populate(doc))\n\n async def update(self, id: Union[str, ObjectId], data: User) -> User | None:\n if isinstance(id, str):\n id = ObjectId(id)\n doc = await self._depopulate(data) # update() takes a full model here\n doc[\"updated_at\"] = datetime.utcnow()\n result = await self.collection.find_one_and_update(\n {\"_id\": id}, {\"$set\": doc}, return_document=True\n )\n key = self._cache_key(id)\n if result is None:\n if self._cache_config.enabled:\n await self._cache.delete(key)\n return None\n raw = dict(result)\n if self._cache_config.enabled:\n await self._cache.set(key, encode_value(raw), self._cache_config.default_ttl)\n return self.model(**await self._populate(raw))\n\n async def get_by_id(self, id: Union[str, ObjectId]) -> User | None:\n if not self._cache_config.enabled:\n # No cache: still a raw fetch + populate \u2014 the base get_by_id()\n # would build a typed model from the raw doc and fail on FK fields.\n raw = await self.collection.find_one({\"_id\": ObjectId(id) if isinstance(id, str) else id})\n return self.model(**await self._populate(raw)) if raw else None\n\n cached = await self._cache.get(self._cache_key(id))\n if cached is not None:\n # Cache hit: decode the RAW doc, then populate before returning.\n data = await self._populate(decode_value(cached))\n return self.model(**data)\n\n # Cache miss: one RAW DB read. The base get_by_id() would rebuild the\n # model from the raw doc, which fails for model-typed FK fields \u2014\n # so fetch the raw dict, cache it, and populate before materializing.\n key = self._cache_key(id)\n raw = await self.collection.find_one({\"_id\": ObjectId(id) if isinstance(id, str) else id})\n if raw is None:\n return None\n await self._cache.set(key, encode_value(raw), self._cache_config.default_ttl)\n return self.model(**await self._populate(raw))\n\n\n# 4. Wire-up \u2014 one backend for both the repo and the registry lifecycle.\ncache = InMemoryCacheBackend(max_entries=20_000, default_ttl=600)\nModelRegistry.set_cache_backend(cache)\n\n\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"app_db\"\n ):\n # Register the repositories the engine resolves refs against:\n # engine.register_repo(\"profiles\", ProfileRepo())\n engine.register_repo(\"users\", CachedUserRepository(cache, engine, [profile_rule]))\n await ModelRegistry.initialize_all()\n await ModelRegistry.initialize_cache()\n yield\n await ModelRegistry.shutdown_cache()\n\n\napp = FastAPI(lifespan=lifespan)\nNote: replace the placeholder engine.register_repo(\"profiles\", ...) line with a repository for Profile. A matching CachedBaseRepository[Profile](\"profiles\", Profile, cache) keeps profile reads cached too \u2014 e.g. engine.register_repo(\"profiles\", CachedBaseRepository[Profile](\"profiles\", Profile, cache)).
ObjectIds in the cache; JSON round-trips them to hex strings, so _populate must normalize str \u2192 ObjectId (shown above).update/create above are model-based (update(id, User)), mirroring PopulatingRepository \u2014 they depopulate before writing and cache a raw snapshot. The plain CachedBaseRepository.update(id, dict) and delete(id) keep working and invalidate the same key.default_ttl \u2014 cached User entries resolve Profile on each read, so profile edits show up within the TTL (or call invalidate_cache(user_id) explicitly)._populate / _depopulate helpers plus the create / update / get_by_id overrides into a mixin and parameterize the rules per subclass.Scenario: A microservice uses an in-memory (or Redis) cache backend and must start the cleanup task on app start, then shut it down cleanly on termination.
"},{"location":"03_use_cases/11_cache_lifecycle/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component DescriptionModelRegistry.set_cache_backend(backend) Registers the single shared backend. ModelRegistry.initialize_cache() Starts the backend (spawns the in-memory TTL cleanup task). Raises RuntimeError if no backend was registered. ModelRegistry.shutdown_cache() Cancels the cleanup task, closes pub/sub, clears the backend. If a repo uses a backend directly Same instance must be registered so initialize_cache starts its task."},{"location":"03_use_cases/11_cache_lifecycle/#example","title":"\ud83d\ude80 Example","text":"Pythonfrom contextlib import asynccontextmanager\n\nfrom fastapi import FastAPI\nfrom mongo_ops import ModelRegistry, MongoConnectionManager\nfrom mongo_ops.cache import InMemoryCacheBackend\n\ncache = InMemoryCacheBackend(max_entries=10_000, default_ttl=300)\nModelRegistry.set_cache_backend(cache) # before any cache-backed repo is used\n\n\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"mydb\"\n ):\n await ModelRegistry.initialize_all() # create indexes (idempotent)\n await ModelRegistry.initialize_cache() # start the TTL cleanup task\n yield\n await ModelRegistry.shutdown_cache() # cancel task + close cleanly\n\n\napp = FastAPI(lifespan=lifespan)\nInMemoryCacheBackend.initialize() spawns an asyncio.Task that evicts expired entries every cleanup_interval seconds. Without a shutdown, the event loop flags the dangling task on exit \u2014 shutdown_cache() cancels it and awaits it.shutdown_cache() also NULs the registry cache backend and closes the Redis pub/sub handle (if Redis).initialize_cache() is called before set_cache_backend(), it raises:RuntimeError: No cache backend registered. Call set_cache_backend() first.\nCacheConfig(enabled=False) bypasses the cache for that repo even after initialize_cache().cleanup_interval seconds for scans; entries also expire on access via the TTL heap (default_ttl=0 expires immediately).CachedBaseRepository that references it \u2014 otherwise a repo may hold an uninitialized backend (no cleanup task, no Redis pub/sub).TransactionManager.execute_transaction","text":"Scenario: Perform several writes across different collections atomically \u2014 e.g., create an Order and decrement Inventory. The low-level start_session context works, but execute_transaction collects results from a list of async operations.
TransactionManager.execute_transaction Runs a list of async callables (each receives a session) inside one transaction; returns the result of each callable, in order. Automatic rollback Any raised exception aborts the transaction and propagates to the caller. start_session The underlying async context manager async with TransactionManager.start_session() as session:."},{"location":"03_use_cases/12_transaction_helper/#example","title":"\ud83d\ude80 Example","text":"Pythonfrom contextlib import asynccontextmanager\nfrom datetime import datetime\n\nfrom bson import ObjectId\nfrom fastapi import FastAPI, HTTPException\nfrom mongo_ops import BaseDocument, ModelRegistry, MongoConnectionManager\nfrom mongo_ops.repository import BaseRepository\nfrom mongo_ops.transactions import TransactionManager\n\n\n# 1. Models\nclass Order(BaseDocument):\n user_id: str = \"\"\n items: list[dict] = [] # [{\"product_id\": ObjectId, \"qty\": int}]\n total: float = 0.0\n\n\nclass Inventory(BaseDocument):\n product_id: ObjectId = None\n quantity: int = 0\n\n\n# 2. Repositories (constructed inside the lifespan \u2014 after connect()).\nclass OrderRepo(BaseRepository[Order]):\n def __init__(self):\n super().__init__(\"orders\", Order)\n\n\nclass InventoryRepo(BaseRepository[Inventory]):\n def __init__(self):\n super().__init__(\"inventory\", Inventory)\n\n\n# 3. The atomic operation.\nasync def create_order_with_inventory(order: Order, order_repo: OrderRepo, inv_repo: InventoryRepo):\n async def insert_order(session):\n doc = order.model_dump(exclude={\"id\"}, exclude_none=True)\n doc[\"created_at\"] = doc[\"updated_at\"] = datetime.utcnow()\n result = await order_repo.collection.insert_one(doc, session=session)\n return await order_repo.collection.find_one({\"_id\": result.inserted_id}, session=session)\n\n async def update_inventory(session):\n for item in order.items:\n await inv_repo.collection.update_one(\n {\"product_id\": ObjectId(item[\"product_id\"])},\n {\"$inc\": {\"quantity\": -item[\"qty\"]}},\n session=session,\n )\n return \"inventory-updated\"\n\n results = await TransactionManager.execute_transaction(\n [insert_order, update_inventory]\n )\n return results[0] # created order document; results[1] == \"inventory-updated\"\n\n\n# 4. FastAPI wiring.\napp = FastAPI()\n\n\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"shop\"\n ):\n await ModelRegistry.initialize_all()\n yield\n\n\napp.lifespan = lifespan\n\n\n@app.post(\"/orders/\", response_model=Order)\nasync def create_order(order: Order):\n created = await create_order_with_inventory(order, OrderRepo(), InventoryRepo())\n if created is None:\n raise HTTPException(status_code=400, detail=\"Transaction failed\")\n return created\nsession=session to find_one/find too.monkeypatch a fake client on mongo_ops.transactions.MongoConnectionManager.get_client and stub start_session \u2014 see tests/test_transactions.py.created_at/updated_at; for raw collection inserts inside the transaction you set them manually (as shown).Scenario: Ensure each collection has the right indexes for fast queries and data integrity \u2014 declared in one place and created at startup.
"},{"location":"03_use_cases/13_index_creation/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component DescriptionModelRegistry.register(indexes=...) Each spec is passed as-is to pymongo create_index. Supported forms: tuple (\"field\", direction), compound list [(\"a\", 1), (\"b\", -1)], or a dict with keys + options. ModelRegistry.initialize_all Creates every registered index during startup (idempotent \u2014 create_index skips existing indexes)."},{"location":"03_use_cases/13_index_creation/#example","title":"\ud83d\ude80 Example","text":"Pythonfrom contextlib import asynccontextmanager\n\nfrom pymongo import ASCENDING, DESCENDING\nfrom mongo_ops import BaseDocument, ModelRegistry, MongoConnectionManager\n\n\nclass User(BaseDocument):\n username: str = \"\"\n email: str = \"\"\n\n\nclass BlogPost(BaseDocument):\n author_id: str = \"\"\n created_at: str = \"\"\n title: str = \"\"\n\n\nclass Passenger(BaseDocument):\n email: str = \"\"\n seat: str = \"\"\n\n\n# 1\ufe0f\u20e3 Single-field index \u2014 email lookups.\nModelRegistry.register(\n collection_name=\"users\",\n model=User,\n indexes=[(\"email\", ASCENDING)],\n)\n\n# 2\ufe0f\u20e3 Composite index \u2014 queries filtering by author + creation date.\nModelRegistry.register(\n collection_name=\"posts\",\n model=BlogPost,\n indexes=[[(\"author_id\", ASCENDING), (\"created_at\", DESCENDING)]],\n)\n\n# 3\ufe0f\u20e3 Unique index with a custom name \u2014 enforce unique emails.\nModelRegistry.register(\n collection_name=\"passengers\",\n model=Passenger,\n indexes=[\n {\n \"keys\": [(\"email\", ASCENDING)],\n \"options\": {\"unique\": True, \"name\": \"uq_passenger_email\"},\n }\n ],\n)\n\n\n# 4\ufe0f\u20e3 Everything is created on startup.\n@asynccontextmanager\nasync def lifespan(_app):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\",\n db_name=\"mydb\",\n ):\n await ModelRegistry.initialize_all()\n yield\ncollection.create_index(spec) \u2014 so MongoDB options like unique, sparse, and expireAfterSeconds (TTL) belong in the options dict.create_index is a no-op when a same-shape index already exists.db.<collection>.getIndexes()\nDuplicateKeyError on startup \u2014 clean the data first.Scenario: Write unit tests that never touch a real MongoDB \u2014 mock the collection, the cache backend, and the client, exactly like the library's own test suite (tests/).
AsyncMock collections Stub find_one, insert_one, find_one_and_update, cursor chains. Patching MongoConnectionManager.get_database Gives repositories a mock collection without a live connection. monkeypatch on get_client Fakes sessions for TransactionManager tests. Reference tests tests/test_populating_repository.py, tests/test_cache.py, tests/test_registry.py, tests/test_transactions.py."},{"location":"03_use_cases/14_testing_guide/#example-boilerplate","title":"\ud83d\ude80 Example Boilerplate","text":"Pythonimport pytest\nfrom unittest.mock import AsyncMock, MagicMock, patch\n\nfrom bson import ObjectId\nfrom mongo_ops.cache import CacheConfig, InMemoryCacheBackend\nfrom mongo_ops.cache.repository import CachedBaseRepository\nfrom mongo_ops.models import BaseDocument\nfrom mongo_ops.registry import ModelRegistry\nfrom mongo_ops.populate import PopulateRule, PopulationEngine\nfrom mongo_ops.repository import PopulatingRepository\n\n\n# ----------------------------------------------------------------------\n# 1. Models (same shape as the library tests)\n# ----------------------------------------------------------------------\nclass Profile(BaseDocument):\n avatar_url: str = \"\"\n\n\nclass User(BaseDocument):\n name: str = \"\"\n profile: Profile | None = None # ObjectId in DB, Profile in memory\n\n\n# ----------------------------------------------------------------------\n# 2. Fixtures\n# ----------------------------------------------------------------------\n@pytest.fixture\ndef mock_collection():\n return AsyncMock()\n\n\n@pytest.fixture\ndef engine():\n profile_repo = AsyncMock()\n return PopulationEngine({\"profiles\": profile_repo})\n\n\n@pytest.fixture\ndef repo(mock_collection, engine):\n with patch(\"mongo_ops.repository.MongoConnectionManager.get_database\") as mock_db:\n mock_db.return_value.__getitem__.return_value = mock_collection\n r = PopulatingRepository(\n \"users\",\n User,\n population_engine=engine,\n populate_rules=[PopulateRule(field_name=\"profile\", collection_name=\"profiles\")],\n )\n r.collection = mock_collection\n return r\n\n\n# ----------------------------------------------------------------------\n# 3. Population \u2014 get_by_id resolves the reference\n# ----------------------------------------------------------------------\n@pytest.mark.asyncio\nasync def test_get_by_id_populates(repo, mock_collection, engine):\n uid, pid = ObjectId(), ObjectId()\n mock_collection.find_one.return_value = {\n \"_id\": uid,\n \"name\": \"Alice\",\n \"profile\": pid, # ObjectId stored in DB\n \"created_at\": \"2024-01-01T00:00:00\",\n \"updated_at\": \"2024-01-01T00:00:00\",\n }\n engine._repos[\"profiles\"].get_by_id.return_value = Profile(id=pid, avatar_url=\"pic.png\")\n\n result = await repo.get_by_id(uid)\n\n assert result is not None\n assert result.name == \"Alice\"\n assert isinstance(result.profile, Profile)\n assert result.profile.avatar_url == \"pic.png\"\n\n\n# ----------------------------------------------------------------------\n# 4. Patch FK guard\n# ----------------------------------------------------------------------\n@pytest.mark.asyncio\nasync def test_patch_rejects_fk_field(repo):\n with pytest.raises(ValueError, match=\"Cannot patch FK fields\"):\n await repo.patch(ObjectId(), {\"profile\": ObjectId()})\n\n\n# ----------------------------------------------------------------------\n# 5. Registry \u2014 index specs pass through to create_index\n# ----------------------------------------------------------------------\n@pytest.mark.asyncio\nasync def test_initialize_all_creates_indexes():\n ModelRegistry.register(\"users\", User, indexes=[(\"email\", 1)])\n\n fake_collection = AsyncMock()\n await ModelRegistry.initialize_all(db={\"users\": fake_collection})\n\n fake_collection.create_index.assert_awaited_once_with((\"email\", 1))\n\n\n# ----------------------------------------------------------------------\n# 6. Cached repository \u2014 cache-first reads + invalidation\n# ----------------------------------------------------------------------\n@pytest.mark.asyncio\nasync def test_cached_get_by_id_populates_cache():\n backend = InMemoryCacheBackend(\n max_entries=100, default_ttl=300, cleanup_interval=9999\n )\n await backend.initialize()\n try:\n with patch(\"mongo_ops.repository.MongoConnectionManager.get_database\") as mock_db:\n mock_collection = AsyncMock()\n mock_db.return_value.__getitem__.return_value = mock_collection\n repo = CachedBaseRepository(\n \"users\", User, backend, CacheConfig(enabled=True)\n )\n repo.collection = mock_collection\n\n oid = ObjectId()\n mock_collection.find_one.return_value = {\n \"_id\": oid,\n \"name\": \"cached\",\n \"created_at\": \"2024-01-01T00:00:00\",\n \"updated_at\": \"2024-01-01T00:00:00\",\n }\n\n first = await repo.get_by_id(oid)\n assert first is not None\n\n mock_collection.find_one.return_value = None # DB now \"empty\"\n second = await repo.get_by_id(oid) # served from cache\n\n assert second is not None\n assert second.name == \"cached\"\n mock_collection.find_one.assert_awaited_once() # only one DB read\n finally:\n await backend.shutdown()\n\n\n# ----------------------------------------------------------------------\n# 7. Transactions \u2014 fake the client's start_session\n# ----------------------------------------------------------------------\nfrom mongo_ops.transactions import TransactionManager\n\n\n@pytest.mark.asyncio\nasync def test_execute_transaction(monkeypatch):\n session_ctx = AsyncMock()\n session_ctx.__aenter__.return_value = AsyncMock()\n session_ctx.__aexit__.return_value = None\n\n client = MagicMock()\n client.start_session = AsyncMock(return_value=session_ctx)\n\n # NOTE: start_transaction must return a context manager, not a coroutine.\n async_session = session_ctx.__aenter__.return_value\n async_session.start_transaction = lambda **_: session_ctx\n\n monkeypatch.setattr(\n \"mongo_ops.transactions.MongoConnectionManager.get_client\",\n lambda: client,\n )\n\n async def fake_op(session):\n return \"ok\"\n\n results = await TransactionManager.execute_transaction([fake_op])\n assert results == [\"ok\"]\npyproject.toml (asyncio_mode = \"auto\"), so @pytest.mark.asyncio tests work out of the box. Run with pytest (coverage reports are enabled there too).conftest.py) and reuse them.cursor chains with MagicMock() + .to_list = AsyncMock(...), exactly like tests/test_repository.py.MongoConnectionManager.lifespan against a local replica set and drop the test database in teardown \u2014 keep it separate from the unit suite.RedisCacheBackend methods (get, set, delete) \u2014 no Redis process required.PopulatingRepository \u2014 the Object \u21c4 ObjectId Lifecycle","text":"Scenario: You want to see what the repository actually does, in which order, before you trust it with your data \u2014 how a Profile becomes an ObjectId for storage and comes back as a Profile on read, and where every piece is wired.
A populate-ruled field is a shape shifter \u2014 the same name holds different things depending on where you look:
Place Value held in the field In the MongoDB docObjectId (or list[ObjectId]) In the cache/JSON hex string (see use case 16) In the app model the referenced model (or None) PopulatingRepository.create / update depopulate (model \u2192 ObjectId) before writing; get_by_id / get_many populate (ObjectId \u2192 model) after reading:
App model MongoDB document App model\nUser(profile=Profile) \u2500\u2500depopulate\u2500\u2500\u25b6 { profile: <ObjectId> } \u2500\u2500populate\u2500\u2500\u25b6 User(profile=Profile)\n (write path) (storage shape) (read path)\ncreate() Step by Step","text":"Given user = User(username=\"alice\", profile=saved_profile) where saved_profile.id exists:
class UserRepository(PopulatingRepository[User]):\n def __init__(self):\n super().__init__(\n collection_name=\"users\",\n model=User,\n population_engine=engine,\n populate_rules=[profile_rule],\n )\n_depopulate, repository.py): for each rule field, if the value is already an ObjectId (or list[ObjectId]) an error is raised \u2014 see _depopulate guards below. Saved models pass.engine.depopulate(user, rules) collapses the graph in place:saved_profile (a BaseDocument with .id) \u2192 becomes saved_profile.id \u2192 ObjectIdProfile without .id (unsaved) \u2192 is left as a model object, which model_dump then embeds as a dict \u2014 see intricacy #3nested_rules are present, the nested model is depopulated recursively first, then collapsedmodel_dump(exclude={\"id\"}, exclude_none=True) produces the raw insert dict; timestamps are added.insert_one(doc) writes {..., \"profile\": ObjectId(\"...\"), ...} to MongoDB.data_to_model runs _populate (the read path below) so create returns a fully populated model, not the raw one._depopulate guards (loud failures > silent corruption)","text":"Stored/held value under a rule field Behaviour ObjectId / list[ObjectId] ValueError(\"...contains ObjectId \u2014 was populate skipped?\") \u2014 the doc was loaded raw (e.g. via BaseRepository or an unpopulated read) and handed back to create/update a saved BaseDocument (has .id) collapse to .id (ObjectId) an unsaved BaseDocument (no .id) kept as a model \u2192 embedded dict below a dict not BaseDocument \u2192 left as-is \u2192 embedded non-BaseDocument, non-dict value AttributeError raised by engine.depopulate no engine / no rules passthrough \u2014 _depopulate just does model_dump, whatever shape you gave is stored"},{"location":"03_use_cases/15_populating_repository_wiring/#read-path-get_by_id-step-by-step","title":"\ud83d\ude80 Read Path \u2014 get_by_id() Step by Step","text":"get_by_id inherits CRUD and only changes data_to_model (repository.py:247):
doc = await self.collection.find_one({\"_id\": id}) # {\"profile\": ObjectId, ...}\nreturn await self.data_to_model(doc) # data_to_model \u2192 self._populate(doc)\n_populate(data) walks each rule field in the raw dict:
ref = data.get(\"profile\") \u2192 ObjectId. If ref is None \u2192 field untouched (stays absent).repo = engine._repos.get(\"profiles\"). If it's not registered, doc = None.doc = await repo.get_by_id(ref) \u2014 a single read on the referenced collection.doc is None \u2192 data[\"profile\"] = None (missing refs resolve to None, never raise).nested_rules \u2192 engine.populate(doc, nested_rules, depth=1) deepens the result.data[\"profile\"] = doc \u2192 self.model(**data) builds the User with a real Profile.Lists behave the same per item; a ref that is a dict (an embedded document) raises ValueError(\"...contains embedded dict(s) \u2014 run repair script\").
Engine is optional. Set population_engine=None and populate_rules=[], and PopulatingRepository is just a BaseRepository \u2014 FK fields come back as raw ObjectId, and your model field type must agree (that is the whole point of the guard in the WRITE path: a Profile | None-typed field populated with a raw ObjectId is a broken round-trip waiting to happen).
from mongo_ops import BaseDocument, PopulatingRepository\nfrom mongo_ops.populate import PopulateRule, PopulationEngine\n\nclass Profile(BaseDocument):\n avatar_url: str = \"\"\n\nclass User(BaseDocument):\n username: str = \"\"\n profile: Profile | None = None # ObjectId in DB, Profile in memory\n\nengine = PopulationEngine({}) # repositories live here, keyed by collection name\n\nprofile_rule = PopulateRule(field_name=\"profile\", collection_name=\"profiles\")\n\nclass UserRepository(PopulatingRepository[User]):\n def __init__(self):\n super().__init__(\"users\", User, population_engine=engine, populate_rules=[profile_rule])\n\ndef wire() -> None:\n \"\"\"Call AFTER connect() \u2014 repositories need a live database.\"\"\"\n engine.register_repo(\"profiles\", PopulatingRepository[Profile](\"profiles\", Profile))\n engine.register_repo(\"users\", UserRepository())\nWiring rules:
engine._repos is keyed by collection_name as written in the rule \u2014 typo \u2192 silent None refs.register_repo needs an already-constructed repository \u2192 call it inside the lifespan (after connect() / MongoConnectionManager.lifespan), not at module import.get_by_id that returns a BaseDocument \u2014 it can be a plain BaseRepository, another PopulatingRepository, or even a cached repo (see use case 10). Population requires no extra DB index on the referenced collection's _id.repo.set_population_engine(new_engine) and repo.set_populate_rules(new_rules) \u2014 the tests exercise both.nested_rules Do NOT Round-Trip Through depopulate","text":"nested_rules are designed for read-side deep population. If the same rules run through the write path (create / update), engine.depopulate behaves differently from plain refs \u2014 verified against the engine:
depopulate does to it Effect on the stored doc scalar FK, no nested_rules collapse to .id stored as ObjectId \u2713 list[ObjectId] FK, no nested_rules collapse each item to .id stored as list[ObjectId] \u2713 list FK with nested_rules items are kept as models stored as embedded dicts \u2717 \u2014 reading back raises \"contains embedded dict(s) \u2014 run repair script\" scalar FK with nested_rules the recursion runs, then the field is assigned None reference lost \u2717 \u2014 written as absent/null In other words: a document carrying deep nested_rules (like UC 09's Author \u2192 books \u2192 publisher) cannot be created / updated as-is. If the graph must be written back, materialize references separately (save each Book to its collection first, then store list[ObjectId] without nested_rules), and keep nested_rules only on read rules you never hand back to depopulate.
ObjectId-holding model to create/update: the \"was populate skipped?\" ValueError is the guard. Read through the repo so the read path can populate first.ValueError. Save referenced docs to their collection first, then reference their id.filter / projection on PopulateRule are declared but not applied by the engine \u2014 don't rely on them.$lookup yet); batch-heavy endpoints should add caching (next use case).Scenario: You want the exact contract of CachedBaseRepository \u2014 what goes into the cache, in what shape, and why a cached read occasionally looks \"wrong\" for populated models \u2014 before wiring it into a service.
\"{key_prefix}{id}\" \u2014 default prefix \"{collection_name}:\" Value json.dumps(model_dump(by_alias=True), default=str) \u2014 a byte string of JSON get_by_id hit decode_value(cached) \u2192 self.model(**data) \u2014 no DB hit get_by_id miss DB read, then the raw doc is cached (model_dump(by_alias=True)) create creates in DB, then caches the result (model_dump) update DB update, then set new value or delete the key if the doc vanished delete DB delete, then removes the key TTL config.default_ttl (default 300 s); in-memory eviction is LRU + expiry So the cache stores JSON snapshots of whole documents \u2014 it is a read-through cache keyed by document id, not a query cache.
"},{"location":"03_use_cases/16_cached_repository_intricacies/#intricacy-1-objectids-become-hex-strings","title":"\u26a0\ufe0f Intricacy #1 \u2014 ObjectIds Become Hex Strings","text":"json.dumps(..., default=str) stringifies every non-JSON value \u2014 most importantly an ObjectId in a FK field:
# model in memory: User(id=..., profile=ObjectId(\"507f1f77bcf86cd799439011\"))\n# cached bytes: b'{\"_id\":\"507f1f77bcf86cd799439011\",\"profile\":\"507f1f77bcf86cd799439011\", ...}'\nOn a cache hit, self.model(**data) must therefore accept a hex string where the raw doc held an ObjectId. This is fine for: - id (typed PyObjectId, which accepts both str and ObjectId), and - created_at / updated_at (ISO strings coerce to datetime).
It is not fine for a field typed as a model.
"},{"location":"03_use_cases/16_cached_repository_intricacies/#intricacy-2-model-typed-fk-fields-fail-on-a-cache-hit","title":"\u26a0\ufe0f Intricacy #2 \u2014 Model-Typed FK Fields Fail on a Cache Hit","text":"If your document has a populated field, e.g.:
Pythonclass User(BaseDocument):\n profile: Profile | None = None # populate-ruled\nthen the base-class hit path self.model(**data) receives profile=\"507f\u2026\" and Pydantic raises a ValidationError \u2014 a string cannot coerce into a Profile. The miss path fails the same way: BaseRepository.get_by_id materializes the model from the raw doc whose profile is an ObjectId \u2014 also a ValidationError. So a plain CachedBaseRepository cannot materialize a model-typed FK field at all, hit or miss \u2014 the problem is the base classes build a typed model straight from raw docs.
The tension is structural: PopulatingRepository types the field as the model; CachedBaseRepository stores the raw (depopulated) shape. You cannot have a single typed model serve both at once. The two resolutions:
profile: Profile | None and cache raw, populating on read. Exact recipe in use case 10: its miss path fetches the raw doc (bypassing the base model-build) and both paths normalize str \u2192 ObjectId before populate, because the JSON round-trip hands you strings.profile: PyObjectId | None and no populate rules; then cached hits validate cleanly, but you've given up population entirely.Do not cache a populated model object through the base class: create caches result.model_dump, so the first cache write stores the embedded-dict shape while later get_by_id misses would repopulate \u2014 inconsistent shapes for the same key, and update overwrites with yet another. Pick one canonical raw shape and stick to it.
Python
from mongo_ops import BaseDocument, CachedBaseRepository, ModelRegistry, MongoConnectionManager\nfrom mongo_ops.cache import CacheConfig, InMemoryCacheBackend\n\nclass Product(BaseDocument):\n name: str = \"\"\n price: float = 0.0\n\ncache = InMemoryCacheBackend(max_entries=10_000, default_ttl=600)\nModelRegistry.set_cache_backend(cache)\n\nclass ProductRepo(CachedBaseRepository[Product]):\n def __init__(self):\n super().__init__(\"products\", Product, cache, CacheConfig(enabled=True, backend=\"memory\"))\n\nasync def inspect_cache(repo_id: str) -> None:\n # After create()/get_by_id(), inspect what is actually stored:\n cached = await cache.get(f\"products:{repo_id}\")\n # b'{\"_id\":\"507f...\",\"name\":\"Widget\",\"price\":9.99,\"created_at\":\"2026-...\",\"updated_at\":\"2026-...\"}'\n print(cached)\n\n stats = await cache.get_stats()\n print(stats.hits, stats.misses, stats.sets, stats.deletes) # CacheStats dataclass\ncache_backend=cache) and the registry needs it (ModelRegistry.set_cache_backend(cache)) so initialize_cache() / shutdown_cache() manage the same object. Shutdown cancels the in-memory TTL cleanup task \u2014 forgetting it leaks an asyncio.Task at app exit.initialize_cache() starts the backend; initialize_all() creates indexes. Both come after connect().warm_cache([ids]) skips keys that already exist, fetches the rest from the DB, and returns how many it wrote \u2014 safe to call repeatedly.invalidate_cache(id) deletes one key; clear_pattern(\"products:*\") wipes a collection.redis.asyncio) Key prefixing prefix baked into the stored key _full_key() applied on every op TTL heap-based, lazy eviction + cleanup task Native SETEX Invalidation broadcast n/a PUBLISH on mongo_ops:cache:invalidate on delete clear_pattern prefix match on stored keys SCAN MATCH in batches"},{"location":"03_use_cases/16_cached_repository_intricacies/#choosing-the-right-layer","title":"\ud83c\udfaf Choosing the Right Layer","text":"Need Use Scalar docs, no refs \u2014 cache-first reads CachedBaseRepository (UC 07, UC 16) Refs resolved on read, no caching PopulatingRepository (UC 08, UC 15) Refs and cache-first reads composed subclass (UC 10) Bulk warm on startup / cache-then-database failover warm_cache + CacheStats"},{"location":"03_use_cases/16_cached_repository_intricacies/#related","title":"Related","text":"Scenario: You want to see what populate/depopulate actually do to your documents \u2014 field by field, at every layer (Python model \u2192 depopulated \u2192 stored in Mongo \u2192 raw read back \u2192 populated).
These examples all ran against the real mongo_ops engine (mongo_ops/populate/engine.py). Timestamps and ObjectIds are abbreviated with \u2026 for readability.
from bson import ObjectId\nfrom mongo_ops import BaseDocument, PopulatingRepository\nfrom mongo_ops.populate import PopulateRule, PopulationEngine\n\nclass Profile(BaseDocument):\n avatar_url: str = \"\"\n bio: str = \"\"\n\nclass Publisher(BaseDocument):\n name: str = \"\"\n\nclass Book(BaseDocument):\n title: str = \"\"\n publisher: Publisher | None = None # nested ref, reached via nested_rules\n\nclass User(BaseDocument):\n name: str = \"\"\n profile: Profile | None = None # scalar FK \u2014 DB stores an ObjectId\n books: list[Book] | None = None # list FK \u2014 DB stores list[ObjectId]\n\nengine = PopulationEngine({}) # repos keyed by collection name, see UC 15\n\n# A rule WITHOUT nested_rules \u2014 the write path is safe.\nprofile_rule = PopulateRule(field_name=\"profile\", collection_name=\"profiles\")\n\n# A rule WITH nested_rules \u2014 designed for READ-side deep population.\nbooks_rule = PopulateRule(\n field_name=\"books\",\n collection_name=\"books\",\n nested_rules=[PopulateRule(field_name=\"publisher\", collection_name=\"publishers\")],\n)\ndepopulate() turns models into references","text":"engine.depopulate(document, rules) walks each rule field and mutates the model in place before you insert_one / find_one_and_update it. What you get on the other side:
nested_rules \u2192 stored as ObjectId","text":"Pythonasync def demo():\n profile = Profile(id=ObjectId(\"656\u202604\"), avatar_url=\"a.png\", bio=\"hi\")\n user = User(id=ObjectId(\"656\u202605\"), name=\"Ada\", profile=profile)\n\n await engine.depopulate(user, [profile_rule])\n\n return user.profile # ObjectId('656\u202604') \u2014 was a Profile model\nWhat Mongo receives (via model_dump, minus _id handling):
{ \"name\": \"Ada\", \"profile\": { \"$oid\": \"656\u202604\" } }\nnested_rules \u2192 reference lost (do not do this)","text":"Pythonprofile_rule_deep = PopulateRule( # same field, but with nested_rules\n field_name=\"profile\",\n collection_name=\"profiles\",\n nested_rules=[PopulateRule(field_name=\"bio\", collection_name=\"profiles\")], # any child field\n)\n\nasync def demo():\n user = User(id=ObjectId(\"656\u202605\"), name=\"Ada\",\n profile=Profile(id=ObjectId(\"656\u202604\"), avatar_url=\"a.png\"))\n await engine.depopulate(user, [profile_rule_deep])\n return user.profile # None \u2014 reference silently dropped!\nBecause depopulate recurses into the child but then assigns the parent field None (see engine.py \u2014 depopulated_value is never set on that branch). On save you lose the reference entirely. There is no warning \u2014 the field is just gone.
nested_rules \u2192 stored as list[ObjectId] (the safe pattern)","text":"Pythonasync def demo():\n b1 = Book(id=ObjectId(\"656\u202606\"), title=\"MongoDB in Action\")\n b2 = Book(id=ObjectId(\"656\u202607\"), title=\"MongoDB: The Definitive Guide\")\n user = User(id=ObjectId(\"656\u202608\"), name=\"Ada\", books=[b1, b2])\n\n await engine.depopulate(user, [PopulateRule(field_name=\"books\", collection_name=\"books\")])\n\n return user.books # [ObjectId('656\u202606'), ObjectId('656\u202607')] \u2014 clean refs\nnested_rules \u2192 stored as embedded documents (do not do this)","text":"Pythonasync def demo():\n publisher = Publisher(id=ObjectId(\"656\u202609\"), name=\"O'Reilly\")\n b1p = Book(id=ObjectId(\"656\u202606\"), title=\"MongoDB in Action\", publisher=publisher)\n user = User(id=ObjectId(\"656\u202608\"), name=\"Ada\", books=[b1p])\n\n await engine.depopulate(user, [books_rule]) # books_rule carries nested_rules\n return user.books[0]\n # <Book id=ObjectId('656\u202606') publisher=ObjectId('656\u202609')>\nbooks stays a list of Book models \u2014 the nested publisher collapsed to ObjectId, but the Book itself was left embedded. Serialized to Mongo you get a nested sub-document:
{ \"name\": \"Ada\", \"books\": [ { \"title\": \"MongoDB in Action\", \"publisher\": { \"$oid\": \"656\u202609\" } } ] }\n\u2026and reading that back through population raises ValueError(\u2026contains embedded dict(s) \u2014 run repair script\u2026), because the engine expects books to hold references, not embedded docs.
Why UC 09's Author \u2192 books \u2192 publisher graph can't be written as-is: a rule with nested_rules breaks the write path (1b logs silently, 1d stores embedded). The safe round-trip: save referenced documents first, store list[ObjectId]/ObjectId with a non-nested rule for reads, and keep nested_rules only on rules you never hand back to depopulate.
populate() turns references back into models","text":"PopulatingRepository.data_to_model(raw_dict) resolves references before building the model. It first runs _populate(dict) \u2014 replacing the raw ObjectId references with full models via the registered repos \u2014 and only then constructs self.model(**data), so every FK field is a real model by construction. The two data_to_model examples below (2a/2b) show the underlying resolution _populate performs per rule. (repo below is a PopulatingRepository wired per UC 15 \u2014 books and publishers repos registered in its PopulationEngine.)
ObjectId \u2192 Profile model","text":"Python# what came back from Mongo (raw dict, profile is an ObjectId reference):\nraw = {\"_id\": ObjectId(\"656\u202605\"), \"name\": \"Ada\", \"profile\": ObjectId(\"656\u202604\")}\n\nasync def demo():\n user = await repo.data_to_model(raw) # _populate() \u2192 model(**data)\n return user.profile # Profile(id=ObjectId('656\u202604'), avatar_url='a.png')\nnested_rules \u2014 two levels deep","text":"Pythonraw = {\"_id\": ObjectId(\"656\u202608\"), \"name\": \"Ada\",\n \"books\": [ObjectId(\"656\u202606\"), ObjectId(\"656\u202607\")]}\n\nasync def demo():\n user = await repo.data_to_model(raw) # _populate() resolves both levels\n return (type(user.books[0]).__name__, # 'Book'\n type(user.books[0].publisher).__name__, # 'Publisher' \u2014 nested_rules reached it\n user.books[0].publisher.name) # \"O'Reilly\"\nWalking through one book's resolution:
Text Onlybooks_rule \u2192 field \"books\", collection \"books\"\n nested_rules[0].publisher rule \u2192 field \"publisher\", collection \"publishers\"\n1. for ObjectId('656\u202606') \u2192 repo(\"books\").get_by_id(...) \u2192 Book(\"MongoDB in Action\")\n2. book has nested rule \u2192 repo(\"publishers\").get_by_id(...) \u2192 Publisher(\"O'Reilly\")\n3. author.books[0] \u2192 Book(publisher=Publisher(\"O'Reilly\"))\nPer-level cost: 1 query per reference (no $lookup). A hub with 20 hooks and a publisher nested rule is 40 queries \u2014 that is why the next section exists.
CachedBaseRepository stores model_dump(by_alias=True) JSON-encoded with default=str (the encode_value helper in mongo_ops/cache/in_memory.py). Crucially it snapshots the raw, just-out-of-Mongo shape \u2014 FK fields still hold ObjectIds \u2014 so the cache holds references, exactly like the DB row:
profile field looks like Mongo doc (raw) \u201cprofile\u201d: ObjectId(\"656\u202604\") model_dump(by_alias=True) \"profile\": ObjectId(\"656\u202604\") encode_value(...) \"...\\\"profile\\\": \\\"656\u202604\\\"...\" \u2014 hex string decode_value(...) \"profile\": \"656\u202604\" \u2014 still a hex string Verbatim (timestamps abbreviated):
Pythonfrom mongo_ops.cache.in_memory import decode_value, encode_value\n\nsnapshot = {\"id\": \"656\u202605\", \"profile\": ObjectId(\"656\u202604\")} # what a cached raw doc looks like\n\ncached = encode_value(snapshot) # json.dumps(snapshot, default=str).encode()\ndecoded = decode_value(cached)\n\ndecoded[\"profile\"] # '656\u202604' \u2014 hex STRING, not ObjectId, not Profile\nTyped-FK caveat: the round trip above only works when the FK field's model type accepts the raw value (str/ObjectId typed). A field typed Profile | None cannot even be materialized from the raw Mongo doc \u2014 both hit and miss raise ValidationError (UC 16's intricacy). That is exactly why UC 10's composed repo overrides get_by_id/create/update to carry raw data through the cache and resolve refs with _populate.
Two consequences (the \"intricacies\", detailed in UC 16):
decode_value gives you hex-string references. A field typed Profile | None receiving a hex string raises Pydantic ValidationError \u2014 so a plain CachedBaseRepository cannot serve model-typed FK fields, hit or miss. The composed pattern in UC 10 re-resolves them in its get_by_id: data = decode_value(await cache.get(key)), then populated = await self._populate(data), then self.model(**populated).BaseRepository.get_by_id reads the raw Mongo doc whose profile is an ObjectId and immediately raises the same ValidationError \u2014 which is why UC 10 overrides get_by_id to fetch raw, cache raw, then populate. (Real ObjectIds are covered by UC 16's table.)ObjectId store, model-struct reads PopulatingRepository (UC 08) Deep nested reads (2+ levels) PopulatingRepository + nested_rules (UC 09) \u2014 read side only Same shape, but cache-first reads UC 10 composed repo (engine + cache + _populate) Scalar docs, no refs, hot get_by_id CachedBaseRepository (UC 07, UC 16) Writing a reference graph back depopulate without nested_rules \u2014 save children first, then point at their ids"},{"location":"03_use_cases/17_populate_depopulate_examples/#related","title":"Related","text":"Scenario: A reporting dashboard needs grouped statistics, computed fields, and joined data \u2014 operations that don't map to single-document CRUD.
"},{"location":"03_use_cases/18_aggregation_pipelines/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Descriptioncollection.aggregate() Raw Motor aggregation \u2014 no base-repo helper exists for pipelines. Repository wrapper Keeps pipeline logic behind a domain method so callers never touch the collection directly. $group, $lookup, $project Common pipeline stages demonstrated in a single repository."},{"location":"03_use_cases/18_aggregation_pipelines/#example","title":"\ud83d\ude80 Example","text":"Pythonfrom fastapi import FastAPI\nfrom mongo_ops import BaseDocument, BaseRepository\n\n\n# ---------------------------\n# Models\n# ---------------------------\nclass Order(BaseDocument):\n user_id: str\n status: str = \"pending\"\n total: float = 0.0\n\n\nclass OrderRepository(BaseRepository[Order]):\n def __init__(self):\n super().__init__(\"orders\", Order)\n\n async def revenue_by_status(self) -> list[dict]:\n \"\"\"Group orders by status, sum totals.\"\"\"\n pipeline = [\n {\"$group\": {\"_id\": \"$status\", \"revenue\": {\"$sum\": \"$total\"}, \"count\": {\"$sum\": 1}}},\n {\"$sort\": {\"revenue\": -1}},\n ]\n return await self.collection.aggregate(pipeline).to_list(length=100)\n\n async def top_spenders(self, limit: int = 10) -> list[dict]:\n \"\"\"Top N users by total spend, with a lookup to resolve user names.\"\"\"\n pipeline = [\n {\"$group\": {\"_id\": \"$user_id\", \"total_spent\": {\"$sum\": \"$total\"}, \"order_count\": {\"$sum\": 1}}},\n {\"$sort\": {\"total_spent\": -1}},\n {\"$limit\": limit},\n {\"$lookup\": {\n \"from\": \"users\",\n \"localField\": \"_id\",\n \"foreignField\": \"_id\",\n \"as\": \"user\",\n }},\n {\"$unwind\": {\"path\": \"$user\", \"preserveNullAndEmptyArrays\": True}},\n {\"$project\": {\n \"total_spent\": 1,\n \"order_count\": 1,\n \"user_name\": {\"$ifNull\": [\"$user.username\", \"unknown\"]},\n }},\n ]\n return await self.collection.aggregate(pipeline).to_list(length=limit)\n\n async def daily_summary(self, days: int = 30) -> list[dict]:\n \"\"\"Order counts grouped by day for the last N days.\"\"\"\n from datetime import datetime, timedelta\n\n cutoff = datetime.utcnow() - timedelta(days=days)\n pipeline = [\n {\"$match\": {\"created_at\": {\"$gte\": cutoff}}},\n {\"$group\": {\n \"_id\": {\"$dateToString\": {\"format\": \"%Y-%m-%d\", \"date\": \"$created_at\"}},\n \"orders\": {\"$sum\": 1},\n \"revenue\": {\"$sum\": \"$total\"},\n }},\n {\"$sort\": {\"_id\": 1}},\n ]\n return await self.collection.aggregate(pipeline).to_list(length=days)\n\n\napp = FastAPI()\norder_repo = OrderRepository()\n\n\n@app.get(\"/reports/revenue-by-status\")\nasync def revenue_by_status():\n return await order_repo.revenue_by_status()\n\n\n@app.get(\"/reports/top-spenders\")\nasync def top_spenders(limit: int = 10):\n return await order_repo.top_spenders(limit)\n\n\n@app.get(\"/reports/daily-summary\")\nasync def daily_summary(days: int = 30):\n return await order_repo.daily_summary(days)\nNote: lifespan wiring omitted \u2014 copy from use case 01.
"},{"location":"03_use_cases/18_aggregation_pipelines/#tips","title":"\ud83d\udca1 Tips","text":"$lookup sparingly; for read-heavy workloads, prefer application-level population via use case 08 which benefits from the cache.$match and $sort stages \u2014 otherwise the aggregation runs a collection scan (see use case 13).Scenario: A background job needs to update or delete hundreds of documents in one call \u2014 doing it one-by-one would be prohibitively slow.
"},{"location":"03_use_cases/19_bulk_operations/#whats-new","title":"\ud83d\udce6 What's New?","text":"Component Descriptionupdate_many / delete_many Motor's multi-document write operations. Repository wrapper Keeps bulk logic behind a domain method so callers pass IDs and business parameters, not raw dicts. BulkWriteError Raised when ordered=False and some operations fail \u2014 catch it for partial-success handling."},{"location":"03_use_cases/19_bulk_operations/#example","title":"\ud83d\ude80 Example","text":"Pythonfrom datetime import datetime\n\nfrom bson import ObjectId, BulkWriteError\nfrom fastapi import FastAPI, HTTPException\nfrom mongo_ops import BaseDocument, BaseRepository\n\n\n# ---------------------------\n# Model\n# ---------------------------\nclass Task(BaseDocument):\n status: str = \"pending\"\n assignee_id: str | None = None\n priority: int = 0\n\n\n# ---------------------------\n# Repository\n# ---------------------------\nclass TaskRepository(BaseRepository[Task]):\n def __init__(self):\n super().__init__(\"tasks\", Task)\n\n async def bulk_update_status(self, ids: list[str], status: str) -> int:\n \"\"\"Set status on many tasks at once. Returns the count of modified documents.\"\"\"\n object_ids = [ObjectId(i) for i in ids]\n result = await self.collection.update_many(\n {\"_id\": {\"$in\": object_ids}},\n {\"$set\": {\"status\": status, \"updated_at\": datetime.utcnow()}},\n )\n return result.modified_count\n\n async def bulk_reassign(self, old_assignee: str, new_assignee: str) -> int:\n \"\"\"Move all pending tasks from one assignee to another.\"\"\"\n result = await self.collection.update_many(\n {\"assignee_id\": old_assignee, \"status\": \"pending\"},\n {\"$set\": {\"assignee_id\": new_assignee, \"updated_at\": datetime.utcnow()}},\n )\n return result.modified_count\n\n async def bulk_archive(self, ids: list[str]) -> int:\n \"\"\"Delete many tasks at once. Returns the count of deleted documents.\"\"\"\n object_ids = [ObjectId(i) for i in ids]\n result = await self.collection.delete_many({\"_id\": {\"$in\": object_ids}})\n return result.deleted_count\n\n async def bulk_upsert_users(self, records: list[dict]) -> dict:\n \"\"\"Insert-or-update many user records. Returns inserted/modified counts.\"\"\"\n from pymongo import UpdateOne\n\n ops = [\n UpdateOne(\n {\"email\": rec[\"email\"]},\n {\"$set\": rec},\n upsert=True,\n )\n for rec in records\n ]\n try:\n result = await self.collection.bulk_write(ops, ordered=False)\n return {\n \"inserted\": result.upserted_count,\n \"modified\": result.modified_count,\n }\n except BulkWriteError as exc:\n return {\n \"inserted\": exc.details.get(\"upsertedCount\", 0),\n \"modified\": exc.details.get(\"modifiedCount\", 0),\n \"errors\": exc.details.get(\"writeErrors\", []),\n }\n\n\napp = FastAPI()\ntask_repo = TaskRepository()\n\n\n@app.post(\"/tasks/bulk-status\")\nasync def bulk_status(ids: list[str], status: str):\n count = await task_repo.bulk_update_status(ids, status)\n return {\"modified\": count}\n\n\n@app.post(\"/tasks/bulk-reassign\")\nasync def reassign(old_assignee: str, new_assignee: str):\n count = await task_repo.bulk_reassign(old_assignee, new_assignee)\n return {\"modified\": count}\n\n\n@app.post(\"/tasks/bulk-archive\")\nasync def archive(ids: list[str]):\n count = await task_repo.bulk_archive(ids)\n return {\"deleted\": count}\nNote: lifespan wiring omitted \u2014 copy from use case 01.
"},{"location":"03_use_cases/19_bulk_operations/#tips","title":"\ud83d\udca1 Tips","text":"invalidate_cache(id) for each affected document, or reinitialize the cache (see use case 11).ordered=False for upserts so one bad record doesn't block the rest. Catch BulkWriteError and surface the per-operation errors.bulkWrite payload limit.update_many / delete_many require a {\"_id\": {\"$in\": [...]}} filter to scope the operation \u2014 never pass an unscoped filter.