Vishesh 'ironeagle' Bangotra
a32eeaf2b4
- collect.py copies each repo's built site contents directly to a top-level <repo>/ dir (was libs|apis|wiki/<repo>/site nesting) - nginx uses `index index.html lib/index.html api/index.html` so /dagpipe/ -> lib/index.html, /auth-server/ -> api/index.html are served internally with no redirects - _index regenerates links against the flat layout (/dagpipe/lib/, /auth-server/api/, /mongo-ops/, /blog/...) - config.yml static entries point at vendored blog/ + media-manager/ - Dockerfile copies per-repo dirs flat into the nginx html root - removed stale libs/, apis/, wiki/, tutorials/ category trees
1 line
67 KiB
JSON
1 line
67 KiB
JSON
{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"\ud83e\udde9 mongo-ops \u2014 Async MongoDB Operations Layer for FastAPI","text":"<p><code>mongo-ops</code> is a modular, high-performance MongoDB operations library designed for FastAPI microservices. It standardizes repository patterns, async CRUD operations, and model management \u2014 with extensible components for both Beanie and Motor.</p>"},{"location":"#key-features","title":"\ud83d\ude80 Key Features","text":"<ul> <li>\ud83e\uddf1 Unified repository pattern for MongoDB collections</li> <li>\u26a1 Fully asynchronous (Motor-based)</li> <li>\ud83e\uddec Pydantic v2 data model integration</li> <li>\ud83e\uddf0 Built-in CRUD and aggregation utilities</li> <li>\ud83d\udd12 Transaction and session helpers</li> <li>\ud83e\udde9 Optional Beanie ORM integration</li> <li>\ud83e\uddea Pytest-friendly architecture</li> </ul>"},{"location":"#installation","title":"\ud83d\udce6 Installation","text":"<p>From your internal PyPI:</p> Bash<pre><code>pip install --extra-index-url https://$PYPI_USERNAME:$PYPI_PASSWORD@pip.aetoskia.com/simple mongo-ops\n</code></pre> <p>From local source:</p> Bash<pre><code>pip install -e .\n</code></pre>"},{"location":"#documentation-structure","title":"\ud83d\udcc1 Documentation Structure","text":"Section Description Overview Core concept and architecture overview Core Components <code>BaseDocument</code>, <code>MongoConnectionManager</code>, <code>BaseRepository</code> Use Cases CRUD, transactions, pagination, and more Best Practices Recommended repo structure and patterns Common Patterns Service layer, aggregation, soft deletes Error Handling Centralized error and exception management Testing Pytest configuration and fixtures"},{"location":"#related-resources","title":"\ud83d\udd17 Related Resources","text":"<ul> <li>Source Code: Gitea Repository</li> <li>Internal PyPI: pip.aetoskia.com/simple/mongo-ops</li> <li>Drone CI: Auto-builds and publishes tagged releases.</li> </ul> <p>\u00a9 Aetoskia Internal \u2014 <code>mongo-ops</code> 0.1.4</p>"},{"location":"01_overview/","title":"Overview","text":""},{"location":"01_overview/#library-overview","title":"Library Overview","text":"<p><code>mongo-ops</code> is a modular MongoDB operations layer for FastAPI microservices. It provides:</p> <ul> <li>Async connection management</li> <li>Base document models with auto-timestamps</li> <li>Generic CRUD repository pattern</li> <li>Transaction support</li> <li>Model registration system</li> </ul>"},{"location":"02_components/","title":"02 components","text":""},{"location":"02_components/#core-components","title":"Core Components","text":""},{"location":"02_components/#1-mongoconnectionmanager","title":"1. MongoConnectionManager","text":"<p>Manages MongoDB connections with async lifecycle. Methods:</p> <ul> <li><code>connect(uri, db_name, **kwargs)</code> - Connect to MongoDB</li> <li><code>disconnect()</code> - Close connection</li> <li><code>get_database()</code> - Get current database instance</li> <li><code>get_client()</code> - Get current client instance</li> <li><code>lifespan(uri, db_name, **kwargs)</code> - Context manager for FastAPI lifespan</li> </ul>"},{"location":"02_components/#2-basedocument","title":"2. BaseDocument","text":"<p>Base model for all MongoDB documents. Provides:</p> <ul> <li><code>id</code> (aliased to <code>_id</code>) - ObjectId</li> <li><code>created_at</code> - Auto-generated timestamp</li> <li><code>updated_at</code> - Auto-updated timestamp</li> </ul>"},{"location":"02_components/#3-baserepositoryt","title":"3. BaseRepository[T]","text":"<p>Generic repository with CRUD operations:</p> <ul> <li><code>create(data: T) -> T</code></li> <li><code>get_by_id(id: str | ObjectId) -> Optional[T]</code></li> <li><code>get_many(filter, skip, limit, sort) -> List[T]</code></li> <li><code>update(id, data: Dict) -> Optional[T]</code></li> <li><code>delete(id) -> bool</code></li> <li><code>count(filter) -> int</code></li> </ul>"},{"location":"02_components/#4-transactionmanager","title":"4. TransactionManager","text":"<p>Handles multi-document transactions:</p> <ul> <li><code>start_session()</code> - Context manager for transactions</li> <li><code>execute_transaction(operations)</code> - Execute multiple operations atomically</li> </ul>"},{"location":"02_components/#5-modelregistry","title":"5. ModelRegistry","text":"<p>Register and initialize models:</p> <ul> <li><code>register(collection_name, model, indexes)</code> - Register a model</li> <li><code>initialize_all()</code> - Create all indexes</li> <li><code>get_model(collection_name)</code> - Get registered model</li> <li><code>list_collections()</code> - List all registered collections</li> </ul>"},{"location":"02_components/#6-cache-backend","title":"6. Cache Backend","text":""},{"location":"02_components/#61-cachebackend-abstract","title":"6.1 CacheBackend (abstract)","text":"<ul> <li>Defines the async interface for cache operations: <code>get</code>, <code>set</code>, <code>delete</code>, <code>exists</code>, <code>clear_pattern</code>, <code>get_stats</code>, <code>initialize</code>, <code>shutdown</code>.</li> <li>Provides <code>CacheStats</code> for hit/miss/size metrics and <code>CircularReferenceError</code> for cycle detection.</li> </ul>"},{"location":"02_components/#62-cacheconfig","title":"6.2 CacheConfig","text":"<ul> <li>Dataclass to configure caching (<code>enabled</code>, <code>backend</code>, <code>redis_client</code>, <code>default_ttl</code>, <code>max_entries</code>, <code>key_prefix</code>, <code>cleanup_interval</code>).</li> </ul>"},{"location":"02_components/#63-inmemorycachebackend","title":"6.3 InMemoryCacheBackend","text":"<ul> <li>In\u2011process cache using an LRU <code>OrderedDict</code> and a TTL heap.</li> <li>Background task periodically evicts expired entries.</li> </ul>"},{"location":"02_components/#64-rediscachebackend","title":"6.4 RedisCacheBackend","text":"<ul> <li>Distributed cache based on <code>redis.asyncio</code>.</li> <li>JSON serialisation, optional pub/sub invalidation.</li> </ul>"},{"location":"02_components/#65-cachedbaserepositoryt","title":"6.5 CachedBaseRepository[T]","text":"<ul> <li>Extends <code>BaseRepository</code> with transparent ID\u2011based caching.</li> <li>Methods:</li> <li><code>get_by_id</code> \u2013 cache\u2011first lookup.</li> <li><code>create</code> \u2013 stores newly created doc in cache.</li> <li><code>update</code> / <code>delete</code> \u2013 invalidate or refresh cache.</li> <li><code>warm_cache(ids)</code> \u2013 pre\u2011load a list of IDs.</li> <li><code>invalidate_cache(id)</code> \u2013 manual invalidation.</li> </ul>"},{"location":"02_components/#7-population-engine","title":"7. Population Engine","text":""},{"location":"02_components/#71-populaterule","title":"7.1 PopulateRule","text":"<ul> <li>Dataclass defining a population rule: <code>field_name</code>, <code>collection_name</code>, <code>ref_field</code>, optional <code>nested_rules</code>, <code>max_depth</code>, <code>filter</code>, <code>projection</code>.</li> </ul>"},{"location":"02_components/#72-populationengine","title":"7.2 PopulationEngine","text":"<ul> <li>Recursively resolves references according to <code>PopulateRule</code> list.</li> <li>Detects circular references and raises <code>CircularReferenceError</code>.</li> <li>Supports per\u2011rule depth limits, filters and projections.</li> </ul>"},{"location":"02_components/#73-populatingrepositoryt","title":"7.3 PopulatingRepository[T]","text":"<ul> <li>Extends <code>BaseRepository</code> to automatically populate related documents.</li> <li>Accepts an optional <code>PopulationEngine</code> and a list of <code>PopulateRule</code>.</li> <li>Provides <code>_populate</code> and <code>_depopulate</code> helpers used in <code>get_by_id</code> and elsewhere.</li> </ul>"},{"location":"04_best_practices/","title":"Best Practices","text":""},{"location":"04_best_practices/#best-practices","title":"Best Practices","text":"<ol> <li>Always use ModelRegistry.register() before initializing the database connection</li> <li>Use lifespan context manager for proper connection lifecycle</li> <li>Inherit from BaseDocument for all models to get auto-timestamps</li> <li>Create custom repository classes for business logic instead of mixing it with models</li> <li>Use transactions for operations that modify multiple documents</li> <li>Add indexes during model registration for frequently queried fields</li> <li>Implement pagination for list endpoints to avoid performance issues</li> <li>Use type hints for better IDE support and type checking</li> </ol>"},{"location":"05_patterns/","title":"Common Patterns","text":""},{"location":"05_patterns/#common-patterns","title":"Common Patterns","text":""},{"location":"05_patterns/#pattern-1-service-layer-with-repository","title":"Pattern 1: Service Layer with Repository","text":"Python<pre><code>class UserService:\n def __init__(self, user_repo: UserRepository):\n self.user_repo = user_repo\n\n async def register_user(self, username: str, email: str) -> User:\n # Check if user exists\n existing = await self.user_repo.collection.find_one({\"email\": email})\n if existing:\n raise ValueError(\"Email already exists\")\n\n # Create user\n user = User(username=username, email=email)\n return await self.user_repo.create(user)\n</code></pre>"},{"location":"05_patterns/#pattern-2-aggregation-pipeline","title":"Pattern 2: Aggregation Pipeline","text":"Python<pre><code>async def get_user_stats(self, user_id: str) -> dict:\n pipeline = [\n {\"$match\": {\"author_id\": user_id}},\n {\"$group\": {\n \"_id\": None,\n \"total_posts\": {\"$sum\": 1},\n \"total_likes\": {\"$sum\": \"$likes\"}\n }}\n ]\n result = await post_repo.collection.aggregate(pipeline).to_list(1)\n return result[0] if result else {\"total_posts\": 0, \"total_likes\": 0}\n</code></pre>"},{"location":"05_patterns/#pattern-3-bulk-operations","title":"Pattern 3: Bulk Operations","text":"Python<pre><code>async def bulk_update_status(self, ids: List[str], status: str):\n from bson import ObjectId\n object_ids = [ObjectId(id) for id in ids]\n await self.collection.update_many(\n {\"_id\": {\"$in\": object_ids}},\n {\"$set\": {\"status\": status, \"updated_at\": datetime.utcnow()}}\n )\n</code></pre>"},{"location":"06_error_handling/","title":"Error Handling","text":""},{"location":"06_error_handling/#error-handling","title":"Error Handling","text":"Python<pre><code>from fastapi import HTTPException\nfrom pymongo.errors import DuplicateKeyError\nfrom bson.errors import InvalidId\n\n@app.post(\"/users/\")\nasync def create_user(user: User):\n try:\n return await user_repo.create(user)\n except DuplicateKeyError:\n raise HTTPException(status_code=409, detail=\"User already exists\")\n except Exception as e:\n raise HTTPException(status_code=500, detail=str(e))\n\n@app.get(\"/users/{user_id}\")\nasync def get_user(user_id: str):\n try:\n user = await user_repo.get_by_id(user_id)\n if not user:\n raise HTTPException(status_code=404, detail=\"User not found\")\n return user\n except InvalidId:\n raise HTTPException(status_code=400, detail=\"Invalid user ID format\")\n</code></pre>"},{"location":"07_testing_example/","title":"Testing Example","text":""},{"location":"07_testing_example/#testing-example","title":"Testing Example","text":"Python<pre><code>import pytest\nfrom mongo_ops import (\n MongoConnectionManager,\n BaseDocument,\n BaseRepository,\n)\nfrom pydantic import Field\nfrom typing import Optional\n\n# Define Model\nclass User(BaseDocument):\n username: str = Field(..., min_length=3, max_length=50)\n email: str = Field(...)\n is_active: bool = True\n\n# Create Repository\nclass UserRepository(BaseRepository[User]):\n def __init__(self):\n super().__init__(\"users\", User)\n\n\n@pytest.fixture\nasync def db():\n await MongoConnectionManager.connect(\n uri=\"mongodb://localhost:27017\",\n db_name=\"test_db\"\n )\n yield MongoConnectionManager.get_database()\n await MongoConnectionManager.get_client().drop_database(\"test_db\")\n await MongoConnectionManager.disconnect()\n\n@pytest.mark.asyncio\nasync def test_create_user(db):\n repo = UserRepository()\n user = await repo.create(User(username=\"test\", email=\"test@example.com\"))\n assert user.id is not None\n assert user.username == \"test\"\n\n@pytest.mark.asyncio\nasync def test_get_by_id(db):\n repo = UserRepository()\n created = await repo.create(User(username=\"test\", email=\"test@example.com\"))\n fetched = await repo.get_by_id(str(created.id))\n assert fetched.username == created.username\n</code></pre>"},{"location":"03_use_cases/01_basic_crud/","title":"Basic CRUD","text":""},{"location":"03_use_cases/01_basic_crud/#use-case-1-basic-fastapi-crud-api","title":"Use Case 1: Basic FastAPI CRUD API","text":"<p>Scenario: Create a simple user management API with CRUD operations.</p> Python<pre><code>import os\nfrom fastapi import FastAPI, Depends, HTTPException\nfrom contextlib import asynccontextmanager\nfrom pydantic import Field\nfrom mongo_ops import (\n MongoConnectionManager,\n BaseDocument,\n BaseRepository,\n ModelRegistry,\n)\n\n# ---------------------------\n# Model Definition\n# ---------------------------\nclass User(BaseDocument):\n username: str = Field(..., min_length=3, max_length=50)\n email: str = Field(...)\n is_active: bool = True\n\n\n# ---------------------------\n# Repository\n# ---------------------------\nclass UserRepository(BaseRepository[User]):\n def __init__(self):\n super().__init__(\"users\", User)\n\n\n# Register the model only once\nModelRegistry.register(\"users\", User, indexes=[(\"email\", 1)])\n\n\n# ---------------------------\n# FastAPI Lifespan\n# ---------------------------\n@asynccontextmanager\nasync def lifespan(_app: FastAPI):\n \"\"\"Manage MongoDB connection during the app lifecycle.\"\"\"\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\",\n db_name=\"mydb\"\n ):\n await ModelRegistry.initialize_all()\n yield\n\n\napp = FastAPI(lifespan=lifespan)\n\n\n# ---------------------------\n# Dependency Injection\n# ---------------------------\ndef get_user_repository() -> UserRepository:\n \"\"\"Dependency-injected repository for users.\"\"\"\n return UserRepository()\n\n\n# ---------------------------\n# Routes\n# ---------------------------\n@app.post(\"/users/\", response_model=User)\nasync def create_user(user: User, repo: UserRepository = Depends(get_user_repository)):\n return await repo.create(user)\n\n\n@app.get(\"/users/{user_id}\", response_model=User)\nasync def get_user(user_id: str, repo: UserRepository = Depends(get_user_repository)):\n user = await repo.get_by_id(user_id)\n if not user:\n raise HTTPException(status_code=404, detail=\"User not found\")\n return user\n\n\n@app.get(\"/users/\", response_model=list[User])\nasync def list_users(\n skip: int = 0,\n limit: int = 10,\n repo: UserRepository = Depends(get_user_repository),\n):\n return await repo.get_many(skip=skip, limit=limit)\n\n\n@app.put(\"/users/{user_id}\", response_model=User)\nasync def update_user(\n user_id: str,\n email: str,\n repo: UserRepository = Depends(get_user_repository),\n):\n user = await repo.update(user_id, {\"email\": email})\n if not user:\n raise HTTPException(status_code=404, detail=\"User not found\")\n return user\n\n\n@app.delete(\"/users/{user_id}\")\nasync def delete_user(\n user_id: str,\n repo: UserRepository = Depends(get_user_repository),\n):\n deleted = await repo.delete(user_id)\n if not deleted:\n raise HTTPException(status_code=404, detail=\"User not found\")\n return {\"message\": \"User deleted successfully\"}\n</code></pre>"},{"location":"03_use_cases/02_custom_repo/","title":"Custom Repo","text":""},{"location":"03_use_cases/02_custom_repo/#use-case-2-custom-repository-with-business-logic","title":"Use Case 2: Custom Repository with Business Logic","text":"<p>Scenario: E-commerce product catalog with custom search and filtering.</p> Python<pre><code>from mongo_ops import BaseDocument, BaseRepository\nfrom typing import Optional, List\nfrom pydantic import Field\n\nclass Product(BaseDocument):\n name: str = Field(..., min_length=1)\n description: str\n price: float = Field(..., gt=0)\n category: str\n in_stock: bool = True\n quantity: int = Field(default=0, ge=0)\n tags: List[str] = []\n\nclass ProductRepository(BaseRepository[Product]):\n def __init__(self):\n super().__init__(\"products\", Product)\n\n async def search_by_name(self, query: str) -> List[Product]:\n \"\"\"Search products by name (case-insensitive)\"\"\"\n docs = await self.collection.find({\n \"name\": {\"$regex\": query, \"$options\": \"i\"}\n }).to_list(length=100)\n return [self.model(**doc) for doc in docs]\n\n async def get_by_category(self, category: str, in_stock_only: bool = True) -> List[Product]:\n \"\"\"Get products by category\"\"\"\n filter_query = {\"category\": category}\n if in_stock_only:\n filter_query[\"in_stock\"] = True\n return await self.get_many(filter=filter_query)\n\n async def get_low_stock(self, threshold: int = 10) -> List[Product]:\n \"\"\"Get products with low stock\"\"\"\n return await self.get_many(\n filter={\"quantity\": {\"$lt\": threshold}, \"in_stock\": True}\n )\n\n async def update_stock(self, product_id: str, quantity_delta: int) -> Optional[Product]:\n \"\"\"Update product stock (increment/decrement)\"\"\"\n result = await self.collection.find_one_and_update(\n {\"_id\": ObjectId(product_id)},\n {\"$inc\": {\"quantity\": quantity_delta}, \"$set\": {\"updated_at\": datetime.utcnow()}},\n return_document=True\n )\n return self.model(**result) if result else None\n\n# Usage in FastAPI\nfrom fastapi import FastAPI, Query\n\napp = FastAPI()\nproduct_repo = ProductRepository()\n\n@app.get(\"/products/search\", response_model=List[Product])\nasync def search_products(q: str = Query(..., min_length=1)):\n return await product_repo.search_by_name(q)\n\n@app.get(\"/products/category/{category}\", response_model=List[Product])\nasync def products_by_category(category: str, in_stock: bool = True):\n return await product_repo.get_by_category(category, in_stock)\n\n@app.get(\"/products/low-stock\", response_model=List[Product])\nasync def low_stock_products(threshold: int = 10):\n return await product_repo.get_low_stock(threshold)\n\n@app.patch(\"/products/{product_id}/stock\")\nasync def update_product_stock(product_id: str, quantity_delta: int):\n product = await product_repo.update_stock(product_id, quantity_delta)\n if not product:\n raise HTTPException(status_code=404, detail=\"Product not found\")\n return product\n</code></pre>"},{"location":"03_use_cases/03_transactions/","title":"Transactions","text":""},{"location":"03_use_cases/03_transactions/#use-case-3-transaction-support-for-multi-document-operations","title":"Use Case 3: Transaction Support for Multi-Document Operations","text":"<p>Scenario: Order processing system that updates inventory and creates order atomically.</p> Python<pre><code>from mongo_ops import BaseDocument, BaseRepository, TransactionManager\nfrom datetime import datetime\nfrom typing import List\nfrom pydantic import Field\n\nclass Order(BaseDocument):\n user_id: str\n items: List[dict] # [{\"product_id\": \"...\", \"quantity\": 2}]\n total_amount: float\n status: str = \"pending\"\n\nclass OrderRepository(BaseRepository[Order]):\n def __init__(self):\n super().__init__(\"orders\", Order)\n\nclass OrderService:\n def __init__(self, order_repo: OrderRepository, product_repo: ProductRepository):\n self.order_repo = order_repo\n self.product_repo = product_repo\n\n async def create_order_with_inventory_update(self, order: Order) -> Order:\n \"\"\"Create order and update inventory atomically\"\"\"\n\n async def create_order_transaction(session):\n # Insert order\n order_doc = order.model_dump(exclude={\"id\"}, exclude_none=True)\n order_doc[\"created_at\"] = datetime.utcnow()\n order_doc[\"updated_at\"] = datetime.utcnow()\n result = await self.order_repo.collection.insert_one(order_doc, session=session)\n\n # Update inventory for each item\n for item in order.items:\n await self.product_repo.collection.update_one(\n {\"_id\": ObjectId(item[\"product_id\"])},\n {\n \"$inc\": {\"quantity\": -item[\"quantity\"]},\n \"$set\": {\"updated_at\": datetime.utcnow()}\n },\n session=session\n )\n\n # Fetch created order\n created = await self.order_repo.collection.find_one(\n {\"_id\": result.inserted_id},\n session=session\n )\n return Order(**created)\n\n # Execute in transaction\n async with TransactionManager.start_session() as session:\n return await create_order_transaction(session)\n\n# Usage in FastAPI\n@app.post(\"/orders/\", response_model=Order)\nasync def create_order(order: Order):\n order_service = OrderService(order_repo, product_repo)\n try:\n return await order_service.create_order_with_inventory_update(order)\n except Exception as e:\n raise HTTPException(status_code=400, detail=str(e))\n</code></pre>"},{"location":"03_use_cases/04_pagination/","title":"Pagination","text":""},{"location":"03_use_cases/04_pagination/#use-case-4-pagination-filtering","title":"Use Case 4: Pagination & Filtering","text":"<p>Scenario: Blog post API with pagination and filtering.</p> Python<pre><code>from mongo_ops import BaseDocument, BaseRepository\nfrom pydantic import BaseModel, Field\nfrom typing import List, Optional, Generic, TypeVar\n\nT = TypeVar(\"T\")\n\nclass PaginatedResponse(BaseModel, Generic[T]):\n items: List[T]\n total: int\n page: int\n page_size: int\n total_pages: int\n has_next: bool\n has_prev: bool\n\nclass BlogPost(BaseDocument):\n title: str\n content: str\n author_id: str\n published: bool = False\n tags: List[str] = []\n views: int = 0\n\nclass BlogPostRepository(BaseRepository[BlogPost]):\n def __init__(self):\n super().__init__(\"blog_posts\", BlogPost)\n\n async def paginate(\n self,\n page: int = 1,\n page_size: int = 10,\n filter_dict: Optional[dict] = None,\n sort_by: str = \"created_at\",\n sort_order: int = -1\n ) -> PaginatedResponse[BlogPost]:\n \"\"\"Get paginated blog posts\"\"\"\n filter_dict = filter_dict or {}\n skip = (page - 1) * page_size\n\n # Get total count\n total = await self.count(filter_dict)\n\n # Get items\n items = await self.get_many(\n filter=filter_dict,\n skip=skip,\n limit=page_size,\n sort=[(sort_by, sort_order)]\n )\n\n total_pages = (total + page_size - 1) // page_size\n\n return PaginatedResponse(\n items=items,\n total=total,\n page=page,\n page_size=page_size,\n total_pages=total_pages,\n has_next=page < total_pages,\n has_prev=page > 1\n )\n\n async def get_by_author(self, author_id: str, published_only: bool = True):\n \"\"\"Get posts by author\"\"\"\n filter_dict = {\"author_id\": author_id}\n if published_only:\n filter_dict[\"published\"] = True\n return await self.get_many(filter=filter_dict, sort=[(\"created_at\", -1)])\n\n async def search_by_tags(self, tags: List[str]) -> List[BlogPost]:\n \"\"\"Search posts by tags\"\"\"\n return await self.get_many(filter={\"tags\": {\"$in\": tags}, \"published\": True})\n\n# Usage in FastAPI\n@app.get(\"/posts/\", response_model=PaginatedResponse[BlogPost])\nasync def list_posts(\n page: int = 1,\n page_size: int = 10,\n published: Optional[bool] = None,\n author_id: Optional[str] = None\n):\n filter_dict = {}\n if published is not None:\n filter_dict[\"published\"] = published\n if author_id:\n filter_dict[\"author_id\"] = author_id\n\n return await blog_repo.paginate(page, page_size, filter_dict)\n\n@app.get(\"/posts/author/{author_id}\", response_model=List[BlogPost])\nasync def posts_by_author(author_id: str, published: bool = True):\n return await blog_repo.get_by_author(author_id, published)\n\n@app.get(\"/posts/tags\", response_model=List[BlogPost])\nasync def posts_by_tags(tags: List[str] = Query(...)):\n return await blog_repo.search_by_tags(tags)\n</code></pre>"},{"location":"03_use_cases/05_soft_deletes/","title":"Soft Deletes","text":""},{"location":"03_use_cases/05_soft_deletes/#use-case-5-soft-deletes-pattern","title":"Use Case 5: Soft Deletes Pattern","text":"<p>Scenario: Implement soft delete functionality for data recovery.</p> Python<pre><code>from mongo_ops import BaseDocument, BaseRepository\nfrom datetime import datetime\nfrom typing import Optional\n\nclass SoftDeleteDocument(BaseDocument):\n \"\"\"Base document with soft delete support\"\"\"\n is_deleted: bool = False\n deleted_at: Optional[datetime] = None\n deleted_by: Optional[str] = None\n\nclass Task(SoftDeleteDocument):\n title: str\n description: str\n assignee_id: str\n status: str = \"pending\"\n priority: str = \"medium\"\n\nclass SoftDeleteRepository(BaseRepository[T]):\n \"\"\"Repository with soft delete operations\"\"\"\n\n async def soft_delete(self, id: str, deleted_by: str = None) -> Optional[T]:\n \"\"\"Soft delete a document\"\"\"\n return await self.update(id, {\n \"is_deleted\": True,\n \"deleted_at\": datetime.utcnow(),\n \"deleted_by\": deleted_by\n })\n\n async def restore(self, id: str) -> Optional[T]:\n \"\"\"Restore a soft-deleted document\"\"\"\n return await self.update(id, {\n \"is_deleted\": False,\n \"deleted_at\": None,\n \"deleted_by\": None\n })\n\n async def get_active(self, skip: int = 0, limit: int = 100):\n \"\"\"Get only non-deleted documents\"\"\"\n return await self.get_many(\n filter={\"is_deleted\": False},\n skip=skip,\n limit=limit\n )\n\n async def get_deleted(self, skip: int = 0, limit: int = 100):\n \"\"\"Get deleted documents\"\"\"\n return await self.get_many(\n filter={\"is_deleted\": True},\n skip=skip,\n limit=limit\n )\n\n async def permanent_delete(self, id: str) -> bool:\n \"\"\"Permanently delete a document\"\"\"\n return await self.delete(id)\n\nclass TaskRepository(SoftDeleteRepository[Task]):\n def __init__(self):\n super().__init__(\"tasks\", Task)\n\n# Usage in FastAPI\n@app.delete(\"/tasks/{task_id}\")\nasync def soft_delete_task(task_id: str, user_id: str):\n task = await task_repo.soft_delete(task_id, deleted_by=user_id)\n if not task:\n raise HTTPException(status_code=404, detail=\"Task not found\")\n return {\"message\": \"Task deleted\", \"task\": task}\n\n@app.post(\"/tasks/{task_id}/restore\")\nasync def restore_task(task_id: str):\n task = await task_repo.restore(task_id)\n if not task:\n raise HTTPException(status_code=404, detail=\"Task not found\")\n return {\"message\": \"Task restored\", \"task\": task}\n\n@app.get(\"/tasks/\", response_model=List[Task])\nasync def list_active_tasks(skip: int = 0, limit: int = 10):\n return await task_repo.get_active(skip, limit)\n\n@app.get(\"/tasks/deleted\", response_model=List[Task])\nasync def list_deleted_tasks(skip: int = 0, limit: int = 10):\n return await task_repo.get_deleted(skip, limit)\n</code></pre>"},{"location":"03_use_cases/06_multi_model/","title":"Multi Model","text":""},{"location":"03_use_cases/06_multi_model/#use-case-6-multi-model-service-with-registration","title":"Use Case 6: Multi-Model Service with Registration","text":"<p>Scenario: Complete microservice with multiple related models.</p> Python<pre><code>from mongo_ops import (\n MongoConnectionManager,\n BaseDocument,\n BaseRepository,\n ModelRegistry\n)\nfrom fastapi import FastAPI\nfrom contextlib import asynccontextmanager\n\n# Define Models\nclass User(BaseDocument):\n username: str\n email: str\n role: str = \"user\"\n\nclass Post(BaseDocument):\n title: str\n content: str\n author_id: str\n likes: int = 0\n\nclass Comment(BaseDocument):\n post_id: str\n user_id: str\n content: str\n\n# Define Repositories\nclass UserRepository(BaseRepository[User]):\n def __init__(self):\n super().__init__(\"users\", User)\n\nclass PostRepository(BaseRepository[Post]):\n def __init__(self):\n super().__init__(\"posts\", Post)\n\nclass CommentRepository(BaseRepository[Comment]):\n def __init__(self):\n super().__init__(\"comments\", Comment)\n\n# Register all models\nModelRegistry.register(\"users\", User, indexes=[(\"email\", 1), (\"username\", 1)])\nModelRegistry.register(\"posts\", Post, indexes=[(\"author_id\", 1), (\"created_at\", -1)])\nModelRegistry.register(\"comments\", Comment, indexes=[(\"post_id\", 1), (\"user_id\", 1)])\n\n# Setup FastAPI with lifespan\n@asynccontextmanager\nasync def lifespan(app: FastAPI):\n await MongoConnectionManager.connect(\n uri=\"mongodb://localhost:27017\",\n db_name=\"social_app\"\n )\n await ModelRegistry.initialize_all()\n yield\n await MongoConnectionManager.disconnect()\n\napp = FastAPI(lifespan=lifespan)\n\n# Initialize repositories\nuser_repo = UserRepository()\npost_repo = PostRepository()\ncomment_repo = CommentRepository()\n\n# Endpoints\n@app.post(\"/users/\", response_model=User)\nasync def create_user(user: User):\n return await user_repo.create(user)\n\n@app.post(\"/posts/\", response_model=Post)\nasync def create_post(post: Post):\n return await post_repo.create(post)\n\n@app.post(\"/comments/\", response_model=Comment)\nasync def create_comment(comment: Comment):\n return await comment_repo.create(comment)\n</code></pre>"},{"location":"03_use_cases/07_caching/","title":"Use Case 7: Caching for High\u2011Performance Reads","text":"<p>Scenario: A read\u2011heavy API (e.g., product catalog) benefits from an in\u2011memory cache or Redis cache to reduce latency and DB load.</p>"},{"location":"03_use_cases/07_caching/#1-whats-new","title":"1\ufe0f\u20e3 What\u2019s New?","text":"<ul> <li>Cache back\u2011ends: <code>InMemoryCacheBackend</code> (TTL + LRU) and <code>RedisCacheBackend</code> (JSON\u2011serialised values, pub/sub invalidation).</li> <li>Cache configuration via <code>CacheConfig</code> (<code>enabled</code>, <code>backend</code>, <code>default_ttl</code>, \u2026).</li> <li><code>CachedBaseRepository</code> extends <code>BaseRepository</code> and adds:</li> <li>Transparent ID\u2011based caching on <code>get_by_id</code>.</li> <li>Automatic cache population on <code>create</code>.</li> <li>Cache invalidation on <code>update</code>/<code>delete</code>.</li> <li>Utility methods <code>warm_cache(ids)</code> and <code>invalidate_cache(id)</code>.</li> </ul>"},{"location":"03_use_cases/07_caching/#2-quick-start","title":"2\ufe0f\u20e3 Quick Start","text":"Python<pre><code>from mongo_ops import (\n BaseDocument,\n CachedBaseRepository,\n MongoConnectionManager,\n ModelRegistry,\n CacheConfig,\n)\nfrom mongo_ops.cache import InMemoryCacheBackend\nfrom bson import ObjectId\n\n# Define a model\nclass Product(BaseDocument):\n name: str\n price: float\n\n# Initialise cache backend (in\u2011memory example)\ncache = InMemoryCacheBackend(max_entries=10_000, default_ttl=300)\n# Register the cache so the registry can initialise it later\nModelRegistry.set_cache_backend(cache)\n\n# Repository with caching\nclass ProductRepo(CachedBaseRepository[Product]):\n def __init__(self):\n super().__init__(\n collection_name=\"products\",\n model=Product,\n cache_backend=cache,\n config=CacheConfig(enabled=True, backend=\"memory\")\n )\n\n# FastAPI lifespan \u2013 initialise DB and cache\nasync def lifespan(app):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\", db_name=\"shop\"\n ):\n await ModelRegistry.initialize_all()\n await ModelRegistry.initialize_cache() # Starts background cleanup, etc.\n yield\n</code></pre>"},{"location":"03_use_cases/07_caching/#3-using-the-repository","title":"3\ufe0f\u20e3 Using the Repository","text":"Python<pre><code>repo = ProductRepo()\n# Create \u2013 automatically caches the new document\nproduct = await repo.create(Product(name=\"Widget\", price=9.99))\n\n# Normal read \u2013 will hit the cache after the first DB fetch\nfetched = await repo.get_by_id(product.id)\n\n# Update \u2013 cache entry is refreshed\nawait repo.update(product.id, {\"price\": 8.99})\n\n# Delete \u2013 cache entry removed\nawait repo.delete(product.id)\n\n# Warm a set of IDs in advance (e.g., during a bulk load)\nawait repo.warm_cache([ObjectId(\"...\"), ObjectId(\"...\")])\n</code></pre>"},{"location":"03_use_cases/07_caching/#4-redis-backend-optional","title":"4\ufe0f\u20e3 Redis Backend (optional)","text":"<p>If you prefer a distributed cache, swap the backend:</p> Python<pre><code>from mongo_ops.cache import RedisCacheBackend\nfrom redis.asyncio import Redis\n\nredis_client = Redis(host=\"localhost\", port=6379)\nredis_backend = RedisCacheBackend(redis_client, key_prefix=\"prod:\")\nModelRegistry.set_cache_backend(redis_backend)\n</code></pre> <p>The repository code stays the same \u2013 just pass the <code>redis_backend</code> instance to <code>CachedBaseRepository</code>.</p>"},{"location":"03_use_cases/07_caching/#5-when-to-use-caching","title":"5\ufe0f\u20e3 When to Use Caching","text":"<ul> <li>Frequently accessed documents (e.g., product details, configuration settings).</li> <li>Low\u2011write\u2011to\u2011read ratios where cache invalidation cost is acceptable.</li> <li>Distributed deployments where a shared Redis cache syncs invalidations via pub/sub.</li> </ul>"},{"location":"03_use_cases/07_caching/#6-related-docs","title":"6\ufe0f\u20e3 Related Docs","text":"<ul> <li>Core Components \u2013 see <code>docs/02_components.md</code> for the <code>CacheBackend</code> abstraction.</li> <li>Best Practices \u2013 remember to call <code>ModelRegistry.initialize_cache()</code> after DB connection.</li> </ul> <p>Feel free to adapt the TTL, max entries, and backend to your workload.</p>"},{"location":"03_use_cases/08_population/","title":"Use Case 8: Document Population","text":"<p>Scenario: An API needs to return a single, denormalised JSON payload that includes related documents (e.g., a <code>User</code> with an embedded <code>Profile</code>, or an <code>Order</code> with its <code>LineItem</code>s) without the caller having to issue multiple requests.</p>"},{"location":"03_use_cases/08_population/#whats-new","title":"\ud83d\udce6 What\u2019s New?","text":"Component Description <code>PopulateRule</code> Declarative rule that tells the engine how to populate a field.Key attributes:\u2022 <code>field_name</code> \u2013 Target attribute on the source model.\u2022 <code>collection_name</code> \u2013 Collection that stores the referenced documents.\u2022 <code>ref_field</code> \u2013 Name of the reference field (normally an <code>ObjectId</code>).\u2022 Optional <code>nested_rules</code> for deep population, <code>max_depth</code> to bound recursion, and <code>filter</code> / <code>projection</code> for fine\u2011grained queries. <code>PopulationEngine</code> Core engine that resolves a list of <code>PopulateRule</code>s recursively. It:\u2022 Detects circular references and raises <code>CircularReferenceError</code>.\u2022 Honors per\u2011rule <code>max_depth</code> and a global <code>global_max_depth</code> (default\u202f10).\u2022 Supports MongoDB filters and projections for each populated relation. <code>PopulatingRepository</code> Extends <code>BaseRepository</code> with automatic population support. Stores a reference to a <code>PopulationEngine</code> and a list of <code>PopulateRule</code>s, exposing two helpers:\u2022 <code>_populate(document)</code> \u2013 Returns the same document with the requested relations populated.\u2022 <code>_depopulate(document)</code> \u2013 Strips populated fields (useful before serialisation). Cache\u2011aware population (optional) \u2013 The engine works with any <code>CacheBackend</code>. When used together with <code>CachedBaseRepository</code>, populated documents benefit from caching in the same way as regular CRUD results."},{"location":"03_use_cases/08_population/#quick-example","title":"\ud83d\ude80 Quick Example","text":"Python<pre><code>from bson import ObjectId\nfrom mongo_ops import BaseDocument, PopulatingRepository, ModelRegistry\nfrom mongo_ops.populate import PopulateRule, PopulationEngine\nfrom mongo_ops.cache import InMemoryCacheBackend\n\n# ----------------------------------------------------------------------\n# 1\ufe0f\u20e3 Define the data models\n# ----------------------------------------------------------------------\nclass Profile(BaseDocument):\n avatar_url: str = \"\"\n bio: str = \"\"\n\nclass User(BaseDocument):\n username: str = \"\"\n email: str = \"\"\n profile_id: ObjectId = None # Reference to Profile\n profile: Profile | None = None # Populated field (filled by engine)\n\n# ----------------------------------------------------------------------\n# 2\ufe0f\u20e3 (Optional) Set up a cache; helpful for deep graphs\n# ----------------------------------------------------------------------\ncache = InMemoryCacheBackend(max_entries=10_000, default_ttl=300)\nModelRegistry.set_cache_backend(cache)\n\n# ----------------------------------------------------------------------\n# 3\ufe0f\u20e3 Create a PopulationEngine \u2013 repositories will be registered later\n# ----------------------------------------------------------------------\nengine = PopulationEngine({})\n\n# ----------------------------------------------------------------------\n# 4\ufe0f\u20e3 Declare how the `profile` field should be populated\n# ----------------------------------------------------------------------\nprofile_rule = PopulateRule(\n field_name=\"profile\",\n collection_name=\"profiles\",\n ref_field=\"profile_id\",\n # No further nesting in this simple example\n)\n\n# ----------------------------------------------------------------------\n# 5\ufe0f\u20e3 Repository that uses the engine + rule\n# ----------------------------------------------------------------------\nclass UserRepository(PopulatingRepository[User]):\n def __init__(self):\n super().__init__(\n collection_name=\"users\",\n model=User,\n population_engine=engine,\n populate_rules=[profile_rule],\n )\n\n# ----------------------------------------------------------------------\n# 6\ufe0f\u20e3 Register the repository with the engine so it can resolve refs\n# ----------------------------------------------------------------------\nengine.register_repo(\"profiles\", PopulatingRepository[Profile](\"profiles\", Profile))\nengine.register_repo(\"users\", UserRepository())\n\n# ----------------------------------------------------------------------\n# 7\ufe0f\u20e3 FastAPI lifespan \u2013 initialise DB, indexes, and the cache\n# ----------------------------------------------------------------------\nfrom mongo_ops import MongoConnectionManager\nfrom contextlib import asynccontextmanager\n\n@asynccontextmanager\nasync def lifespan(app):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\",\n db_name=\"mydb\"\n ):\n await ModelRegistry.initialize_all()\n # If you set up a cache backend (see step 2), initialize it here\n await ModelRegistry.initialize_cache()\n yield\n</code></pre>"},{"location":"03_use_cases/09_advanced_population/","title":"Use Case 9: Nested Document Population & Circular\u2011Ref Handling","text":"<p>Scenario: A service must return an <code>Author</code> object with its <code>books</code> populated, and each <code>book</code> must include its <code>publisher</code>. The data model contains a possible circular reference (<code>Author.mentor_id</code> \u2192 another <code>Author</code>).</p>"},{"location":"03_use_cases/09_advanced_population/#whats-new","title":"\ud83d\udce6 What\u2019s New?","text":"Component Description <code>PopulateRule</code> (nested) Allows you to declare a hierarchy of rules. In this example we populate <code>books</code> on <code>Author</code>, then <code>publisher</code> on each <code>Book</code>. <code>max_depth</code> Prevents infinite recursion when circular references exist. Circular\u2011reference guard <code>PopulationEngine</code> raises <code>CircularReferenceError</code> if a cycle exceeds <code>max_depth</code> or is revisited."},{"location":"03_use_cases/09_advanced_population/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from bson import ObjectId\nfrom mongo_ops import BaseDocument, PopulatingRepository, ModelRegistry\nfrom mongo_ops.populate import PopulateRule, PopulationEngine\n\n# ----------------------------------------------------------------------\n# 1\ufe0f\u20e3 Define the data models\n# ----------------------------------------------------------------------\nclass Publisher(BaseDocument):\n name: str = \"\"\n country: str = \"\"\n\nclass Book(BaseDocument):\n title: str = \"\"\n publisher_id: ObjectId = None # Reference to Publisher\n publisher: Publisher | None = None\n\nclass Author(BaseDocument):\n name: str = \"\"\n book_ids: list[ObjectId] = [] # References to Book documents\n books: list[Book] | None = None\n mentor_id: ObjectId = None # Optional circular reference to another Author\n mentor: \"Author\" | None = None\n\n# ----------------------------------------------------------------------\n# 2\ufe0f\u20e3 Create a PopulationEngine\n# ----------------------------------------------------------------------\nengine = PopulationEngine({})\n\n# ----------------------------------------------------------------------\n# 3\ufe0f\u20e3 Declare nested rules\n# ----------------------------------------------------------------------\n# Populate the Publisher inside each Book\nbook_rule = PopulateRule(\n field_name=\"publisher\",\n collection_name=\"publishers\",\n ref_field=\"publisher_id\",\n)\n# Populate books on Author, nesting the book_rule\nauthor_rule = PopulateRule(\n field_name=\"books\",\n collection_name=\"books\",\n ref_field=\"book_ids\",\n nested_rules=[book_rule],\n max_depth=3, # safety net for deep graphs\n)\n# Optional mentor rule (demonstrates circular\u2011ref handling)\nmentor_rule = PopulateRule(\n field_name=\"mentor\",\n collection_name=\"authors\",\n ref_field=\"mentor_id\",\n max_depth=2,\n)\n\n# ----------------------------------------------------------------------\n# 4\ufe0f\u20e3 Repository that uses the engine + rules\n# ----------------------------------------------------------------------\nclass AuthorRepository(PopulatingRepository[Author]):\n def __init__(self):\n super().__init__(\n collection_name=\"authors\",\n model=Author,\n population_engine=engine,\n populate_rules=[author_rule, mentor_rule],\n )\n\n# ----------------------------------------------------------------------\n# 5\ufe0f\u20e3 Register all repositories with the engine\n# ----------------------------------------------------------------------\nengine.register_repo(\"publishers\", PopulatingRepository[Publisher](\"publishers\", Publisher))\nengine.register_repo(\"books\", PopulatingRepository[Book](\"books\", Book))\nengine.register_repo(\"authors\", AuthorRepository())\n\n# ----------------------------------------------------------------------\n# 6\ufe0f\u20e3 FastAPI lifespan \u2013 initialise DB & cache (if any)\n# ----------------------------------------------------------------------\nfrom mongo_ops import MongoConnectionManager\nfrom contextlib import asynccontextmanager\n\n@asynccontextmanager\nasync def lifespan(app):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\",\n db_name=\"library\",\n ):\n await ModelRegistry.initialize_all()\n await ModelRegistry.initialize_cache()\n yield\n</code></pre>"},{"location":"03_use_cases/09_advanced_population/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Use <code>max_depth</code> conservatively; a value of <code>2\u20113</code> is enough for most graphs.</li> <li>When you know there are no cycles, you can omit <code>max_depth</code> on nested rules.</li> <li>The <code>CircularReferenceError</code> includes the visited path, which helps debugging.</li> </ul>"},{"location":"03_use_cases/10_cache_and_population/","title":"Use Case 10: Caching + Population (Read\u2011Through + Populated Docs)","text":"<p>Scenario: A service needs fast reads of a <code>User</code> document and its related <code>Profile</code>. The repository should cache the final populated result so subsequent calls hit the cache directly.</p>"},{"location":"03_use_cases/10_cache_and_population/#whats-new","title":"\ud83d\udce6 What\u2019s New?","text":"Component Description <code>CachedBaseRepository</code> Provides transparent ID\u2011based caching for CRUD operations. <code>PopulatingRepository</code> Adds automatic population of referenced documents. Combined usage By inheriting from <code>CachedBaseRepository</code> and wiring a <code>PopulationEngine</code>, the repository caches the populated document, eliminating both DB and population overhead on cache hits."},{"location":"03_use_cases/10_cache_and_population/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from bson import ObjectId\nfrom mongo_ops import BaseDocument, CachedBaseRepository, ModelRegistry\nfrom mongo_ops.populate import PopulateRule, PopulationEngine\nfrom mongo_ops.cache import InMemoryCacheBackend\n\n# ----------------------------------------------------------------------\n# 1\ufe0f\u20e3 Define models\n# ----------------------------------------------------------------------\nclass Profile(BaseDocument):\n avatar_url: str = \"\"\n bio: str = \"\"\n\nclass User(BaseDocument):\n username: str = \"\"\n email: str = \"\"\n profile_id: ObjectId = None # Reference to Profile\n profile: Profile | None = None # Populated field\n\n# ----------------------------------------------------------------------\n# 2\ufe0f\u20e3 Initialise cache backend (in\u2011memory example)\n# ----------------------------------------------------------------------\ncache = InMemoryCacheBackend(max_entries=20_000, default_ttl=600)\nModelRegistry.set_cache_backend(cache)\n\n# ----------------------------------------------------------------------\n# 3\ufe0f\u20e3 Set up a PopulationEngine (repositories will be registered later)\n# ----------------------------------------------------------------------\nengine = PopulationEngine({})\nprofile_rule = PopulateRule(\n field_name=\"profile\",\n collection_name=\"profiles\",\n ref_field=\"profile_id\",\n)\n\n# ----------------------------------------------------------------------\n# 4\ufe0f\u20e3 Cached + Populating repository\n# ----------------------------------------------------------------------\nclass UserRepository(CachedBaseRepository[User]):\n def __init__(self):\n super().__init__(\n collection_name=\"users\",\n model=User,\n cache_backend=cache,\n config=None, # defaults to enabled=True, backend=\"memory\"\n )\n # Attach the population engine after the base repo is ready\n self.population_engine = engine\n self._populate_rules = [profile_rule]\n\n # Override get_by_id to include population before caching\n async def get_by_id(self, id):\n # First attempt cache lookup (as in CachedBaseRepository)\n cached = await super().get_by_id(id)\n if cached:\n return cached\n # Not in cache \u2013 fetch from DB and populate\n doc = await super().get_by_id(id) # this will hit the DB (no cache hit)\n if doc is None:\n return None\n # Populate related docs\n populated = await engine.populate(doc, self._populate_rules)\n # Store the fully populated result in cache for next call\n await self._cache.set(self._cache_key(id), populated)\n return populated\n\n# ----------------------------------------------------------------------\n# 5\ufe0f\u20e3 Register repositories with the engine for population resolution\n# ----------------------------------------------------------------------\nengine.register_repo(\"profiles\", PopulatingRepository[Profile](\"profiles\", Profile))\nengine.register_repo(\"users\", UserRepository())\n\n# ----------------------------------------------------------------------\n# 6\ufe0f\u20e3 FastAPI lifespan \u2013 initialise DB, create indexes, and start cache cleanup\n# ----------------------------------------------------------------------\nfrom mongo_ops import MongoConnectionManager\nfrom contextlib import asynccontextmanager\n\n@asynccontextmanager\nasync def lifespan(app):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\",\n db_name=\"app_db\",\n ):\n await ModelRegistry.initialize_all()\n await ModelRegistry.initialize_cache()\n yield\n</code></pre>"},{"location":"03_use_cases/10_cache_and_population/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>The example overrides <code>get_by_id</code> to keep the caching logic simple; you could also create a mixin that composes <code>CachedBaseRepository</code> and <code>PopulatingRepository</code>.</li> <li>Remember to call <code>await ModelRegistry.initialize_cache()</code> after the DB connection is ready; otherwise the background cleanup task will never start.</li> <li>Cache keys are generated by <code>CachedBaseRepository._cache_key(id)</code>. If you change the repository\u2019s <code>config.key_prefix</code>, the same prefix will be used for populated results.</li> </ul>"},{"location":"03_use_cases/11_cache_lifecycle/","title":"Use Case 11: Proper Cache Lifecycle in FastAPI","text":"<p>Scenario: A microservice uses <code>InMemoryCacheBackend</code> (or Redis) and needs to start the cache cleanup task when the app starts, then shut it down cleanly on termination.</p>"},{"location":"03_use_cases/11_cache_lifecycle/#whats-new","title":"\ud83d\udce6 What\u2019s New?","text":"Component Description <code>ModelRegistry.initialize_cache()</code> Starts the async background task for the registered cache backend. <code>ModelRegistry.shutdown_cache()</code> Gracefully stops the background task and releases resources. FastAPI lifespan integration Demonstrates where to call both init and shutdown methods."},{"location":"03_use_cases/11_cache_lifecycle/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from mongo_ops import MongoConnectionManager, ModelRegistry\nfrom mongo_ops.cache import InMemoryCacheBackend\nfrom contextlib import asynccontextmanager\n\n# Initialise a cache backend (in\u2011memory example)\ncache = InMemoryCacheBackend(max_entries=10_000, default_ttl=300)\nModelRegistry.set_cache_backend(cache)\n\n@asynccontextmanager\nasync def lifespan(app):\n # Start MongoDB connection and cache background task\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\",\n db_name=\"mydb\",\n ):\n await ModelRegistry.initialize_all() # create indexes\n await ModelRegistry.initialize_cache() # start cache cleanup\n yield\n # FastAPI will exit the `with` block here \u2013 clean up cache\n await ModelRegistry.shutdown_cache()\n</code></pre>"},{"location":"03_use_cases/11_cache_lifecycle/#why-a-separate-shutdown-step","title":"Why a separate shutdown step?","text":"<ul> <li>The cache backend may spawn an <code>asyncio.Task</code> that periodically removes expired entries. If the task is left dangling, the event loop may complain about pending tasks on shutdown.</li> <li><code>shutdown_cache()</code> cancels the internal task and waits for it to finish, ensuring a clean exit.</li> </ul>"},{"location":"03_use_cases/11_cache_lifecycle/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Register the cache before calling <code>initialize_cache()</code>; otherwise the registry won\u2019t know which backend to start.</li> <li>For Redis backends, the shutdown step also closes the underlying <code>redis.asyncio.Redis</code> client connection.</li> <li>You can also hook the shutdown into a FastAPI <code>@app.on_event(\"shutdown\")</code> handler if you prefer not to use the lifespan context manager.</li> </ul>"},{"location":"03_use_cases/12_transaction_helper/","title":"Use Case 12: Using <code>TransactionManager.execute_transaction</code>","text":"<p>Scenario: You need to perform several writes across different collections atomically (e.g., creating an <code>Order</code> and updating the <code>Inventory</code> for each ordered item). The low\u2011level <code>start_session</code> context manager works, but the high\u2011level <code>execute_transaction</code> helper makes the code cleaner and returns all operation results in a list.</p>"},{"location":"03_use_cases/12_transaction_helper/#whats-new","title":"\ud83d\udce6 What\u2019s New?","text":"Component Description <code>TransactionManager.execute_transaction</code> Accepts a list of async callables (each receiving a <code>session</code>) and executes them inside a single MongoDB transaction. Returns a list of each callable\u2019s result. Automatic rollback If any operation raises an exception, the transaction is aborted and the exception propagates."},{"location":"03_use_cases/12_transaction_helper/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from mongo_ops import MongoConnectionManager, ModelRegistry\nfrom mongo_ops.transactions import TransactionManager\nfrom mongo_ops.repository import BaseRepository\nfrom mongo_ops.models import BaseDocument\nfrom bson import ObjectId\n\n# ----------------------------------------------------------------------\n# 1\ufe0f\u20e3 Define simple models\n# ----------------------------------------------------------------------\nclass Order(BaseDocument):\n user_id: str\n items: list[dict] # [{\"product_id\": ObjectId, \"qty\": int}]\n total: float = 0.0\n\nclass Inventory(BaseDocument):\n product_id: ObjectId\n quantity: int = 0\n\n# ----------------------------------------------------------------------\n# 2\ufe0f\u20e3 Repositories\n# ----------------------------------------------------------------------\nclass OrderRepo(BaseRepository[Order]):\n def __init__(self):\n super().__init__(\"orders\", Order)\n\nclass InventoryRepo(BaseRepository[Inventory]):\n def __init__(self):\n super().__init__(\"inventory\", Inventory)\n\norder_repo = OrderRepo()\ninv_repo = InventoryRepo()\n\n# ----------------------------------------------------------------------\n# 3\ufe0f\u20e3 Transaction helper\n# ----------------------------------------------------------------------\nasync def create_order_with_inventory(order: Order):\n async def insert_order(session):\n # Insert the order document\n doc = order.model_dump(exclude={\"id\"}, exclude_none=True)\n doc[\"created_at\"] = doc[\"updated_at\"] = __import__(\"datetime\").datetime.utcnow()\n result = await order_repo.collection.insert_one(doc, session=session)\n return await order_repo.collection.find_one({\"_id\": result.inserted_id}, session=session)\n\n async def update_inventory(session):\n # Decrease quantity for each ordered item\n for item in order.items:\n await inv_repo.collection.update_one(\n {\"product_id\": ObjectId(item[\"product_id\"])},\n {\"$inc\": {\"quantity\": -item[\"qty\"]}},\n session=session,\n )\n return \"inventory-updated\"\n\n # Execute both operations atomically\n results = await TransactionManager.execute_transaction([\n lambda s: insert_order(s),\n lambda s: update_inventory(s),\n ])\n # `results[0]` is the created order document, `results[1]` is the marker string\n return results[0]\n\n# ----------------------------------------------------------------------\n# 4\ufe0f\u20e3 FastAPI lifespan \u2013 initialise DB and run the example endpoint\n# ----------------------------------------------------------------------\nfrom fastapi import FastAPI, HTTPException\nfrom contextlib import asynccontextmanager\n\napp = FastAPI()\n\n@asynccontextmanager\nasync def lifespan(app):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\",\n db_name=\"shop\",\n ):\n await ModelRegistry.initialize_all()\n yield\n\napp.lifespan = lifespan\n\n@app.post(\"/orders/\", response_model=Order)\nasync def create_order(order: Order):\n created = await create_order_with_inventory(order)\n if created is None:\n raise HTTPException(status_code=400, detail=\"Transaction failed\")\n return created\n</code></pre>"},{"location":"03_use_cases/12_transaction_helper/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Return values: Each callable can return whatever you need (inserted document, stats, etc.). The helper collects them in order.</li> <li>Error handling: Raising any exception inside a callable will abort the transaction automatically.</li> <li>Read\u2011only ops: You can also run queries inside the transaction \u2013 just pass <code>session=session</code> to the <code>find_one</code>/<code>find</code> calls.</li> <li>Testing: In unit tests you can mock <code>TransactionManager.start_session</code> to ensure the helpers are called.</li> </ul>"},{"location":"03_use_cases/13_index_creation/","title":"Use Case 13: Declaring Indexes (Single\u2011field, Composite, Unique)","text":"<p>Scenario: You want to ensure your collection has the right indexes for fast queries and data integrity. This doc shows how to register models with different index configurations using <code>ModelRegistry.register</code> and have them created automatically during startup.</p>"},{"location":"03_use_cases/13_index_creation/#whats-new","title":"\ud83d\udce6 What\u2019s New?","text":"Component Description <code>ModelRegistry.register</code> (indexes argument) Accepts a list of index specifications. Each spec can be a tuple <code>(field, direction)</code> or a full dict with <code>keys</code> and optional options (e.g., <code>unique</code>). <code>ModelRegistry.initialize_all</code> Scans all registered collections and creates the defined indexes on application start\u2011up."},{"location":"03_use_cases/13_index_creation/#example","title":"\ud83d\ude80 Example","text":"Python<pre><code>from mongo_ops import ModelRegistry, BaseDocument\nfrom pymongo import ASCENDING, DESCENDING\n\n# ----------------------------------------------------------------------\n# 1\ufe0f\u20e3 Simple single\u2011field index (e.g., email lookup)\n# ----------------------------------------------------------------------\nclass User(BaseDocument):\n username: str\n email: str\n\nModelRegistry.register(\n collection_name=\"users\",\n model=User,\n indexes=[(\"email\", ASCENDING)], # creates an index on \"email\"\n)\n\n# ----------------------------------------------------------------------\n# 2\ufe0f\u20e3 Composite index (e.g., queries filtering by user and creation date)\n# ----------------------------------------------------------------------\nclass BlogPost(BaseDocument):\n author_id: str\n created_at: str\n title: str\n\nModelRegistry.register(\n collection_name=\"posts\",\n model=BlogPost,\n indexes=[[(\"author_id\", ASCENDING), (\"created_at\", DESCENDING)]], # compound index\n)\n\n# ----------------------------------------------------------------------\n# 3\ufe0f\u20e3 Unique index (e.g., enforce unique usernames)\n# ----------------------------------------------------------------------\nModelRegistry.register(\n collection_name=\"users\",\n model=User,\n indexes=[\n {\n \"keys\": [(\"username\", ASCENDING)],\n \"options\": {\"unique\": True, \"name\": \"uq_username\"},\n }\n ],\n)\n\n# ----------------------------------------------------------------------\n# 4\ufe0f\u20e3 Initialise all indexes during FastAPI startup\n# ----------------------------------------------------------------------\nfrom mongo_ops import MongoConnectionManager\nfrom contextlib import asynccontextmanager\n\n@asynccontextmanager\nasync def lifespan(app):\n async with MongoConnectionManager.lifespan(\n uri=\"mongodb://localhost:27017\",\n db_name=\"mydb\",\n ):\n # This will create every index declared above (if it does not already exist)\n await ModelRegistry.initialize_all()\n yield\n</code></pre>"},{"location":"03_use_cases/13_index_creation/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Idempotence: <code>initialize_all</code> uses <code>create_index</code>, which is safe to run multiple times \u2013 MongoDB will skip creation if the index already exists.</li> <li>Index options: You can also set <code>expireAfterSeconds</code> for TTL indexes, <code>sparse</code> for sparse indexes, etc., by adding them to the <code>options</code> dict.</li> <li>Verification: After the app starts, you can verify indexes with the Mongo shell: JavaScript<pre><code>db.<collection>.getIndexes()\n</code></pre></li> </ul>"},{"location":"03_use_cases/14_testing_guide/","title":"Use Case 14: Testing Guide \u2013 Mocking Motor & Population Engine","text":"<p>Scenario: You want to write unit tests for repositories that use async Motor collections and the <code>PopulationEngine</code> without connecting to a real MongoDB instance.</p>"},{"location":"03_use_cases/14_testing_guide/#whats-new","title":"\ud83d\udce6 What\u2019s New?","text":"Component Description <code>AsyncMock</code> for collections Allows you to stub <code>find_one</code>, <code>insert_one</code>, etc., and control returned data. Patching <code>MongoConnectionManager.get_database</code> Redirects repository initialization to a mock collection. Mocking <code>PopulationEngine</code> Replace the real engine with a lightweight stub that returns pre\u2011crafted objects. Example test files <code>tests/test_populating_repository.py</code> and <code>tests/test_transactions.py</code> are used as reference implementations."},{"location":"03_use_cases/14_testing_guide/#example-test-boilerplate","title":"\ud83d\ude80 Example Test Boilerplate","text":"Python<pre><code>import pytest\nfrom unittest.mock import AsyncMock, patch\nfrom bson import ObjectId\n\nfrom mongo_ops.models import BaseDocument\nfrom mongo_ops.populate import PopulateRule, PopulationEngine\nfrom mongo_ops.repository import PopulatingRepository\n\n# ----------------------------------------------------------------------\n# 1\ufe0f\u20e3 Define simple models (same as in the library tests)\n# ----------------------------------------------------------------------\nclass Profile(BaseDocument):\n avatar_url: str = \"\"\n\nclass User(BaseDocument):\n name: str = \"\"\n profile_id: ObjectId | None = None\n profile: Profile | None = None\n\n# ----------------------------------------------------------------------\n# 2\ufe0f\u20e3 Fixture \u2013 mock Motor collection\n# ----------------------------------------------------------------------\n@pytest.fixture\ndef mock_collection():\n return AsyncMock()\n\n# ----------------------------------------------------------------------\n# 3\ufe0f\u20e3 Fixture \u2013 mock PopulationEngine (optional)\n# ----------------------------------------------------------------------\n@pytest.fixture\ndef engine():\n # Provide a real engine but replace the repo for \"profiles\" with a mock\n profile_repo = AsyncMock()\n return PopulationEngine({\"profiles\": profile_repo})\n\n# ----------------------------------------------------------------------\n# 4\ufe0f\u20e3 Fixture \u2013 repository under test\n# ----------------------------------------------------------------------\n@pytest.fixture\ndef repo(mock_collection, engine):\n with patch(\"mongo_ops.repository.MongoConnectionManager.get_database\") as mock_db:\n mock_db.return_value.__getitem__.return_value = mock_collection\n repo = PopulatingRepository(\n \"users\",\n User,\n population_engine=engine,\n populate_rules=[\n PopulateRule(\n field_name=\"profile\",\n collection_name=\"profiles\",\n ref_field=\"profile_id\",\n )\n ],\n )\n repo.collection = mock_collection\n return repo\n\n# ----------------------------------------------------------------------\n# 5\ufe0f\u20e3 Test \u2013 `get_by_id` populates the related document\n# ----------------------------------------------------------------------\n@pytest.mark.asyncio\nasync def test_get_by_id_populates(repo, mock_collection, engine):\n # Arrange \u2013 user document returned from the DB\n uid = ObjectId()\n pid = ObjectId()\n mock_collection.find_one.return_value = {\n \"_id\": uid,\n \"name\": \"Alice\",\n \"profile_id\": pid,\n \"created_at\": \"2024-01-01T00:00:00\",\n \"updated_at\": \"2024-01-01T00:00:00\",\n }\n # Mock the profile repo to return a concrete Profile instance\n engine._repos[\"profiles\"].get_by_id.return_value = Profile(id=pid, avatar_url=\"pic.png\")\n\n # Act\n result = await repo.get_by_id(uid)\n\n # Assert\n assert result is not None\n assert result.name == \"Alice\"\n assert result.profile is not None\n assert result.profile.avatar_url == \"pic.png\"\n\n# ----------------------------------------------------------------------\n# 6\ufe0f\u20e3 Test \u2013 transaction helper uses the real `TransactionManager`\n# ----------------------------------------------------------------------\nfrom mongo_ops.transactions import TransactionManager\n\n@pytest.mark.asyncio\nasync def test_execute_transaction(monkeypatch):\n # Fake a Motor client session that records calls\n async def fake_op(session):\n return \"ok\"\n # Execute with the helper \u2013 it should return a list with the result\n results = await TransactionManager.execute_transaction([fake_op])\n assert results == [\"ok\"]\n</code></pre>"},{"location":"03_use_cases/14_testing_guide/#tips","title":"\ud83d\udca1 Tips","text":"<ul> <li>Never hit the network: All DB calls are mocked, so tests run instantly.</li> <li>Reuse fixtures: Keep the <code>mock_collection</code> and <code>engine</code> fixtures in a <code>conftest.py</code> file for other repo tests.</li> <li>Coverage: The same pattern works for <code>CachedBaseRepository</code> \u2013 just mock <code>cache_backend</code> methods (<code>get</code>, <code>set</code>, <code>delete</code>).</li> <li>AsyncTestCase: If you prefer <code>unittest</code> style, use <code>IsolatedAsyncioTestCase</code> from Python 3.8+.</li> </ul>"}]} |