17. Populate & Depopulate — Worked Examples

Scenario: You want to see what populate/depopulate actually do to your documents — field by field, at every layer (Python model → depopulated → stored in Mongo → raw read back → populated).

These examples all ran against the real mongo_ops engine (mongo_ops/populate/engine.py). Timestamps and ObjectIds are abbreviated with … for readability.

---

Models & rules used below

Python

from bson import ObjectId
from mongo_ops import BaseDocument, PopulatingRepository
from mongo_ops.populate import PopulateRule, PopulationEngine

class Profile(BaseDocument):
    avatar_url: str = ""
    bio: str = ""

class Publisher(BaseDocument):
    name: str = ""

class Book(BaseDocument):
    title: str = ""
    publisher: Publisher | None = None    # nested ref, reached via nested_rules

class User(BaseDocument):
    name: str = ""
    profile: Profile | None = None        # scalar FK — DB stores an ObjectId
    books: list[Book] | None = None       # list FK    — DB stores list[ObjectId]

engine = PopulationEngine({})             # repos keyed by collection name, see UC 15

# A rule WITHOUT nested_rules — the write path is safe.
profile_rule = PopulateRule(field_name="profile", collection_name="profiles")

# A rule WITH nested_rules — designed for READ-side deep population.
books_rule = PopulateRule(
    field_name="books",
    collection_name="books",
    nested_rules=[PopulateRule(field_name="publisher", collection_name="publishers")],
)

---

1. Write path — depopulate() turns models into references

engine.depopulate(document, rules) walks each rule field and mutates the model in place before you insert_one / find_one_and_update it. What you get on the other side:

1a. Scalar FK, no nested_rules → stored as ObjectId

Python

async def demo():
    profile = Profile(id=ObjectId("656…04"), avatar_url="a.png", bio="hi")
    user    = User(id=ObjectId("656…05"), name="Ada", profile=profile)

    await engine.depopulate(user, [profile_rule])

    return user.profile          # ObjectId('656…04')   — was a Profile model

What Mongo receives (via model_dump, minus _id handling):

JSON

{ "name": "Ada", "profile": { "$oid": "656…04" } }

1b. Scalar FK with nested_rules → reference lost (do not do this)

Python

profile_rule_deep = PopulateRule(                 # same field, but with nested_rules
    field_name="profile",
    collection_name="profiles",
    nested_rules=[PopulateRule(field_name="bio", collection_name="profiles")],  # any child field
)

async def demo():
    user = User(id=ObjectId("656…05"), name="Ada",
                profile=Profile(id=ObjectId("656…04"), avatar_url="a.png"))
    await engine.depopulate(user, [profile_rule_deep])
    return user.profile          # None           — reference silently dropped!

Because depopulate recurses into the child but then assigns the parent field None (see engine.py — depopulated_value is never set on that branch). On save you lose the reference entirely. There is no warning — the field is just gone.

1c. List FK, no nested_rules → stored as list[ObjectId] (the safe pattern)

Python

async def demo():
    b1 = Book(id=ObjectId("656…06"), title="MongoDB in Action")
    b2 = Book(id=ObjectId("656…07"), title="MongoDB: The Definitive Guide")
    user = User(id=ObjectId("656…08"), name="Ada", books=[b1, b2])

    await engine.depopulate(user, [PopulateRule(field_name="books", collection_name="books")])

    return user.books          # [ObjectId('656…06'), ObjectId('656…07')]   — clean refs

1d. List FK with nested_rules → stored as embedded documents (do not do this)

Python

async def demo():
    publisher = Publisher(id=ObjectId("656…09"), name="O'Reilly")
    b1p = Book(id=ObjectId("656…06"), title="MongoDB in Action", publisher=publisher)
    user = User(id=ObjectId("656…08"), name="Ada", books=[b1p])

    await engine.depopulate(user, [books_rule])     # books_rule carries nested_rules
    return user.books[0]
     # <Book id=ObjectId('656…06') publisher=ObjectId('656…09')>

books stays a list of Book models — the nested publisher collapsed to ObjectId, but the Book itself was left embedded. Serialized to Mongo you get a nested sub-document:

JSON

{ "name": "Ada", "books": [ { "title": "MongoDB in Action", "publisher": { "$oid": "656…09" } } ] }

…and reading that back through population raises ValueError(…contains embedded dict(s) — run repair script…), because the engine expects books to hold references, not embedded docs.

Why UC 09's Author → books → publisher graph can't be written as-is: a rule with nested_rules breaks the write path (1b logs silently, 1d stores embedded). The safe round-trip: save referenced documents first, store list[ObjectId]/ObjectId with a non-nested rule for reads, and keep nested_rules only on rules you never hand back to depopulate.

---

2. Read path — populate() turns references back into models

PopulatingRepository.data_to_model(raw_dict) resolves references before building the model. It first runs _populate(dict) — replacing the raw ObjectId references with full models via the registered repos — and only then constructs self.model(**data), so every FK field is a real model by construction. The two data_to_model examples below (2a/2b) show the underlying resolution _populate performs per rule. (repo below is a PopulatingRepository wired per UC 15 — books and publishers repos registered in its PopulationEngine.)

2a. Scalar FK — ObjectId → Profile model

Python

# what came back from Mongo (raw dict, profile is an ObjectId reference):
raw = {"_id": ObjectId("656…05"), "name": "Ada", "profile": ObjectId("656…04")}

async def demo():
    user = await repo.data_to_model(raw)           # _populate() → model(**data)
    return user.profile          # Profile(id=ObjectId('656…04'), avatar_url='a.png')

2b. List FK + nested_rules — two levels deep

Python

raw = {"_id": ObjectId("656…08"), "name": "Ada",
       "books": [ObjectId("656…06"), ObjectId("656…07")]}

async def demo():
    user = await repo.data_to_model(raw)           # _populate() resolves both levels
    return (type(user.books[0]).__name__,                  # 'Book'
            type(user.books[0].publisher).__name__,        # 'Publisher' — nested_rules reached it
            user.books[0].publisher.name)                  # "O'Reilly"

Walking through one book's resolution:

Text Only

books_rule                → field "books",            collection "books"
  nested_rules[0].publisher rule → field "publisher", collection "publishers"
1. for ObjectId('656…06')   → repo("books").get_by_id(...)         → Book("MongoDB in Action")
2. book has nested rule     → repo("publishers").get_by_id(...)    → Publisher("O'Reilly")
3. author.books[0]          → Book(publisher=Publisher("O'Reilly"))

Per-level cost: 1 query per reference (no $lookup). A hub with 20 hooks and a publisher nested rule is 40 queries — that is why the next section exists.

---

3. Cache round trip — what actually goes into the cache

CachedBaseRepository stores model_dump(by_alias=True) JSON-encoded with default=str (the encode_value helper in mongo_ops/cache/in_memory.py). Crucially it snapshots the raw, just-out-of-Mongo shape — FK fields still hold ObjectIds — so the cache holds references, exactly like the DB row:

Layer	profile field looks like
Mongo doc (raw)	“profile”: ObjectId("656…04")
model_dump(by_alias=True)	"profile": ObjectId("656…04")
encode_value(...)	"...\"profile\": \"656…04\"..." — hex string
decode_value(...)	"profile": "656…04" — still a hex string

Verbatim (timestamps abbreviated):

Python

from mongo_ops.cache.in_memory import decode_value, encode_value

snapshot = {"id": "656…05", "profile": ObjectId("656…04")}   # what a cached raw doc looks like

cached   = encode_value(snapshot)     # json.dumps(snapshot, default=str).encode()
decoded  = decode_value(cached)

decoded["profile"]                    # '656…04'  — hex STRING, not ObjectId, not Profile

Typed-FK caveat: the round trip above only works when the FK field's model type accepts the raw value (str/ObjectId typed). A field typed Profile | None cannot even be materialized from the raw Mongo doc — both hit and miss raise ValidationError (UC 16's intricacy). That is exactly why UC 10's composed repo overrides get_by_id/create/update to carry raw data through the cache and resolve refs with _populate.

Two consequences (the "intricacies", detailed in UC 16):

1. On a cache hit, decode_value gives you hex-string references. A field typed Profile | None receiving a hex string raises Pydantic ValidationError — so a plain CachedBaseRepository cannot serve model-typed FK fields, hit or miss. The composed pattern in UC 10 re-resolves them in its get_by_id: data = decode_value(await cache.get(key)), then populated = await self._populate(data), then self.model(**populated).
2. On a miss, BaseRepository.get_by_id reads the raw Mongo doc whose profile is an ObjectId and immediately raises the same ValidationError — which is why UC 10 overrides get_by_id to fetch raw, cache raw, then populate. (Real ObjectIds are covered by UC 16's table.)

---

4. When to use which

You need…	Use
Scalar ObjectId store, model-struct reads	PopulatingRepository (UC 08)
Deep nested reads (2+ levels)	PopulatingRepository + nested_rules (UC 09) — read side only
Same shape, but cache-first reads	UC 10 composed repo (engine + cache + _populate)
Scalar docs, no refs, hot get_by_id	CachedBaseRepository (UC 07, UC 16)
Writing a reference graph back	depopulate without nested_rules — save children first, then point at their ids

---

Related

• 08 – Population · 09 – Advanced population · 10 – Cache + Population · 15 – PopulatingRepository wiring · 16 – CachedRepository intricacies