Skip to content

Use Case 15: Inside PopulatingRepository — the Object ⇄ ObjectId Lifecycle

Scenario: You want to see what the repository actually does, in which order, before you trust it with your data — how a Profile becomes an ObjectId for storage and comes back as a Profile on read, and where every piece is wired.

📦 Two Representations, One Field

A populate-ruled field is a shape shifter — the same name holds different things depending on where you look:

Place Value held in the field
In the MongoDB doc ObjectId (or list[ObjectId])
In the cache/JSON hex string (see use case 16)
In the app model the referenced model (or None)

PopulatingRepository.create / update depopulate (model → ObjectId) before writing; get_by_id / get_many populate (ObjectId → model) after reading:

Text Only
App model                         MongoDB document                     App model
User(profile=Profile)  ──depopulate──▶  { profile: <ObjectId> }  ──populate──▶  User(profile=Profile)
                        (write path)      (storage shape)          (read path)

🚀 Write Path — create() Step by Step

Given user = User(username="alice", profile=saved_profile) where saved_profile.id exists:

Python
class UserRepository(PopulatingRepository[User]):
    def __init__(self):
        super().__init__(
            collection_name="users",
            model=User,
            population_engine=engine,
            populate_rules=[profile_rule],
        )
  1. Guard check (_depopulate, repository.py): for each rule field, if the value is already an ObjectId (or list[ObjectId]) an error is raised — see _depopulate guards below. Saved models pass.
  2. engine.depopulate(user, rules) collapses the graph in place:
  3. saved_profile (a BaseDocument with .id) → becomes saved_profile.id → ObjectId
  4. a Profile without .id (unsaved) → is left as a model object, which model_dump then embeds as a dict — see intricacy #3
  5. when nested_rules are present, the nested model is depopulated recursively first, then collapsed
  6. model_dump(exclude={"id"}, exclude_none=True) produces the raw insert dict; timestamps are added.
  7. insert_one(doc) writes {..., "profile": ObjectId("..."), ...} to MongoDB.
  8. On the way out, data_to_model runs _populate (the read path below) so create returns a fully populated model, not the raw one.

The _depopulate guards (loud failures > silent corruption)

Stored/held value under a rule field Behaviour
ObjectId / list[ObjectId] ValueError("...contains ObjectId — was populate skipped?") — the doc was loaded raw (e.g. via BaseRepository or an unpopulated read) and handed back to create/update
a saved BaseDocument (has .id) collapse to .id (ObjectId)
an unsaved BaseDocument (no .id) kept as a model → embedded dict below
a dict not BaseDocument → left as-is → embedded
non-BaseDocument, non-dict value AttributeError raised by engine.depopulate
no engine / no rules passthrough — _depopulate just does model_dump, whatever shape you gave is stored

🚀 Read Path — get_by_id() Step by Step

get_by_id inherits CRUD and only changes data_to_model (repository.py:247):

Python
doc = await self.collection.find_one({"_id": id})   # {"profile": ObjectId, ...}
return await self.data_to_model(doc)                 # data_to_model → self._populate(doc)

_populate(data) walks each rule field in the raw dict:

  1. ref = data.get("profile") → ObjectId. If ref is None → field untouched (stays absent).
  2. Look up the repo: repo = engine._repos.get("profiles"). If it's not registered, doc = None.
  3. doc = await repo.get_by_id(ref) — a single read on the referenced collection.
  4. If doc is None → data["profile"] = None (missing refs resolve to None, never raise).
  5. If the rule has nested_rules → engine.populate(doc, nested_rules, depth=1) deepens the result.
  6. data["profile"] = doc → self.model(**data) builds the User with a real Profile.

Lists behave the same per item; a ref that is a dict (an embedded document) raises ValueError("...contains embedded dict(s) — run repair script").

Engine is optional. Set population_engine=None and populate_rules=[], and PopulatingRepository is just a BaseRepository — FK fields come back as raw ObjectId, and your model field type must agree (that is the whole point of the guard in the WRITE path: a Profile | None-typed field populated with a raw ObjectId is a broken round-trip waiting to happen).

🔌 How It's Wired

Python
from mongo_ops import BaseDocument, PopulatingRepository
from mongo_ops.populate import PopulateRule, PopulationEngine

class Profile(BaseDocument):
    avatar_url: str = ""

class User(BaseDocument):
    username: str = ""
    profile: Profile | None = None   # ObjectId in DB, Profile in memory

engine = PopulationEngine({})                      # repositories live here, keyed by collection name

profile_rule = PopulateRule(field_name="profile", collection_name="profiles")

class UserRepository(PopulatingRepository[User]):
    def __init__(self):
        super().__init__("users", User, population_engine=engine, populate_rules=[profile_rule])

def wire() -> None:
    """Call AFTER connect() — repositories need a live database."""
    engine.register_repo("profiles", PopulatingRepository[Profile]("profiles", Profile))
    engine.register_repo("users", UserRepository())

Wiring rules:

  • engine._repos is keyed by collection_name as written in the rule — typo → silent None refs.
  • register_repo needs an already-constructed repository → call it inside the lifespan (after connect() / MongoConnectionManager.lifespan), not at module import.
  • The referenced repository only needs a get_by_id that returns a BaseDocument — it can be a plain BaseRepository, another PopulatingRepository, or even a cached repo (see use case 10). Population requires no extra DB index on the referenced collection's _id.
  • Swap at runtime: repo.set_population_engine(new_engine) and repo.set_populate_rules(new_rules) — the tests exercise both.

⚠️ Intricacy — nested_rules Do NOT Round-Trip Through depopulate

nested_rules are designed for read-side deep population. If the same rules run through the write path (create / update), engine.depopulate behaves differently from plain refs — verified against the engine:

Rule shape What depopulate does to it Effect on the stored doc
scalar FK, no nested_rules collapse to .id stored as ObjectId ✓
list[ObjectId] FK, no nested_rules collapse each item to .id stored as list[ObjectId] ✓
list FK with nested_rules items are kept as models stored as embedded dicts ✗ — reading back raises "contains embedded dict(s) — run repair script"
scalar FK with nested_rules the recursion runs, then the field is assigned None reference lost ✗ — written as absent/null

In other words: a document carrying deep nested_rules (like UC 09's Author → books → publisher) cannot be created / updated as-is. If the graph must be written back, materialize references separately (save each Book to its collection first, then store list[ObjectId] without nested_rules), and keep nested_rules only on read rules you never hand back to depopulate.

💡 Tips

  • Never pass a raw ObjectId-holding model to create/update: the "was populate skipped?" ValueError is the guard. Read through the repo so the read path can populate first.
  • Unsaved references are a write-once trap: they embed as dicts, and re-reading raises the "run repair script" ValueError. Save referenced docs to their collection first, then reference their id.
  • filter / projection on PopulateRule are declared but not applied by the engine — don't rely on them.
  • Reads cost 1 query per reference (no $lookup yet); batch-heavy endpoints should add caching (next use case).