mongodb

# MongoDB — modeling, indexing, aggregation, transactions, ops Engine-level MongoDB 8.2 guidance: model documents for the queries you actually run, pick the index the planner will use, write aggregation pipelines that stay index-eligible, run multi-document transactions with correct retry, and operate/secure a deployment. Driver-agnostic — every example is `mongosh` shell syntax that maps 1:1 to the official drivers (Node, Python, Go, Java, Rust). This skill owns the **server's query and the index it picks**, not any ODM's API. ## When to use / When NOT to use **When to use:** - Document modeling: embed vs reference, the 16 MB cap, one-to-many/many-to-many, the subset/extended-reference/bucket/computed/outlier patterns, taming unbounded array growth. - Index decisions: single-field, compound (the ESR ordering rule), multikey, partial, TTL, text, wildcard, `2dsphere`; and when an index is NOT worth it. - Any query that is slow or scans too much; reading `explain("executionStats")`. - Aggregation pipelines: stage order so `$match`/`$sort` hit an index, `$lookup` cost, `$unwind` explosion, `$group`/`$sort` memory limits and `allowDiskUse`, `$merge`/`$out`, faceting. - Multi-document transactions: sessions, `withTransaction` retry semantics, read/write concern. - Operating/securing: replica set, read preference, write concern, Atlas tier choice, Atlas Search & Vector Search, Queryable Encryption, role-based access, connection-pool knobs. **When NOT to use:** - Relational schema / SQL / `EXPLAIN ANALYZE` → [`postgresdb`](../postgresdb/SKILL.md). Different engine, planner, and concurrency model. - ODM/driver API ergonomics (Mongoose pre-save hooks, the Node driver's `bulkWrite` return shape, `updateMany`'s result object) → that tool's own docs. This skill owns the server query and the index the server picks, not the JS object the driver hands back. - App-layer caching as a product (Redis-in-front-of-reads). - Cloud-console click-paths — we give the shell command / connection string, not the Atlas UI tour. - Picking a vector store *across engines* (Pinecone vs Weaviate). Atlas Vector Search *inside* Mongo is in scope; cross-engine selection is not. Deep dives: [data-modeling](references/data-modeling.md) (embed/reference tree, all six patterns, 16 MB math, polymorphic & schema versioning) · [aggregation](references/aggregation.md) (per-stage index eligibility, `$lookup` variants, `$facet`, window fns, `$merge`/`$out`, reading pipeline `explain`) · [transactions-and-ops](references/transactions-and-ops.md) (retry wrappers, concern semantics, Atlas tiers, Vector Search, Queryable Encryption, RBAC, pooling, change streams). ## Non-negotiables 1. **Design for the queries you run, not the shape of your data.** The schema is the set of documents that make your common reads single-document and index-eligible. 2. **Never let an array grow unbounded inside a document.** It walks toward the 16 MB cap, bloats every read of the parent, and kills update performance — reference or bucket it. 3. **The hard ceiling is 16 MB per document.** If a one-to-many can exceed it, you reference; there is no `TOAST`-style overflow here. 4. **A single-document write is already atomic.** Reach for a multi-document transaction *only* when two or more documents must change together — otherwise you are paying for nothing. 5. **Every transaction retries on the `TransientTransactionError` label** (and commit retries on `UnknownTransactionCommitResult`). `withTransaction` does both for you; a hand-rolled loop must. 6. **Index by ESR: Equality fields, then the Sort field, then Range fields.** This order lets one compound index serve the filter, the sort, and the range without an in-memory sort. 7. **Read `explain("executionStats")` before and after adding an index** — confirm `IXSCAN`, not `COLLSCAN`, and `totalKeysExamined ≈ nReturned`. Or it didn't happen. 8. **`w:"majority"` for money and state transitions**, read concern `"majority"`/`"snapshot"` when a read must reflect a durable write. `w:1` can be rolled back on a primary failover. 9. **Money is `Decimal128` (`NumberDecimal("...")`), never a JS `double`.** Binary floats drift; `0.1 + 0.2 !== 0.3` in your ledger. 10. **Never store a secret in plaintext.** Use Queryable Encryption / client-side field-level encryption; never commit a `mongodb://user:pass@` literal. ## Decision rules ### Embed or reference | Relationship | Choose | Why | | --- | --- | --- | | Read together, small, bounded (address on a user) | embed | one read, no `$lookup`, atomic update | | One-to-few, bounded (≤ a few dozen, won't grow) | embed | stays well under 16 MB | | One-to-many, growth not bounded (comments on a post) | reference | array would chase the 16 MB cap | | Many-to-many (students↔courses) | reference (array of ids on the lighter side) | shared, independently mutated | | Child shared across parents | reference | one source of truth, no duplication drift | | Child independently and frequently mutated | reference | avoid rewriting a big parent per child edit | | High-cardinality / huge child set | reference (+ optional subset embed) | keep the hot read small | ### Which schema pattern | Symptom | Pattern | What it does | | --- | --- | --- | | List view reads 3 fields of a heavy doc | **subset** | embed only the hot fields, reference the rest | | `$lookup` on every read just to show a name/price | **extended reference** | copy the few joined fields you display | | Unbounded time-ordered events (readings, logs) | **bucket** | group N events per doc by time window | | Same `count`/`sum` recomputed on every read | **computed** | store the rollup, update it on write | | 1% of docs break the shape (a few mega-children) | **outlier** | flag them, overflow into linked docs | | One collection holds several entity shapes | **polymorphic** | a `type` discriminator + shared `_id` space | Full Bad→Good documents for each in [data-modeling](references/data-modeling.md). ### Which index type | Access pattern | Index | Note | | --- | --- | --- | | `=` on one field | single-field | also covers the field's sort | | filter + sort + range together | compound, ordered **ESR** | one index serves all three | | query into an array field | multikey (automatic on an array key) | one multikey field per compound index | | query only a subset of docs (`status:"active"`) | partial (`partialFilterExpression`) | smaller, cheaper to maintain | | auto-expire docs after a time | TTL (`expireAfterSeconds` on a Date) | single-field only; deletes in background | | language-aware text search | text **or** Atlas Search | Atlas Search is far richer; text is legacy | | unpredictable / many query shapes on subdocs | wildcard (`"$**"`) | last resort; never beats a targeted index | | geospatial proximity / within | `2dsphere` | GeoJSON `Point`/`Polygon` | | vector similarity (8.2, Community+) | Atlas/Vector Search index | see transactions-and-ops ref | ### When NOT to add an index - Low-cardinality field (a boolean, a 3-value status) — the planner skips it; `COLLSCAN` wins. - Tiny collection — a collection scan reads one or two pages; the index is pure write tax. - A field already the **left prefix** of an existing compound index — redundant. - Write-heavy field rarely filtered — every index is paid on every insert/update. - "Just in case" indexes — an unused index costs writes and RAM, returns nothing. ## Copy-paste patterns Every fence is `mongosh` syntax. ### Model the document for the read (Bad → Good) ```javascript // BAD: comments embedded in the post — array grows without bound toward 16 MB, // every post read drags the entire comment history, money is a float. db.posts.insertOne({ _id: ObjectId(), title: "Indexing 101", authorId: ObjectId(), price: 9.99, // double — drifts in arithmetic comments: [ /* ...unbounded... */ ] // chases the 16 MB cap }) // GOOD: post stays small; comments referenced; money is Decimal128; // the few fields the feed needs are duplicated (extended reference). db.posts.insertOne({ _id: ObjectId(), title: "Indexing 101", author: { _id: ObjectId(), name: "Ada" }, // extended ref: name shown without a $lookup price: NumberDecimal("9.99"), commentCount: 0, // computed rollup, bumped on write createdAt: new Date() }) db.comments.insertOne({ _id: ObjectId(), postId: ObjectId(), body: "…", createdAt: new Date() }) ``` ### Compound index in ESR order + the query that uses it ```javascript // Feed query: filter by author (equality), sort by date (sort), bound by a date (range). // ESR => author first, then the sort/range key. db.posts.createIndex({ "author._id": 1, createdAt: -1 }) db.posts.find({ "author._id": authorId, createdAt: { $gte: since } }) .sort({ createdAt: -1 }) .limit(20) // Confirm the plan: IXSCAN on the index above, no in-memory SORT stage. ``` ### Partial + TTL indexes ```javascript // Partial: index only the rows you actually query (active orders), not the archive. db.orders.createIndex( { customerId: 1, createdAt: -1 }, { partialFilterExpression: { status: "active" } } ) // TTL: expire sessions 30 minutes after lastSeen. Field MUST be a Date. db.sessions.createIndex({ lastSeen: 1 }, { expireAfterSeconds: 1800 }) ``` ### Aggregation: `$match` first, `$lookup`, `$group` with `allowDiskUse` ```javascript db.orders.aggregate([ // $match FIRST so it uses the compound index and shrinks the working set early. { $match: { status: "paid", createdAt: { $gte: since } } }, { $sort: { createdAt: -1 } }, // index-eligible here, before any $group/$project { $lookup: { from: "customers", localField: "customerId", foreignField: "_id", as: "customer", pipeline: [ { $project: { name: 1 } } ] // project inside $lookup: pull only what you need }}, { $group: { _id: "$customerId", total: { $sum: "$amount" } } } ], { allowDiskUse: true }) // $group/$sort spill past 100 MB/stage; this lets large groups complete, // it is NOT a substitute for a missing $match index — see anti-patterns. ``` ### Read `explain("executionStats")` — the four numbers ```javascript db.posts.find({ "author._id": authorId }).sort({ createdAt: -1 }) .explain("executionStats") ``` Read these before declaring a fix: 1. **`winningPlan.stage`** — must be `IXSCAN` (or `FETCH`→`IXSCAN`), not `COLLSCAN`. 2. **`totalKeysExamined` vs `nReturned`** — close means the index is selective; a huge ratio means the index scans far more than it returns (wrong key order, low selectivity). 3. **A `SORT` stage** — an in-memory sort the index should have satisfied; reorder by ESR to remove it. 4. **`rejectedPlans`** — what the planner considered and dropped; a near-miss hints at a better index. ### Multi-document transaction with full retry ```javascript // Use withTransaction — it retries the body on TransientTransactionError and retries the // commit on UnknownTransactionCommitResult for you. Requires a replica set / sharded cluster. const session = db.getMongo().startSession(); try { session.withTransaction(() => { const orders = session.getDatabase("shop").orders; const ledger = session.getDatabase("shop").ledger; orders.updateOne({ _id: orderId, status: "pending" }, { $set: { status: "paid" } }, { session }); ledger.insertOne({ orderId, amount: NumberDecimal("9.99"), at: new Date() }, { session }); }, { readConcern: { level: "snapshot" }, writeConcern: { w: "majority" } }); } finally { session.endSession(); } // If both writes target ONE document, drop the transaction — that write is already atomic. ``` ### `bulkWrite` upsert ```javascript db.inventory.bulkWrite([ { updateOne: { filter: { sku: "ABC-1" }, update: { $inc: { qty: 5 }, $setOnInsert: { createdAt: new Date() } }, upsert: true }} ], { ordered: false }) // ordered:false keeps going past one failed op and parallelizes ``` ### Change stream (resumable tail) ```javascript // Watch only the events you care about; persist resumeToken to restart without gaps. const cs = db.orders.watch([{ $match: { operationType: { $in: ["insert", "update"] } } }]); while (cs.hasNext()) { const change = cs.next(); /* process; save change._id as resume token */ } ``` More variants (`$facet`, window functions, `$merge`/`$out`, vector search) live in the references. ## Anti-patterns / rationalizations → STOP | Rationalization | Reality → STOP | | --- | --- | | "Embed all the comments, it's one read" | Array grows unbounded toward 16 MB and bloats every post read. Reference or bucket. | | "`$lookup` is just a JOIN, use it everywhere" | Mongo is not relational; per-document `$lookup` is expensive. Prefer modeling (extended reference) so the read needs no join. | | "Wrap this single-document update in a transaction to be safe" | A single-doc write is already atomic. The transaction adds latency and a replica-set requirement for zero gain. | | "`COLLSCAN` is fine, it's fast on my 100 docs" | It is O(n); at 4M docs it is a full table read. Add the index now and prove `IXSCAN`. | | "Set `allowDiskUse:true` and the slow pipeline is fixed" | That masks a missing `$match` index by spilling to disk. Fix stage order / add the index first. | | "Store the price as a number, round on display" | JS doubles drift across `$sum`/`$inc`. Use `NumberDecimal` (`Decimal128`). | | "Group the whole collection, no `$match`" | A blocking `$group` over everything blows the 100 MB/stage limit. `$match` first to shrink it. | | "One collection for users, orders, logs — fewer to manage" | Mixed shapes kill index selectivity and balloon working set. Split by access pattern. | | "`$where` lets me run a quick JS predicate" | Runs JS per document, no index, a server-side injection surface. Use query operators / `$expr`. | | "Index every field just in case" | Each index is a write tax and RAM cost; unused indexes return nothing. Index for real query shapes only. | ## Quick reference ### Read/write concern matrix | Need | Write concern | Read concern | Note |