Cut Your MongoDB Costs by 79% with Shape‑First Optimizations

Protect your database from future fires to avoid high-scale capital loss at your Series A stage

The Day the Bill Went Nuclear

The call came through at 2:17.

Atlas caused another unsolicited production cluster scale-up that resulted in an M60 machine at a monthly cost of $15k. The board wanted to know why burn increased by 20% while M60 serves as an expensive $15 k/month system.

I opened the profiler:

db.system.profile.aggregate([
  { $match: { millis: { $gt: 100 } } },
  { $group: {
      _id: { op: "$op", ns: "$ns", query: "$command.filter" },
      n: { $sum: 1 },
      avgMs: { $avg: "$millis" }
  }},
  { $sort: { n: -1 } }, { $limit: 10 }
]).pretty();

Each dashboard widget required the offender to pull in a total of 1.7 GB every minute before rendering. The sheer quantity of memory usage created mountain peaks in the graph that resembled Everest.

The M30 servers now operate with one of these clusters. The solution did not entail an increase in shards. Three common flaws known as shape crimes existed in the codebase before elimination.

Crime Scene Investigation

2.1 N + 1 Query Tsunami

This is recognized as an anti-pattern—when ordering one set of orders requires running N separate queries to retrieve order lines.

// Incorrect:  Orders   +   1 000 extra queries
const orders = await db.orders.find({ userId }).toArray();
for (const o of orders) {
  o.lines = await db.orderLines.find({ orderId: o._id }).toArray();
}

Hidden taxes

Refactor (4 lines):

// Success: Single round‑trip, 1 read unit per order
db.orders.aggregate([
  { $match: { userId } },
  { $lookup: {
      from: "orderLines",
      localField: "_id",
      foreignField: "orderId",
      as: "lines"
  }},
  { $project: { lines: 1, total: 1, ts: 1 } }
]);

Latency p95 dropped from 2 300 ms to 160 ms.

Atlas read‑ops: 101 → 1. That’s 99 % off—no coupon code needed.

2.2 Unbounded Query Firehose

Sure—just not in a single cursor.

// Failure: Streams 30 months of data through the API gateway
db.events.find({ userId }).toArray();

Fix: hard‑cap the batch and project only the fields you render.

db.events.find(
  { userId, ts: { $gte: ISODate(new Date() - 1000*60*60*24*30) } },
  { _id: 0, ts: 1, page: 1, ref: 1 }     // projection
).sort({ ts: -1 }).limit(1_000);

Then let Mongo clean up behind you:

// 90‑day sliding window
db.events.createIndex({ ts: 1 }, { expireAfterSeconds: 60*60*24*90 });

2.3 Jumbo‑Document Money Pit

Mongo caps documents at 16 MB, but anything over 256 KB is already a red flag.

{
  "_id": "...",
  "type": "invoice",
  "customer": { /* 700 kB */ },
  "pdf": BinData(0,"..."),        // 4 MB binary
  "history": [ /* 1 200 delta rows */ ],
  "ts": ISODate()
}

Why it hurts

1. The whole doc is paged in even if you read one field.

2. WiredTiger can’t store as many docs per page → lower cache hit ratio.

3. Index entries are huge → bloom filter misses → more disk seeks.

Solution: schema‑by‑access‑pattern:

graph TD
  Invoice[(invoices<br/>&lt;2 kB)] -->|ref| Hist[history<br/>&lt;1 kB * N]
  Invoice -->|ref| Bin[pdf‑store (S3/GridFS)]

Four More Shape Crimes You’re Probably Guilty Of

1. Low‑cardinality index head ({ type: 1, ts: -1 })—re‑order it to { userId: 1, ts: -1 }.
2. $regex starts‑with on non‑indexed field—string scan from hell.
3. findOneAndUpdate queue—document‑level locking bottleneck; use Redis/Kafka.
4. skip + large offset pagination—Mongo must count every skipped doc; switch to range (ts, _id) cursors.

4 · Cost Anatomy 101

Let’s do the math.

Total: $1,643.

Apply the fixes:

• Reads fall 70 % → $210
• Transfer falls 80 % → $75
• Storage falls 60 % → $192

48‑Hour Rescue Sprint (Battle‑tested Timeline)

Self‑Audit Checklist—Pin It Above Your Desk

• Largest doc ÷ median > 10? → Refactor.

• Cursor returns > 1 000 docs? → Paginate.

• TTL on every event/store table? (Yes/No)

• Any index where cardinality < 10 %? → Drop/re‑order.

• Profiler slowops > 1 % total ops? → Optimize or cache.

If primary cache hits remain under 90% it is wise to separate collections or add additional RAM memory post fixes.

Place the checklist on your laptop with adhesive glue after laminating it for printing.

Why Shape Beats Indexes

MongoDB’s query planner does a cost‑based search across candidate plans. The cost vector includes:

workUnits = ixScans + fetches + sorts + #docs returned

Indexes only reduce ixScans. Bad shape inflates fetches and sorts, which often dominate. Example:

db.logs.find(
  { ts: { $gte: start, $lt: end }, level: "error" }
).sort({ level: 1, ts: -1 });

Index { level: 1, ts: -1 } does not help planner avoid fetching every document when it adds a predicate to an unmentioned array field in your projections. Net result: 20 k fetches for 200 hits. Index should precede shape operations in daily operations.

Live Metrics You Should Be Watching (Grafana PromQL)

# WiredTiger cache hit ratio
(rate(wiredtiger_blockmanager_blocks_read[1m]) /
 (rate(wiredtiger_blockmanager_blocks_read[1m]) +
  rate(wiredtiger_blockmanager_blocks_read_from_cache[1m]))
) < 0.10

Alert if > 10 % misses for 5 m.

# Docs scanned vs returned
rate(mongodb_ssm_metrics_documents[1m]{state="scanned"}) /
rate(mongodb_ssm_metrics_documents[1m]{state="returned"}) > 100

Hands‑On: Thin‑Slice Migration Script

Need to crack a 1‑TB events collection into clicks, views, logins without downtime? Use the double‑write / backfill pattern.

// 1. Add trigger
const changeStream = db.events.watch([], { fullDocument: 'updateLookup' });
changeStream.on('change', ev => {
  const dest = db[`${ev.fullDocument.type}s`];
  dest.insertOne({ ...ev.fullDocument });
});

// 2. Backfill historical in chunks
let lastId = ObjectId("000000...");
while (true) {
  const batch = db.events.find({_id: {$gt: lastId}}).sort({_id: 1}).limit(10_000);
  if (!batch.hasNext()) break;
  const docs = batch.toArray();
  docs.forEach(d => db[`${d.type}s`].insertOne(d));
  lastId = docs[docs.length - 1]._id;
}

Zero downtime, minimal extra storage (thanks to TTL), everyone sleeps.

When Sharding Is the Answer

According to the rule of thumb you should shard only if you verify one of these conditions comes true after optimizing your database:

1. The system operates with a working set representing more than 80 percent of RAM regardless of the cache performance rate.

2. The system generates more than 15 thousand operations per second in its peak write performance when using one primary server.

3. Your main priorities should be maintaining sub-70-millisecond multi-region latency because high AWS billing costs do not represent your critical concern.

The decision should be simple when the conditions do not match these rules.

Case Study Wrap‑Up

Takeaway: Debt vs. Death Spiral

The requirement to deliver quickly stands as mandatory, but maintaining poor quality work belongs to voluntary debt accumulation. Cloud provider charges you compound interest that accumulates with a yearly percentage rate of 1000 % on your unpaid debt. We examined the high‑interest credit cards of MongoDB as they represent the five shape crimes we studied. Shedding these debts within your current sprint period will result in grateful technical and financial statements.

You need to open the profiler to work with the $lookup pistons while adding TTL dust, and then deploy your project in lean mode. Your board and your dev team and your pager at 02:17 will obtain quality rest.

Proceed with the refactoring of your code until the next autoscaling incident happens.