Mongodb Partitioning

At Whoz , we build a SaaS platform that helps professional services companies manage their talent staffing. At the heart of our product lies a concept called a worklog — a record of time spent by a user on a given activity. Every consultant, every day, on every project, generates worklogs. It sounds simple. And for years, it was. Then the numbers caught up with us. The Problem: A Collection That Never Stops Growing Our worklog MongoDB collection had reached 530 million documents , representing just over 32 GB of data. And the growth rate was accelerating — not just because we were onboarding more clients, but because users were increasingly splitting their activity into finer-grained entries, generating more worklogs per person per day than ever before. A worklog document looks roughly like this: { "date" : "2024-03-15" , "talentId" : "abc123" , "workspaceId" : "ws456" , "duration" : 0.5 , "activityType" : "TASK" , "taskId" : "task789" } Simple enough. But at 530 million of them, even the most routine operations become painful: Backup : nearly 1 hour Restore : up to 4 hours Schema migrations : we hadn't dared run one at full scale yet — and that alone was a warning sign Every year, the collection grows faster than the year before. The backup and restore windows were becoming operationally risky. We needed to act. Exploring Our Options We identified three potential approaches before settling on a solution. Option 1 — MongoDB Sharding Sharding is MongoDB's native horizontal scaling mechanism. It distributes a collection across multiple shards, each backed by its own replica set. On paper, it looked like a match. In practice, we ran into a fundamental mismatch with our actual needs. Our core issue wasn't query throughput — worklogs from three years ago are rarely queried, and when they are, performance expectations are low. Our issue was operational overhead : backup time, restore time, and the cost of running large batch operations over the full dataset. Sharding woul