I have a Linux box running 5 instances of MongoDB, each one for a different environment. The server works fine during the day, most of the time at ~90% memory usage. But recently I started seeing that one (sometimes more) of the
mongod instances running there died every night when my backup script ran, courtesy of the OS’s out-of-memory Killer. Thanks to Azure I can see the pattern very clearly:
After noticing that swap wasn’t enabled for the server, enabling it, and seeing that
mongod processes kept dying nightly, I discovered that MongoDB does not use swap because it uses memory-mapped files:
Nevertheless, systems running MongoDB do not need swap for routine operation. Database files are memory-mapped and should constitute most of your MongoDB memory use. Therefore, it is unlikely that mongod will ever use any swap space in normal operation. The operating system will release memory from the memory mapped files without needing swap and MongoDB can write data to the data files without needing the swap system.
So enabling swap didn’t solve my problem of dying instances, but it’s something that the server should have anyway so I left it enabled.
I kept reading MongoDB’s official documentation and ran into this:
The default WiredTiger internal cache size value assumes that there is a single mongod instance per machine. If a single machine contains multiple MongoDB instances, then you should decrease the setting to accommodate the other mongod instances.
That sounded pretty promising! I started by looking at my instances to see what the current value was. You can do that by running
db.serverStatus().wiredTiger.cache in the Mongo shell, and looking for property “maximum bytes configured” in the output document.
Sure enough, the server has 16GB of RAM, and those ~7.8GB are more or less what I’d expect based on the
0.5 * (RAM-GB - 1GB) calculation in the docs. The issue is that all five instances have the same value!
So off I went and changed that setting to 3GB instead… and voilà! Stable DB server again, even with 5 separate instances of Mongo running in there.
Thanks for the post.
I can’t find the “MongoDB does not use swap” documentation that’s linked here. I was wondering what version of MongoDB does that text snippet come from.
Also, did you tweak around vm.swappiness setting on Linux to see if that made any difference – https://docs.mongodb.com/manual/administration/production-notes/#swap
Hey! When I ran into this we were working with 3.4 and 3.6, I think it applied to both. I managed to find it in their legacy docs: https://docs.mongodb.com/v3.6/faq/diagnostics/#do-i-need-to-configure-swap-space, and some references to memory mapped files in the 4.0 docs as well: https://docs.mongodb.com/v3.6/faq/diagnostics/#do-i-need-to-configure-swap-space. Didn’t even know there was a 5.0 already :). Just now that I found that piece of documentation again I noticed that it’s under the MMAPv1 storage engine, so I’m not sure if it really applied in our case. The rest of the post deals with the recommendation for WiredTiger though, so that explains why it helped. About the vm.swappiness setting, I didn’t try it; it’s the first time I hear of it, to be honest.