How to avoid massive number of files open at once?Issues with MongoDB install on Ubuntu 8.04 LTSHow would I template an SLS using saltstackReplicated MongoDB server slower than simple shardsEnabling WiredTiger engine in MongoDB 3MongoDB connections are failing with “SocketException… SEND_ERROR”MongoDB secondary crashes on initial sync because of too many journal files on RAID 10Why Does MongoDB Give an Out Of Disk Space Error When Space Is Available?WiredTiger storage engine cause delay in MongoDB startMongoDB “top” command shows dropped databasesRestore mongoDB by --repair and WiredTiger
Collision domain question
What is monoid homomorphism exactly?
Reverse ColorFunction or ColorData
How important are good looking people in a novel/story?
Is it normal for gliders not to have attitude indicators?
TIP120 Transistor + Solenoid Failing Randomly
Which "exotic salt" can lower water's freezing point by –70 °C?
Is crescere the correct word meaning to to grow or cultivate?
My large rocket is still flipping over
Huffman Code in C++
Why does blending blueberries, milk, banana and vanilla extract cause the mixture to have a yogurty consistency?
Can an Iranian citizen enter the USA on a Dutch passport?
Primes in a Diamond
How long does it take a postcard to get from USA to Germany?
What happens if I accidentally leave an app running and click "Install Now" in Software Updater?
Transistor gain, what if there is not enough current?
Copper as an adjective to refer to something made of copper
How to use awk to extract data from a file based on the content of another file?
Is there a reason why Turkey took the Balkan territories of the Ottoman Empire, instead of Greece or another of the Balkan states?
Has the United States ever had a non-Christian President?
As a GM, is it bad form to ask for a moment to think when improvising?
Two denim hijabs
How is Pauli's exclusion principle still valid in these cases?
Why increasing of the temperature of the objects like wood, paper etc. doesn't fire them?
How to avoid massive number of files open at once?
Issues with MongoDB install on Ubuntu 8.04 LTSHow would I template an SLS using saltstackReplicated MongoDB server slower than simple shardsEnabling WiredTiger engine in MongoDB 3MongoDB connections are failing with “SocketException… SEND_ERROR”MongoDB secondary crashes on initial sync because of too many journal files on RAID 10Why Does MongoDB Give an Out Of Disk Space Error When Space Is Available?WiredTiger storage engine cause delay in MongoDB startMongoDB “top” command shows dropped databasesRestore mongoDB by --repair and WiredTiger
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
I've been running MongoDB for several years under the MMAPv1 storage engine, but now need to upgrade to WiredTiger. (Mainly so I can use event streams, but also because mmapv1 has been deprecated).
My application uses a handful of collections for every customer which means a LOT of collections. Due to the namespace file limits of MMAPv1 these are split across multiple databases. Sharding isn't necessary because although there are many collections, they are not large. Basically, this works fine and is stable.
However, switching to WiredTiger in my dev environment (which is Mac OS) I notice that not only does WiredTiger use a separate file for every collection and every index, but starting the mongod process I see that ALL FILES ARE OPEN before any collections are accessed.
I estimate that migrating my production environment (CentOS 7) to use wiredTiger I could end up with close to a million files, and this will grow.
I'm aware of the ulimit issue and have read the documentation. The advice says that the recommended ulimit is 64,000. It concerns me that this is so much lower than what I'll end up with.
So my questions ...
Is it normal that ALL files are open at once? or does it sound like I'm doing something wrong?
Is this only a Mac issue? (I'm yet to test a migration on Linux)
If I set a ridiculously high
ulimit, would this impact the server in any negative way?
mongodb wiredtiger
asked Apr 27 at 19:21
TimTim
2621515
add a comment |
I've been running MongoDB for several years under the MMAPv1 storage engine, but now need to upgrade to WiredTiger. (Mainly so I can use event streams, but also because mmapv1 has been deprecated).
My application uses a handful of collections for every customer which means a LOT of collections. Due to the namespace file limits of MMAPv1 these are split across multiple databases. Sharding isn't necessary because although there are many collections, they are not large. Basically, this works fine and is stable.
However, switching to WiredTiger in my dev environment (which is Mac OS) I notice that not only does WiredTiger use a separate file for every collection and every index, but starting the mongod process I see that ALL FILES ARE OPEN before any collections are accessed.
I estimate that migrating my production environment (CentOS 7) to use wiredTiger I could end up with close to a million files, and this will grow.
I'm aware of the ulimit issue and have read the documentation. The advice says that the recommended ulimit is 64,000. It concerns me that this is so much lower than what I'll end up with.
So my questions ...
Is it normal that ALL files are open at once? or does it sound like I'm doing something wrong?
Is this only a Mac issue? (I'm yet to test a migration on Linux)
If I set a ridiculously high
ulimit, would this impact the server in any negative way?
mongodb wiredtiger
asked Apr 27 at 19:21
TimTim
2621515
add a comment |
I've been running MongoDB for several years under the MMAPv1 storage engine, but now need to upgrade to WiredTiger. (Mainly so I can use event streams, but also because mmapv1 has been deprecated).
My application uses a handful of collections for every customer which means a LOT of collections. Due to the namespace file limits of MMAPv1 these are split across multiple databases. Sharding isn't necessary because although there are many collections, they are not large. Basically, this works fine and is stable.
However, switching to WiredTiger in my dev environment (which is Mac OS) I notice that not only does WiredTiger use a separate file for every collection and every index, but starting the mongod process I see that ALL FILES ARE OPEN before any collections are accessed.
I estimate that migrating my production environment (CentOS 7) to use wiredTiger I could end up with close to a million files, and this will grow.
I'm aware of the ulimit issue and have read the documentation. The advice says that the recommended ulimit is 64,000. It concerns me that this is so much lower than what I'll end up with.
So my questions ...
Is it normal that ALL files are open at once? or does it sound like I'm doing something wrong?
Is this only a Mac issue? (I'm yet to test a migration on Linux)
If I set a ridiculously high
ulimit, would this impact the server in any negative way?
mongodb wiredtiger
asked Apr 27 at 19:21
TimTim
2621515
I've been running MongoDB for several years under the MMAPv1 storage engine, but now need to upgrade to WiredTiger. (Mainly so I can use event streams, but also because mmapv1 has been deprecated).
My application uses a handful of collections for every customer which means a LOT of collections. Due to the namespace file limits of MMAPv1 these are split across multiple databases. Sharding isn't necessary because although there are many collections, they are not large. Basically, this works fine and is stable.
However, switching to WiredTiger in my dev environment (which is Mac OS) I notice that not only does WiredTiger use a separate file for every collection and every index, but starting the mongod process I see that ALL FILES ARE OPEN before any collections are accessed.
I estimate that migrating my production environment (CentOS 7) to use wiredTiger I could end up with close to a million files, and this will grow.
I'm aware of the ulimit issue and have read the documentation. The advice says that the recommended ulimit is 64,000. It concerns me that this is so much lower than what I'll end up with.
So my questions ...
Is it normal that ALL files are open at once? or does it sound like I'm doing something wrong?
Is this only a Mac issue? (I'm yet to test a migration on Linux)
If I set a ridiculously high
ulimit, would this impact the server in any negative way?
mongodb wiredtiger
mongodb wiredtiger
asked Apr 27 at 19:21
TimTim
2621515
asked Apr 27 at 19:21
TimTim
2621515
asked Apr 27 at 19:21
TimTim
2621515
asked Apr 27 at 19:21
TimTim
2621515
asked Apr 27 at 19:21
TimTim
2621515
2621515
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Generally, mongodb use pool of connection, You can limit the number of incoming connections using the net.maxIncomingConnections run-time option. By restricting the number of incoming connections you can prevent a cascade effect where the mongos creates too many connections on the mongod instances.
Regarding your questions my answers inline:
Is it normal that ALL files are open at once? or does it sound like I'm doing something wrong?
My Ans: it is not wrong process make persistent connections(Pool) since making connection is costlier operation.
Is this only a Mac issue? (I'm yet to test a migration on Linux):
My Ans: It will work on same way only changes would be limits for each environments.
If I set a ridiculously high ulimit, would this impact the server in any negative way?
My Ans: Once change the limit it surely impact system resource consumption, try to get number of connection require by your application with open threads and tune settings accordingly.
Hope above answers will help you.
answered Apr 28 at 8:11
asktyagiasktyagi
976
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "2"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f964872%2fhow-to-avoid-massive-number-of-files-open-at-once%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Generally, mongodb use pool of connection, You can limit the number of incoming connections using the net.maxIncomingConnections run-time option. By restricting the number of incoming connections you can prevent a cascade effect where the mongos creates too many connections on the mongod instances.
Regarding your questions my answers inline:
Is it normal that ALL files are open at once? or does it sound like I'm doing something wrong?
My Ans: it is not wrong process make persistent connections(Pool) since making connection is costlier operation.
Is this only a Mac issue? (I'm yet to test a migration on Linux):
My Ans: It will work on same way only changes would be limits for each environments.
If I set a ridiculously high ulimit, would this impact the server in any negative way?
My Ans: Once change the limit it surely impact system resource consumption, try to get number of connection require by your application with open threads and tune settings accordingly.
Hope above answers will help you.
answered Apr 28 at 8:11
asktyagiasktyagi
976
add a comment |
Generally, mongodb use pool of connection, You can limit the number of incoming connections using the net.maxIncomingConnections run-time option. By restricting the number of incoming connections you can prevent a cascade effect where the mongos creates too many connections on the mongod instances.
Regarding your questions my answers inline:
Is it normal that ALL files are open at once? or does it sound like I'm doing something wrong?
My Ans: it is not wrong process make persistent connections(Pool) since making connection is costlier operation.
Is this only a Mac issue? (I'm yet to test a migration on Linux):
My Ans: It will work on same way only changes would be limits for each environments.
If I set a ridiculously high ulimit, would this impact the server in any negative way?
My Ans: Once change the limit it surely impact system resource consumption, try to get number of connection require by your application with open threads and tune settings accordingly.
Hope above answers will help you.
answered Apr 28 at 8:11
asktyagiasktyagi
976
add a comment |
Generally, mongodb use pool of connection, You can limit the number of incoming connections using the net.maxIncomingConnections run-time option. By restricting the number of incoming connections you can prevent a cascade effect where the mongos creates too many connections on the mongod instances.
Regarding your questions my answers inline:
Is it normal that ALL files are open at once? or does it sound like I'm doing something wrong?
My Ans: it is not wrong process make persistent connections(Pool) since making connection is costlier operation.
Is this only a Mac issue? (I'm yet to test a migration on Linux):
My Ans: It will work on same way only changes would be limits for each environments.
If I set a ridiculously high ulimit, would this impact the server in any negative way?
My Ans: Once change the limit it surely impact system resource consumption, try to get number of connection require by your application with open threads and tune settings accordingly.
Hope above answers will help you.
answered Apr 28 at 8:11
asktyagiasktyagi
976
Generally, mongodb use pool of connection, You can limit the number of incoming connections using the net.maxIncomingConnections run-time option. By restricting the number of incoming connections you can prevent a cascade effect where the mongos creates too many connections on the mongod instances.
Regarding your questions my answers inline:
Is it normal that ALL files are open at once? or does it sound like I'm doing something wrong?
My Ans: it is not wrong process make persistent connections(Pool) since making connection is costlier operation.
Is this only a Mac issue? (I'm yet to test a migration on Linux):
My Ans: It will work on same way only changes would be limits for each environments.
If I set a ridiculously high ulimit, would this impact the server in any negative way?
My Ans: Once change the limit it surely impact system resource consumption, try to get number of connection require by your application with open threads and tune settings accordingly.
Hope above answers will help you.
answered Apr 28 at 8:11
asktyagiasktyagi
976
edited Apr 28 at 9:40
answered Apr 28 at 8:11
asktyagiasktyagi
976
answered Apr 28 at 8:11
asktyagiasktyagi
976
answered Apr 28 at 8:11
asktyagiasktyagi
976
976
add a comment |
add a comment |
Thanks for contributing an answer to Server Fault!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f964872%2fhow-to-avoid-massive-number-of-files-open-at-once%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
