High %wa in top on KVM hostmemory leak? RHEL 5.5. RSS show ok, almost no free memory left, swap used heavilyRAID setup for high speed readingcpusets versus top '1'Anyone else experiencing high rates of Linux server crashes during a leap second day?High load cause?Amazon EC2 Server running very slowOpenNebula (KVM) + OpenvSwitch, high CPU load on high bandwidth useRandom High CPU loadLinux KVM QEMU Host with high CPU load
Does cooling a potato change the nature of its carbohydrates?
Weird thing in 737 cabin
How to sort human readable size
What is this airplane that sits in front of Barringer High School in Newark, NJ?
Basic power tool set for Home repair and simple projects
Is this a valid proof that A = B given A ∩ B = A ∪ B?
How to address players struggling with simple controls?
If the mass of the Earth is decreasing by sending debris in space, does its angular momentum also decrease?
Are there examples of rowers who also fought?
Can a character with the Polearm Master feat make an opportunity attack against an invisible creature that enters their reach?
How to avoid offending original culture when making conculture inspired from original
In a Fish that is not a Fish
Having some issue with notation in a Hilbert space
How to recover a single blank shot from a film camera
I have found ports on my Samsung smart tv running a display service. What can I do with it?
Justifying Affordable Bespoke Spaceships
What is the precise meaning of "подсел на мак"?
Simplify, equivalent for (p ∨ ¬q) ∧ (¬p ∨ ¬q)
What is the context for Napoleon's quote "[the Austrians] did not know the value of five minutes"?
I wish, I yearn, for an answer to this riddle
Would a 7805 5v regulator drain a 9v battery?
Is there any possible way to get these hearts as Adult Link?
Is it a bad idea to have a pen name with only an initial for a surname?
Time travel short story where someone from the past follows the travelers back
High %wa in top on KVM host
memory leak? RHEL 5.5. RSS show ok, almost no free memory left, swap used heavilyRAID setup for high speed readingcpusets versus top '1'Anyone else experiencing high rates of Linux server crashes during a leap second day?High load cause?Amazon EC2 Server running very slowOpenNebula (KVM) + OpenvSwitch, high CPU load on high bandwidth useRandom High CPU loadLinux KVM QEMU Host with high CPU load
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
I have a server running several KVM virtualized guests that's reporting a high %wa in top. The server hardware is as follows:
- 4x2TB in hardware RAID10
- 64GB RAM
- Dual E5-2620
top output
% top - 03:56:51 up 12:24, 1 user, load average: 17.13, 14.02, 12.88
Tasks: 582 total, 2 running, 563 sleeping, 17 stopped, 0 zombie
Cpu(s): 1.3%us, 3.4%sy, 0.0%ni, 64.9%id, 30.4%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 65881560k total, 25619484k used, 40262076k free, 781820k buffers
Swap: 4194296k total, 240788k used, 3953508k free, 7971412k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3203 qemu 20 0 1930m 1.0g 5064 S 31.8 1.7 272:16.38 qemu-kvm
64232 qemu 20 0 1732m 1.0g 4500 S 12.9 1.7 90:42.31 qemu-kvm
90795 qemu 20 0 2203m 1.0g 4508 S 12.9 1.6 1:45.63 qemu-kvm
32095 qemu 20 0 1602m 1.0g 4500 S 12.6 1.7 100:41.43 qemu-kvm
89081 qemu 20 0 2133m 1.0g 4512 S 9.9 1.6 4:47.92 qemu-kvm
71839 qemu 20 0 1592m 1.0g 4500 S 9.6 1.6 14:44.13 qemu-kvm
66958 qemu 20 0 1592m 1.0g 4500 S 9.3 1.6 108:36.42 qemu-kvm
69070 qemu 20 0 1592m 1.0g 4496 S 9.3 1.6 16:14.73 qemu-kvm
23726 qemu 20 0 1518m 1.0g 4488 R 7.9 1.6 82:14.87 qemu-kvm
3222 root 20 0 0 0 0 S 1.3 0.0 7:30.04 kvm-pit-wq
80204 qemu 20 0 4490m 2.7g 4504 S 1.3 4.3 11:43.57 qemu-kvm
91488 root 20 0 15428 1664 952 R 0.7 0.0 0:02.88 top
347 root 39 19 0 0 0 S 0.3 0.0 1:28.38 kipmi0
5534 qemu 20 0 3861m 2.9g 4528 S 0.3 4.7 10:08.12 qemu-kvm
5554 root 20 0 0 0 0 S 0.3 0.0 0:08.51 vhost-5534
23746 root 20 0 0 0 0 S 0.3 0.0 0:04.74 vhost-23726
64252 root 20 0 0 0 0 S 0.3 0.0 0:07.05 vhost-64232
We've been having this problem since we've received this server, please help me to figure it out.
Screenshot from iostat output:
linux performance raid
edited Apr 13 '13 at 11:23
DukeLion
2,97611216
asked Apr 13 '13 at 1:02
George YoussryGeorge Youssry
11
add a comment |
I have a server running several KVM virtualized guests that's reporting a high %wa in top. The server hardware is as follows:
- 4x2TB in hardware RAID10
- 64GB RAM
- Dual E5-2620
top output
% top - 03:56:51 up 12:24, 1 user, load average: 17.13, 14.02, 12.88
Tasks: 582 total, 2 running, 563 sleeping, 17 stopped, 0 zombie
Cpu(s): 1.3%us, 3.4%sy, 0.0%ni, 64.9%id, 30.4%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 65881560k total, 25619484k used, 40262076k free, 781820k buffers
Swap: 4194296k total, 240788k used, 3953508k free, 7971412k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3203 qemu 20 0 1930m 1.0g 5064 S 31.8 1.7 272:16.38 qemu-kvm
64232 qemu 20 0 1732m 1.0g 4500 S 12.9 1.7 90:42.31 qemu-kvm
90795 qemu 20 0 2203m 1.0g 4508 S 12.9 1.6 1:45.63 qemu-kvm
32095 qemu 20 0 1602m 1.0g 4500 S 12.6 1.7 100:41.43 qemu-kvm
89081 qemu 20 0 2133m 1.0g 4512 S 9.9 1.6 4:47.92 qemu-kvm
71839 qemu 20 0 1592m 1.0g 4500 S 9.6 1.6 14:44.13 qemu-kvm
66958 qemu 20 0 1592m 1.0g 4500 S 9.3 1.6 108:36.42 qemu-kvm
69070 qemu 20 0 1592m 1.0g 4496 S 9.3 1.6 16:14.73 qemu-kvm
23726 qemu 20 0 1518m 1.0g 4488 R 7.9 1.6 82:14.87 qemu-kvm
3222 root 20 0 0 0 0 S 1.3 0.0 7:30.04 kvm-pit-wq
80204 qemu 20 0 4490m 2.7g 4504 S 1.3 4.3 11:43.57 qemu-kvm
91488 root 20 0 15428 1664 952 R 0.7 0.0 0:02.88 top
347 root 39 19 0 0 0 S 0.3 0.0 1:28.38 kipmi0
5534 qemu 20 0 3861m 2.9g 4528 S 0.3 4.7 10:08.12 qemu-kvm
5554 root 20 0 0 0 0 S 0.3 0.0 0:08.51 vhost-5534
23746 root 20 0 0 0 0 S 0.3 0.0 0:04.74 vhost-23726
64252 root 20 0 0 0 0 S 0.3 0.0 0:07.05 vhost-64232
We've been having this problem since we've received this server, please help me to figure it out.
Screenshot from iostat output:
linux performance raid
edited Apr 13 '13 at 11:23
DukeLion
2,97611216
asked Apr 13 '13 at 1:02
George YoussryGeorge Youssry
11
Just two comments: a) for a RAID10 you need 2n HDDs, w/ n>=2, so whatever you're running, it's not a RAID10 w/ only two disks. b) I/O wait can occur w/ network devices as well as with local storage.
– tink
Apr 13 '13 at 6:19
for the type of cpu and memory on your machine i would look into having a 6 or 8 drive raid 10. to get a lot of perfomance if your gonna use raid 8 drives may wanna look into a raid 100
– WojonsTech
Apr 13 '13 at 8:21
That's not good to provide extra IO performance just because you have a lot of CPU and RAM, it's about application requirements, not ruthless 'cool server' competetion. Idea of RAID 100 is to spread data across multiple controllers using software raid0, so I doubt it's a common thing to use.
– DukeLion
Apr 17 '13 at 7:10
add a comment |
I have a server running several KVM virtualized guests that's reporting a high %wa in top. The server hardware is as follows:
- 4x2TB in hardware RAID10
- 64GB RAM
- Dual E5-2620
top output
% top - 03:56:51 up 12:24, 1 user, load average: 17.13, 14.02, 12.88
Tasks: 582 total, 2 running, 563 sleeping, 17 stopped, 0 zombie
Cpu(s): 1.3%us, 3.4%sy, 0.0%ni, 64.9%id, 30.4%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 65881560k total, 25619484k used, 40262076k free, 781820k buffers
Swap: 4194296k total, 240788k used, 3953508k free, 7971412k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3203 qemu 20 0 1930m 1.0g 5064 S 31.8 1.7 272:16.38 qemu-kvm
64232 qemu 20 0 1732m 1.0g 4500 S 12.9 1.7 90:42.31 qemu-kvm
90795 qemu 20 0 2203m 1.0g 4508 S 12.9 1.6 1:45.63 qemu-kvm
32095 qemu 20 0 1602m 1.0g 4500 S 12.6 1.7 100:41.43 qemu-kvm
89081 qemu 20 0 2133m 1.0g 4512 S 9.9 1.6 4:47.92 qemu-kvm
71839 qemu 20 0 1592m 1.0g 4500 S 9.6 1.6 14:44.13 qemu-kvm
66958 qemu 20 0 1592m 1.0g 4500 S 9.3 1.6 108:36.42 qemu-kvm
69070 qemu 20 0 1592m 1.0g 4496 S 9.3 1.6 16:14.73 qemu-kvm
23726 qemu 20 0 1518m 1.0g 4488 R 7.9 1.6 82:14.87 qemu-kvm
3222 root 20 0 0 0 0 S 1.3 0.0 7:30.04 kvm-pit-wq
80204 qemu 20 0 4490m 2.7g 4504 S 1.3 4.3 11:43.57 qemu-kvm
91488 root 20 0 15428 1664 952 R 0.7 0.0 0:02.88 top
347 root 39 19 0 0 0 S 0.3 0.0 1:28.38 kipmi0
5534 qemu 20 0 3861m 2.9g 4528 S 0.3 4.7 10:08.12 qemu-kvm
5554 root 20 0 0 0 0 S 0.3 0.0 0:08.51 vhost-5534
23746 root 20 0 0 0 0 S 0.3 0.0 0:04.74 vhost-23726
64252 root 20 0 0 0 0 S 0.3 0.0 0:07.05 vhost-64232
We've been having this problem since we've received this server, please help me to figure it out.
Screenshot from iostat output:
linux performance raid
edited Apr 13 '13 at 11:23
DukeLion
2,97611216
asked Apr 13 '13 at 1:02
George YoussryGeorge Youssry
11
I have a server running several KVM virtualized guests that's reporting a high %wa in top. The server hardware is as follows:
- 4x2TB in hardware RAID10
- 64GB RAM
- Dual E5-2620
top output
% top - 03:56:51 up 12:24, 1 user, load average: 17.13, 14.02, 12.88
Tasks: 582 total, 2 running, 563 sleeping, 17 stopped, 0 zombie
Cpu(s): 1.3%us, 3.4%sy, 0.0%ni, 64.9%id, 30.4%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 65881560k total, 25619484k used, 40262076k free, 781820k buffers
Swap: 4194296k total, 240788k used, 3953508k free, 7971412k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3203 qemu 20 0 1930m 1.0g 5064 S 31.8 1.7 272:16.38 qemu-kvm
64232 qemu 20 0 1732m 1.0g 4500 S 12.9 1.7 90:42.31 qemu-kvm
90795 qemu 20 0 2203m 1.0g 4508 S 12.9 1.6 1:45.63 qemu-kvm
32095 qemu 20 0 1602m 1.0g 4500 S 12.6 1.7 100:41.43 qemu-kvm
89081 qemu 20 0 2133m 1.0g 4512 S 9.9 1.6 4:47.92 qemu-kvm
71839 qemu 20 0 1592m 1.0g 4500 S 9.6 1.6 14:44.13 qemu-kvm
66958 qemu 20 0 1592m 1.0g 4500 S 9.3 1.6 108:36.42 qemu-kvm
69070 qemu 20 0 1592m 1.0g 4496 S 9.3 1.6 16:14.73 qemu-kvm
23726 qemu 20 0 1518m 1.0g 4488 R 7.9 1.6 82:14.87 qemu-kvm
3222 root 20 0 0 0 0 S 1.3 0.0 7:30.04 kvm-pit-wq
80204 qemu 20 0 4490m 2.7g 4504 S 1.3 4.3 11:43.57 qemu-kvm
91488 root 20 0 15428 1664 952 R 0.7 0.0 0:02.88 top
347 root 39 19 0 0 0 S 0.3 0.0 1:28.38 kipmi0
5534 qemu 20 0 3861m 2.9g 4528 S 0.3 4.7 10:08.12 qemu-kvm
5554 root 20 0 0 0 0 S 0.3 0.0 0:08.51 vhost-5534
23746 root 20 0 0 0 0 S 0.3 0.0 0:04.74 vhost-23726
64252 root 20 0 0 0 0 S 0.3 0.0 0:07.05 vhost-64232
We've been having this problem since we've received this server, please help me to figure it out.
Screenshot from iostat output:
linux performance raid
linux performance raid
edited Apr 13 '13 at 11:23
DukeLion
2,97611216
asked Apr 13 '13 at 1:02
George YoussryGeorge Youssry
11
edited Apr 13 '13 at 11:23
DukeLion
2,97611216
asked Apr 13 '13 at 1:02
George YoussryGeorge Youssry
11
edited Apr 13 '13 at 11:23
DukeLion
2,97611216
edited Apr 13 '13 at 11:23
DukeLion
2,97611216
edited Apr 13 '13 at 11:23
DukeLion
2,97611216
2,97611216
asked Apr 13 '13 at 1:02
George YoussryGeorge Youssry
11
asked Apr 13 '13 at 1:02
George YoussryGeorge Youssry
11
asked Apr 13 '13 at 1:02
George YoussryGeorge Youssry
11
11
Just two comments: a) for a RAID10 you need 2n HDDs, w/ n>=2, so whatever you're running, it's not a RAID10 w/ only two disks. b) I/O wait can occur w/ network devices as well as with local storage.
– tink
Apr 13 '13 at 6:19
for the type of cpu and memory on your machine i would look into having a 6 or 8 drive raid 10. to get a lot of perfomance if your gonna use raid 8 drives may wanna look into a raid 100
– WojonsTech
Apr 13 '13 at 8:21
That's not good to provide extra IO performance just because you have a lot of CPU and RAM, it's about application requirements, not ruthless 'cool server' competetion. Idea of RAID 100 is to spread data across multiple controllers using software raid0, so I doubt it's a common thing to use.
– DukeLion
Apr 17 '13 at 7:10
add a comment |
Just two comments: a) for a RAID10 you need 2n HDDs, w/ n>=2, so whatever you're running, it's not a RAID10 w/ only two disks. b) I/O wait can occur w/ network devices as well as with local storage.
– tink
Apr 13 '13 at 6:19
for the type of cpu and memory on your machine i would look into having a 6 or 8 drive raid 10. to get a lot of perfomance if your gonna use raid 8 drives may wanna look into a raid 100
– WojonsTech
Apr 13 '13 at 8:21
That's not good to provide extra IO performance just because you have a lot of CPU and RAM, it's about application requirements, not ruthless 'cool server' competetion. Idea of RAID 100 is to spread data across multiple controllers using software raid0, so I doubt it's a common thing to use.
– DukeLion
Apr 17 '13 at 7:10
Just two comments: a) for a RAID10 you need 2n HDDs, w/ n>=2, so whatever you're running, it's not a RAID10 w/ only two disks. b) I/O wait can occur w/ network devices as well as with local storage.
– tink
Apr 13 '13 at 6:19
Just two comments: a) for a RAID10 you need 2n HDDs, w/ n>=2, so whatever you're running, it's not a RAID10 w/ only two disks. b) I/O wait can occur w/ network devices as well as with local storage.
– tink
Apr 13 '13 at 6:19
for the type of cpu and memory on your machine i would look into having a 6 or 8 drive raid 10. to get a lot of perfomance if your gonna use raid 8 drives may wanna look into a raid 100
– WojonsTech
Apr 13 '13 at 8:21
for the type of cpu and memory on your machine i would look into having a 6 or 8 drive raid 10. to get a lot of perfomance if your gonna use raid 8 drives may wanna look into a raid 100
– WojonsTech
Apr 13 '13 at 8:21
That's not good to provide extra IO performance just because you have a lot of CPU and RAM, it's about application requirements, not ruthless 'cool server' competetion. Idea of RAID 100 is to spread data across multiple controllers using software raid0, so I doubt it's a common thing to use.
– DukeLion
Apr 17 '13 at 7:10
That's not good to provide extra IO performance just because you have a lot of CPU and RAM, it's about application requirements, not ruthless 'cool server' competetion. Idea of RAID 100 is to spread data across multiple controllers using software raid0, so I doubt it's a common thing to use.
– DukeLion
Apr 17 '13 at 7:10
add a comment |
2 Answers
2
active
oldest
votes
2x 4TB drives is a lot of space, but not much performance at all.
For example, a quick hunt around shows Western Digital make an enterprise 4TB disk in Nearline-SAS form, with 7,200 RPM spindle speed , and Wikipedia gives an estimate for that kind of drive handling 75 to 150 input/ouput operations per second (IOPS), depending how you estimate.
In a RAID-1, that gives you up to 300 requests per second serviced by the disks, with any more coming from the disk cache, controller cache, OS and service caches.
What do the virtual machines do? Unless you mean 2x 4TB pools, each made up of lots of smaller disks, I suspect you just don't have the storage performance to support a dozen virtual machines.
answered Apr 13 '13 at 1:31
TessellatingHecklerTessellatingHeckler
5,23432040
Thanks for your reply, I have another server with almost the same configuration but from different provider and all is going well, I think that there is something wrong with one of the hard-drives, they are Toshiba DT01ACA200, could you please let me know how to detect that this problem is from the HDD ?
– George Youssry
Apr 13 '13 at 1:55
Sorry, I don't know specifically how you can check that, apart from looking for error messages in the server log files such as /var/log/messages, or using a S.M.A.R.T. utility to query the drive status. Incidentally, those are consumer/desktop class drives, benchmarking 80 read IOPS and 150 write IOPS.
– TessellatingHeckler
Apr 13 '13 at 2:31
add a comment |
run iostat -xk 1 to see how utilised are your HDDs.
If you have low number of read/write requests with high utilisation % - your HDD might be broken.
Or there could be some heavy IO process overloading it.
UPD: You need to check status of your raid array, it might be rebuilding or malfunctioning
answered Apr 13 '13 at 5:06
DukeLionDukeLion
2,97611216
Ohh sorry we have 4x2 as it is RAID10, attached a screenshot from iostat -xk 1 command i.imgur.com/7g5mfem.jpg
– George Youssry
Apr 13 '13 at 7:02
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "2"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f498951%2fhigh-wa-in-top-on-kvm-host%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
2x 4TB drives is a lot of space, but not much performance at all.
For example, a quick hunt around shows Western Digital make an enterprise 4TB disk in Nearline-SAS form, with 7,200 RPM spindle speed , and Wikipedia gives an estimate for that kind of drive handling 75 to 150 input/ouput operations per second (IOPS), depending how you estimate.
In a RAID-1, that gives you up to 300 requests per second serviced by the disks, with any more coming from the disk cache, controller cache, OS and service caches.
What do the virtual machines do? Unless you mean 2x 4TB pools, each made up of lots of smaller disks, I suspect you just don't have the storage performance to support a dozen virtual machines.
answered Apr 13 '13 at 1:31
TessellatingHecklerTessellatingHeckler
5,23432040
Thanks for your reply, I have another server with almost the same configuration but from different provider and all is going well, I think that there is something wrong with one of the hard-drives, they are Toshiba DT01ACA200, could you please let me know how to detect that this problem is from the HDD ?
– George Youssry
Apr 13 '13 at 1:55
Sorry, I don't know specifically how you can check that, apart from looking for error messages in the server log files such as /var/log/messages, or using a S.M.A.R.T. utility to query the drive status. Incidentally, those are consumer/desktop class drives, benchmarking 80 read IOPS and 150 write IOPS.
– TessellatingHeckler
Apr 13 '13 at 2:31
add a comment |
2x 4TB drives is a lot of space, but not much performance at all.
For example, a quick hunt around shows Western Digital make an enterprise 4TB disk in Nearline-SAS form, with 7,200 RPM spindle speed , and Wikipedia gives an estimate for that kind of drive handling 75 to 150 input/ouput operations per second (IOPS), depending how you estimate.
In a RAID-1, that gives you up to 300 requests per second serviced by the disks, with any more coming from the disk cache, controller cache, OS and service caches.
What do the virtual machines do? Unless you mean 2x 4TB pools, each made up of lots of smaller disks, I suspect you just don't have the storage performance to support a dozen virtual machines.
answered Apr 13 '13 at 1:31
TessellatingHecklerTessellatingHeckler
5,23432040
Thanks for your reply, I have another server with almost the same configuration but from different provider and all is going well, I think that there is something wrong with one of the hard-drives, they are Toshiba DT01ACA200, could you please let me know how to detect that this problem is from the HDD ?
– George Youssry
Apr 13 '13 at 1:55
Sorry, I don't know specifically how you can check that, apart from looking for error messages in the server log files such as /var/log/messages, or using a S.M.A.R.T. utility to query the drive status. Incidentally, those are consumer/desktop class drives, benchmarking 80 read IOPS and 150 write IOPS.
– TessellatingHeckler
Apr 13 '13 at 2:31
add a comment |
2x 4TB drives is a lot of space, but not much performance at all.
For example, a quick hunt around shows Western Digital make an enterprise 4TB disk in Nearline-SAS form, with 7,200 RPM spindle speed , and Wikipedia gives an estimate for that kind of drive handling 75 to 150 input/ouput operations per second (IOPS), depending how you estimate.
In a RAID-1, that gives you up to 300 requests per second serviced by the disks, with any more coming from the disk cache, controller cache, OS and service caches.
What do the virtual machines do? Unless you mean 2x 4TB pools, each made up of lots of smaller disks, I suspect you just don't have the storage performance to support a dozen virtual machines.
answered Apr 13 '13 at 1:31
TessellatingHecklerTessellatingHeckler
5,23432040
2x 4TB drives is a lot of space, but not much performance at all.
For example, a quick hunt around shows Western Digital make an enterprise 4TB disk in Nearline-SAS form, with 7,200 RPM spindle speed , and Wikipedia gives an estimate for that kind of drive handling 75 to 150 input/ouput operations per second (IOPS), depending how you estimate.
In a RAID-1, that gives you up to 300 requests per second serviced by the disks, with any more coming from the disk cache, controller cache, OS and service caches.
What do the virtual machines do? Unless you mean 2x 4TB pools, each made up of lots of smaller disks, I suspect you just don't have the storage performance to support a dozen virtual machines.
answered Apr 13 '13 at 1:31
TessellatingHecklerTessellatingHeckler
5,23432040
answered Apr 13 '13 at 1:31
TessellatingHecklerTessellatingHeckler
5,23432040
answered Apr 13 '13 at 1:31
TessellatingHecklerTessellatingHeckler
5,23432040
answered Apr 13 '13 at 1:31
TessellatingHecklerTessellatingHeckler
5,23432040
5,23432040
Thanks for your reply, I have another server with almost the same configuration but from different provider and all is going well, I think that there is something wrong with one of the hard-drives, they are Toshiba DT01ACA200, could you please let me know how to detect that this problem is from the HDD ?
– George Youssry
Apr 13 '13 at 1:55
Sorry, I don't know specifically how you can check that, apart from looking for error messages in the server log files such as /var/log/messages, or using a S.M.A.R.T. utility to query the drive status. Incidentally, those are consumer/desktop class drives, benchmarking 80 read IOPS and 150 write IOPS.
– TessellatingHeckler
Apr 13 '13 at 2:31
add a comment |
Thanks for your reply, I have another server with almost the same configuration but from different provider and all is going well, I think that there is something wrong with one of the hard-drives, they are Toshiba DT01ACA200, could you please let me know how to detect that this problem is from the HDD ?
– George Youssry
Apr 13 '13 at 1:55
Sorry, I don't know specifically how you can check that, apart from looking for error messages in the server log files such as /var/log/messages, or using a S.M.A.R.T. utility to query the drive status. Incidentally, those are consumer/desktop class drives, benchmarking 80 read IOPS and 150 write IOPS.
– TessellatingHeckler
Apr 13 '13 at 2:31
Thanks for your reply, I have another server with almost the same configuration but from different provider and all is going well, I think that there is something wrong with one of the hard-drives, they are Toshiba DT01ACA200, could you please let me know how to detect that this problem is from the HDD ?
– George Youssry
Apr 13 '13 at 1:55
Thanks for your reply, I have another server with almost the same configuration but from different provider and all is going well, I think that there is something wrong with one of the hard-drives, they are Toshiba DT01ACA200, could you please let me know how to detect that this problem is from the HDD ?
– George Youssry
Apr 13 '13 at 1:55
Sorry, I don't know specifically how you can check that, apart from looking for error messages in the server log files such as /var/log/messages, or using a S.M.A.R.T. utility to query the drive status. Incidentally, those are consumer/desktop class drives, benchmarking 80 read IOPS and 150 write IOPS.
– TessellatingHeckler
Apr 13 '13 at 2:31
Sorry, I don't know specifically how you can check that, apart from looking for error messages in the server log files such as /var/log/messages, or using a S.M.A.R.T. utility to query the drive status. Incidentally, those are consumer/desktop class drives, benchmarking 80 read IOPS and 150 write IOPS.
– TessellatingHeckler
Apr 13 '13 at 2:31
add a comment |
run iostat -xk 1 to see how utilised are your HDDs.
If you have low number of read/write requests with high utilisation % - your HDD might be broken.
Or there could be some heavy IO process overloading it.
UPD: You need to check status of your raid array, it might be rebuilding or malfunctioning
answered Apr 13 '13 at 5:06
DukeLionDukeLion
2,97611216
Ohh sorry we have 4x2 as it is RAID10, attached a screenshot from iostat -xk 1 command i.imgur.com/7g5mfem.jpg
– George Youssry
Apr 13 '13 at 7:02
add a comment |
run iostat -xk 1 to see how utilised are your HDDs.
If you have low number of read/write requests with high utilisation % - your HDD might be broken.
Or there could be some heavy IO process overloading it.
UPD: You need to check status of your raid array, it might be rebuilding or malfunctioning
answered Apr 13 '13 at 5:06
DukeLionDukeLion
2,97611216
Ohh sorry we have 4x2 as it is RAID10, attached a screenshot from iostat -xk 1 command i.imgur.com/7g5mfem.jpg
– George Youssry
Apr 13 '13 at 7:02
add a comment |
run iostat -xk 1 to see how utilised are your HDDs.
If you have low number of read/write requests with high utilisation % - your HDD might be broken.
Or there could be some heavy IO process overloading it.
UPD: You need to check status of your raid array, it might be rebuilding or malfunctioning
answered Apr 13 '13 at 5:06
DukeLionDukeLion
2,97611216
run iostat -xk 1 to see how utilised are your HDDs.
If you have low number of read/write requests with high utilisation % - your HDD might be broken.
Or there could be some heavy IO process overloading it.
UPD: You need to check status of your raid array, it might be rebuilding or malfunctioning
answered Apr 13 '13 at 5:06
DukeLionDukeLion
2,97611216
edited Apr 13 '13 at 10:34
answered Apr 13 '13 at 5:06
DukeLionDukeLion
2,97611216
answered Apr 13 '13 at 5:06
DukeLionDukeLion
2,97611216
answered Apr 13 '13 at 5:06
DukeLionDukeLion
2,97611216
2,97611216
Ohh sorry we have 4x2 as it is RAID10, attached a screenshot from iostat -xk 1 command i.imgur.com/7g5mfem.jpg
– George Youssry
Apr 13 '13 at 7:02
add a comment |
Ohh sorry we have 4x2 as it is RAID10, attached a screenshot from iostat -xk 1 command i.imgur.com/7g5mfem.jpg
– George Youssry
Apr 13 '13 at 7:02
Ohh sorry we have 4x2 as it is RAID10, attached a screenshot from iostat -xk 1 command i.imgur.com/7g5mfem.jpg
– George Youssry
Apr 13 '13 at 7:02
Ohh sorry we have 4x2 as it is RAID10, attached a screenshot from iostat -xk 1 command i.imgur.com/7g5mfem.jpg
– George Youssry
Apr 13 '13 at 7:02
add a comment |
Thanks for contributing an answer to Server Fault!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f498951%2fhigh-wa-in-top-on-kvm-host%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Just two comments: a) for a RAID10 you need 2n HDDs, w/ n>=2, so whatever you're running, it's not a RAID10 w/ only two disks. b) I/O wait can occur w/ network devices as well as with local storage.
– tink
Apr 13 '13 at 6:19
for the type of cpu and memory on your machine i would look into having a 6 or 8 drive raid 10. to get a lot of perfomance if your gonna use raid 8 drives may wanna look into a raid 100
– WojonsTech
Apr 13 '13 at 8:21
That's not good to provide extra IO performance just because you have a lot of CPU and RAM, it's about application requirements, not ruthless 'cool server' competetion. Idea of RAID 100 is to spread data across multiple controllers using software raid0, so I doubt it's a common thing to use.
– DukeLion
Apr 17 '13 at 7:10