NFS client intermittent performance issues (backlog queue wait?)Analyzing Linux NFS server performanceGood NFS configuration for Small Network (NFS Performance Issues)Issues mounting NFS share on clientNFS / DRBD / XFS Performance issuesmdadm raid10 or nfs performance issues?NFSv3 client differences in read/write performanceWindows NFS client keeps writing read-only with on NFS server with squash_allcpu load and nfs performanceHow to identify which NFS client generates disk IO?How to fix accumulating NFS DELEG locks necessitating a hard power-down of the client?
What exactly is ineptocracy?
Why is it a bad idea to hire a hitman to eliminate most corrupt politicians?
Was the Stack Exchange "Happy April Fools" page fitting with the '90's code?
In Bayesian inference, why are some terms dropped from the posterior predictive?
What is an equivalently powerful replacement spell for the Yuan-Ti's Suggestion spell?
Theorists sure want true answers to this!
What is required to make GPS signals available indoors?
Do creatures with a listed speed of "0 ft., fly 30 ft. (hover)" ever touch the ground?
Could the museum Saturn V's be refitted for one more flight?
Rotate ASCII Art by 45 Degrees
How can saying a song's name be a copyright violation?
What is a Samsaran Word™?
Unlock My Phone! February 2018
Are British MPs missing the point, with these 'Indicative Votes'?
Why was the shrink from 8″ made only to 5.25″ and not smaller (4″ or less)
How does a dynamic QR code work?
GFCI outlets - can they be repaired? Are they really needed at the end of a circuit?
Is this draw by repetition?
Send out email when Apex Queueable fails and test it
Can compressed videos be decoded back to their uncompresed original format?
Fair gambler's ruin problem intuition
How to install cross-compiler on Ubuntu 18.04?
How do conventional missiles fly?
If a warlock makes a Dancing Sword their pact weapon, is there a way to prevent it from disappearing if it's farther away for more than a minute?
NFS client intermittent performance issues (backlog queue wait?)
Analyzing Linux NFS server performanceGood NFS configuration for Small Network (NFS Performance Issues)Issues mounting NFS share on clientNFS / DRBD / XFS Performance issuesmdadm raid10 or nfs performance issues?NFSv3 client differences in read/write performanceWindows NFS client keeps writing read-only with on NFS server with squash_allcpu load and nfs performanceHow to identify which NFS client generates disk IO?How to fix accumulating NFS DELEG locks necessitating a hard power-down of the client?
I have a weird issue with one of the NFS clients (Ubuntu 16.04 LTS).
I have been working for the past few days to try and debug this, but so far without success.
After I mount the partition, everything works perfectly for a few days, transfer speed is 1 Gbps between the client and server. After a few days the speed drops to less than 10 mbps and even a simple directory listing takes a few seconds and I/O wait is at 100%
What I noticed is that the backlog wait, especially for the write operations is extremely high:
root@srv:~# mountstats /mnt/data
Stats for 192.168.0.15:/mnt/data mounted on /mnt/data:
NFS mount options: rw,vers=4.0,rsize=16384,wsize=16384,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.150,local_lock=none
NFS server capabilities: caps=0xffdf,wtmult=512,dtsize=16384,bsize=0,namlen=255
NFSv4 capability flags: bm0=0xfdffbfff,bm1=0xf9be3e,bm2=0x0,acl=0x3,pnfs=notconfigured
NFS security flavor: 1 pseudoflavor: 0
NFS byte counts:
applications read 8168679407142 bytes via read(2)
applications wrote 4833000353435 bytes via write(2)
applications read 0 bytes via O_DIRECT read(2)
applications wrote 0 bytes via O_DIRECT write(2)
client read 4218977852758 bytes via NFS READ
client wrote 4832098253207 bytes via NFS WRITE
RPC statistics:
561421762 RPC requests sent, 561421608 RPC replies received (1 XIDs not found)
average backlog queue length: 0
READ:
263822474 ops (46%) 0 retrans (0%) 0 major timeouts
avg bytes sent per op: 184 avg bytes received per op: 16051
backlog wait: 8.772689 RTT: 27.972131 total execute time: 36.752241 (milliseconds)
WRITE:
295296111 ops (52%) 0 retrans (0%) 0 major timeouts
avg bytes sent per op: 16567 avg bytes received per op: 132
backlog wait: 62468603019.791718 RTT: 78.030143 total execute time: 62468603097.830574 (milliseconds)
There are no errors, no warnings, I tried debugging with "echo 1 > /proc/sys/vm/block_dump" (which has done miracles for me in the past), but this time there's nothing visible related to NFS.
Any idea how I can debug further and see what's causing the extremely high backlog wait?
linux nfs nfs4
asked Mar 15 at 7:43
TonyTony
86211
add a comment |
I have a weird issue with one of the NFS clients (Ubuntu 16.04 LTS).
I have been working for the past few days to try and debug this, but so far without success.
After I mount the partition, everything works perfectly for a few days, transfer speed is 1 Gbps between the client and server. After a few days the speed drops to less than 10 mbps and even a simple directory listing takes a few seconds and I/O wait is at 100%
What I noticed is that the backlog wait, especially for the write operations is extremely high:
root@srv:~# mountstats /mnt/data
Stats for 192.168.0.15:/mnt/data mounted on /mnt/data:
NFS mount options: rw,vers=4.0,rsize=16384,wsize=16384,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.150,local_lock=none
NFS server capabilities: caps=0xffdf,wtmult=512,dtsize=16384,bsize=0,namlen=255
NFSv4 capability flags: bm0=0xfdffbfff,bm1=0xf9be3e,bm2=0x0,acl=0x3,pnfs=notconfigured
NFS security flavor: 1 pseudoflavor: 0
NFS byte counts:
applications read 8168679407142 bytes via read(2)
applications wrote 4833000353435 bytes via write(2)
applications read 0 bytes via O_DIRECT read(2)
applications wrote 0 bytes via O_DIRECT write(2)
client read 4218977852758 bytes via NFS READ
client wrote 4832098253207 bytes via NFS WRITE
RPC statistics:
561421762 RPC requests sent, 561421608 RPC replies received (1 XIDs not found)
average backlog queue length: 0
READ:
263822474 ops (46%) 0 retrans (0%) 0 major timeouts
avg bytes sent per op: 184 avg bytes received per op: 16051
backlog wait: 8.772689 RTT: 27.972131 total execute time: 36.752241 (milliseconds)
WRITE:
295296111 ops (52%) 0 retrans (0%) 0 major timeouts
avg bytes sent per op: 16567 avg bytes received per op: 132
backlog wait: 62468603019.791718 RTT: 78.030143 total execute time: 62468603097.830574 (milliseconds)
There are no errors, no warnings, I tried debugging with "echo 1 > /proc/sys/vm/block_dump" (which has done miracles for me in the past), but this time there's nothing visible related to NFS.
Any idea how I can debug further and see what's causing the extremely high backlog wait?
linux nfs nfs4
asked Mar 15 at 7:43
TonyTony
86211
add a comment |
I have a weird issue with one of the NFS clients (Ubuntu 16.04 LTS).
I have been working for the past few days to try and debug this, but so far without success.
After I mount the partition, everything works perfectly for a few days, transfer speed is 1 Gbps between the client and server. After a few days the speed drops to less than 10 mbps and even a simple directory listing takes a few seconds and I/O wait is at 100%
What I noticed is that the backlog wait, especially for the write operations is extremely high:
root@srv:~# mountstats /mnt/data
Stats for 192.168.0.15:/mnt/data mounted on /mnt/data:
NFS mount options: rw,vers=4.0,rsize=16384,wsize=16384,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.150,local_lock=none
NFS server capabilities: caps=0xffdf,wtmult=512,dtsize=16384,bsize=0,namlen=255
NFSv4 capability flags: bm0=0xfdffbfff,bm1=0xf9be3e,bm2=0x0,acl=0x3,pnfs=notconfigured
NFS security flavor: 1 pseudoflavor: 0
NFS byte counts:
applications read 8168679407142 bytes via read(2)
applications wrote 4833000353435 bytes via write(2)
applications read 0 bytes via O_DIRECT read(2)
applications wrote 0 bytes via O_DIRECT write(2)
client read 4218977852758 bytes via NFS READ
client wrote 4832098253207 bytes via NFS WRITE
RPC statistics:
561421762 RPC requests sent, 561421608 RPC replies received (1 XIDs not found)
average backlog queue length: 0
READ:
263822474 ops (46%) 0 retrans (0%) 0 major timeouts
avg bytes sent per op: 184 avg bytes received per op: 16051
backlog wait: 8.772689 RTT: 27.972131 total execute time: 36.752241 (milliseconds)
WRITE:
295296111 ops (52%) 0 retrans (0%) 0 major timeouts
avg bytes sent per op: 16567 avg bytes received per op: 132
backlog wait: 62468603019.791718 RTT: 78.030143 total execute time: 62468603097.830574 (milliseconds)
There are no errors, no warnings, I tried debugging with "echo 1 > /proc/sys/vm/block_dump" (which has done miracles for me in the past), but this time there's nothing visible related to NFS.
Any idea how I can debug further and see what's causing the extremely high backlog wait?
linux nfs nfs4
asked Mar 15 at 7:43
TonyTony
86211
I have a weird issue with one of the NFS clients (Ubuntu 16.04 LTS).
I have been working for the past few days to try and debug this, but so far without success.
After I mount the partition, everything works perfectly for a few days, transfer speed is 1 Gbps between the client and server. After a few days the speed drops to less than 10 mbps and even a simple directory listing takes a few seconds and I/O wait is at 100%
What I noticed is that the backlog wait, especially for the write operations is extremely high:
root@srv:~# mountstats /mnt/data
Stats for 192.168.0.15:/mnt/data mounted on /mnt/data:
NFS mount options: rw,vers=4.0,rsize=16384,wsize=16384,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.150,local_lock=none
NFS server capabilities: caps=0xffdf,wtmult=512,dtsize=16384,bsize=0,namlen=255
NFSv4 capability flags: bm0=0xfdffbfff,bm1=0xf9be3e,bm2=0x0,acl=0x3,pnfs=notconfigured
NFS security flavor: 1 pseudoflavor: 0
NFS byte counts:
applications read 8168679407142 bytes via read(2)
applications wrote 4833000353435 bytes via write(2)
applications read 0 bytes via O_DIRECT read(2)
applications wrote 0 bytes via O_DIRECT write(2)
client read 4218977852758 bytes via NFS READ
client wrote 4832098253207 bytes via NFS WRITE
RPC statistics:
561421762 RPC requests sent, 561421608 RPC replies received (1 XIDs not found)
average backlog queue length: 0
READ:
263822474 ops (46%) 0 retrans (0%) 0 major timeouts
avg bytes sent per op: 184 avg bytes received per op: 16051
backlog wait: 8.772689 RTT: 27.972131 total execute time: 36.752241 (milliseconds)
WRITE:
295296111 ops (52%) 0 retrans (0%) 0 major timeouts
avg bytes sent per op: 16567 avg bytes received per op: 132
backlog wait: 62468603019.791718 RTT: 78.030143 total execute time: 62468603097.830574 (milliseconds)
There are no errors, no warnings, I tried debugging with "echo 1 > /proc/sys/vm/block_dump" (which has done miracles for me in the past), but this time there's nothing visible related to NFS.
Any idea how I can debug further and see what's causing the extremely high backlog wait?
linux nfs nfs4
linux nfs nfs4
asked Mar 15 at 7:43
TonyTony
86211
asked Mar 15 at 7:43
TonyTony
86211
asked Mar 15 at 7:43
TonyTony
86211
asked Mar 15 at 7:43
TonyTony
86211
asked Mar 15 at 7:43
TonyTony
86211
86211
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Just in case anyone has the same problem, the only fix I could find was to force NFS3 instead NFS4. The problem has disappeared now.
answered yesterday
TonyTony
86211
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "2"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f958398%2fnfs-client-intermittent-performance-issues-backlog-queue-wait%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Just in case anyone has the same problem, the only fix I could find was to force NFS3 instead NFS4. The problem has disappeared now.
answered yesterday
TonyTony
86211
add a comment |
Just in case anyone has the same problem, the only fix I could find was to force NFS3 instead NFS4. The problem has disappeared now.
answered yesterday
TonyTony
86211
add a comment |
Just in case anyone has the same problem, the only fix I could find was to force NFS3 instead NFS4. The problem has disappeared now.
answered yesterday
TonyTony
86211
Just in case anyone has the same problem, the only fix I could find was to force NFS3 instead NFS4. The problem has disappeared now.
answered yesterday
TonyTony
86211
answered yesterday
TonyTony
86211
answered yesterday
TonyTony
86211
answered yesterday
TonyTony
86211
86211
add a comment |
add a comment |
Thanks for contributing an answer to Server Fault!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f958398%2fnfs-client-intermittent-performance-issues-backlog-queue-wait%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
