Otdfbt

LSI MegaRAID 9260-8i controller is limited to max 200 MB/s transfer rates.
The server is a HP DL180 G6, with CentOS 7 (64 bits) and we are testing 4TB SAS drives (Model: WD4001FYYG).
The controller is using iBBU08 (512 cache).
We have tested enabling/disabling cache and direct I/O, but it doesn't solve the problem.

According to our tests, writing concurrently to two different virtual disks (a RAID10 drive of 6 disks and a RAID0 drive of a single disk) we get max. 200 MB/s when reading and max. 200 MB/s when writing.

We verified that the performance decreases when operating concurrently on a different drive because the bandwith (aprox. 200 MB/s) is shared among different independent operating disk drives operations (bottleneck).

Conclusion:

The LSI controller is limiting the bandwidth to max 200 MB/s.

Why is this happening?
How can we fix it?
May it be related with the PCI card?
Can we measure the transfer rate?

PS: Issue was filed in support ticket SR # P00117431, but we stopped getting answers from AVAGOTECH (ex- LSI) after sending them detailed info.

Thanks

This are our IO tests:

--- 1) Single drive IO test ---

Write test:

# sync
# echo 3 > /proc/sys/vm/drop_caches
# dd if=/dev/zero of=/tmp/test bs=8k count=1M conv=fsync

1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 46.7041 s, 184 MB/s

Read test:

# sync
# echo 3 > /proc/sys/vm/drop_caches
# dd if=/dev/zero of=/tmp/test bs=8k count=1M conv=fsync

1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 47.1691 s, 182 MB/s

--- 2) Two drives concurrent IO tests ---

We will repeat the previous test, but running the same IO operations on a second independent drive at the same time.
As a result, the same drive now only performs 50%, which proves that the IO's on the second drive (/mnt/sdb/test) are sharing some limited resources on the LSI controller.

Write test:

Process 1:

[root@hp ~]# sync
[root@hp ~]# echo 3 > /proc/sys/vm/drop_caches
[root@hp ~]# dd if=/dev/zero of=/tmp/test bs=8k count=1M conv=fsync
1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 87.8613 s, 97.8 MB/s

Process 2:

[root@hp ~]# dd if=/dev/zero of=/mnt/sdb/test bs=8k count=1M conv=fsync
1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 86.3504 s, 99.5 MB/s

Read test:

Process 1:

[root@hp ~]# dd if=/tmp/test of=/dev/null bs=8k count=1M
1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 81.5574 s, 105 MB/s

Process 2:

[root@hp ~]# dd if=/mnt/sdb/test of=/dev/null bs=8k count=1M
1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 84.2258 s, 102 MB/s

This can easily happen because of the 4k sectors on the drive are 512e (you didn't specify the drive model, so it's a bit of the wild guess, but considering it's the 4Tb drive, I'd say it's Advanced Format). So I would check if your OS is aware about the size of your sector, unless you want read-modify-write cycling. This means proper partition alignment and the block size of the filesystems you are using.

And yeah, there's no such thing as HP DL 180 G7, gen6 was the last, then the model index changed from 180.

Just in case, there's pretty decent article for you (yeah, you are using CentOS, but... you know, it's basically the same stuff when it comes to internals).

Another thing you should probaby check and enable - it's the controller write cache, if you have a BBU, of course.

well there's really not enough info here yet.

First, what are the drives? Model and SAS version supported?

Second, are you writing to the array from another drive or array? Or writing and reading to and from the same array? If you're keeping it all in teh same array then you're splitting the available IO for the drives themselves in half(at best) regardless of the fact that SAS is full duplex if the data is distributed across the same disks and you're reading/writing each disk its own limitation on what it can handle from a disk operations standpoint...

Also, if you're reading or writing back and forth from the one Raid0 to the RAID10... then you're bottleneck is the single drive. You'll only ever get the max speed that one drive can handle. Which by the way 200MB(1.6 Gbs roughly) isn't bad for a single HDD.

Maybe you are hitting the max IOPS delivered by your card: with 8K packets, writing/reading at 200 MB/s means a sunstained rate of about 25K IOPS.

Can you re-try with larger packets (ie: use bs=1M or similar)? Does it change anything?