AN!Cluster Tutorial 2 - Disk Performance Tuning: Difference between revisions
Line 53: | Line 53: | ||
{| class="wikitable sortable" | {| class="wikitable sortable" | ||
|- | |- | ||
! Write Speed | ! Write Speed | ||
! Strip Size | ! Strip Size | ||
! Disk Cache | ! Disk Cache | ||
! IO Policy | ! IO Policy | ||
Line 60: | Line 60: | ||
! Raw output | ! Raw output | ||
|- | |- | ||
|style="text-align:center; vertical-align:middle;" class="code"| | |style="text-align:center; vertical-align:middle;" class="code"| 1136.36 MiB/s | ||
|style="text-align:center; vertical-align:middle;"| 64 | |style="text-align:center; vertical-align:middle;"| 64 KiB | ||
|style="text-align:center; vertical-align:middle;"| Disabled | |style="text-align:center; vertical-align:middle;"| Disabled | ||
|style="text-align:center; vertical-align:middle;"| Direct | |style="text-align:center; vertical-align:middle;"| Direct | ||
|style="text-align:center; vertical-align:middle;"| Base-line | |style="text-align:center; vertical-align:middle;"| Base-line | ||
|<div class="toccolours mw-collapsible mw-collapsed"> | |<div class="toccolours mw-collapsible mw-collapsed"> | ||
<div class="mw-collapsible-content"><syntaxhighlight lang=" | <div class="mw-collapsible-content"><syntaxhighlight lang="text"> | ||
[root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60 | |||
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop. | Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop. | ||
Virtual disk properties: | Virtual disk properties: | ||
Line 74: | Line 75: | ||
- Encryption: ....... [FDE] | - Encryption: ....... [FDE] | ||
- Write Cache: ...... [WriteBack] | - Write Cache: ...... [WriteBack] | ||
- Start Time: ....... [2015-12-08, | - Start Time: ....... [2015-12-08, 15:58:42] | ||
- Estimated End Time: [2015-12-08, | - Estimated End Time: [2015-12-08, 16:03:42] | ||
Please wait, the first loop has started... | Please wait, the first loop has started... | ||
Pass: [1], speed: [ | Pass: [1], speed: [1149.50 MiB/sec]. | ||
Pass: [2], speed: [1133.90 MiB/sec]. | |||
Pass: [3], speed: [1158.80 MiB/sec]. | |||
Pass: [4], speed: [1141.30 MiB/sec]. | |||
Pass: [5], speed: [1153.60 MiB/sec]. | |||
Average write speed: [1147.42 MiB/sec]. | |||
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60 | |||
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop. | |||
Virtual disk properties: | |||
- Strip Size: ....... [64 KiB] | |||
- Disk Cache: ....... [Disabled] | |||
- Active Operations: [None] | |||
- Encryption: ....... [FDE] | |||
- Write Cache: ...... [WriteBack] | |||
- Start Time: ....... [2015-12-08, 15:58:43] | |||
- Estimated End Time: [2015-12-08, 16:03:43] | |||
Please wait, the first loop has started... | |||
Pass: [1], speed: [1119.20 MiB/sec]. | |||
Pass: [2], speed: [1130.20 MiB/sec]. | Pass: [2], speed: [1130.20 MiB/sec]. | ||
Pass: [3], speed: [ | Pass: [3], speed: [1156.70 MiB/sec]. | ||
Pass: [4], speed: [ | Pass: [4], speed: [1126.40 MiB/sec]. | ||
Pass: [5], speed: [ | Pass: [5], speed: [1149.30 MiB/sec]. | ||
Average write speed: [ | Average write speed: [1136.36 MiB/sec]. | ||
</syntaxhighlight></div></div> | </syntaxhighlight></div></div> | ||
|- | |- | ||
|style="text-align:center; vertical-align:middle;" class="code"| | |style="text-align:center; vertical-align:middle;" class="code"| 1111.44 MiB/s | ||
|style="text-align:center; vertical-align:middle;"| 128 | |style="text-align:center; vertical-align:middle;"| 128 KiB | ||
|style="text-align:center; vertical-align:middle;"| Disabled | |style="text-align:center; vertical-align:middle;"| Disabled | ||
|style="text-align:center; vertical-align:middle;"| Direct | |style="text-align:center; vertical-align:middle;"| Direct | ||
|style="text-align:center; vertical-align:middle;"| No appreciable difference over 64 KiB | |style="text-align:center; vertical-align:middle;"| No appreciable difference over 64 KiB | ||
|<div class="toccolours mw-collapsible mw-collapsed"> | |<div class="toccolours mw-collapsible mw-collapsed"> | ||
<div class="mw-collapsible-content"><syntaxhighlight lang=" | <div class="mw-collapsible-content"><syntaxhighlight lang="text"> | ||
[root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60 | |||
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop. | |||
Virtual disk properties: | |||
- Strip Size: ....... [128 KiB] | |||
- Disk Cache: ....... [Disabled] | |||
- Active Operations: [None] | |||
- Encryption: ....... [FDE] | |||
- Write Cache: ...... [WriteBack] | |||
- Start Time: ....... [2015-12-08, 16:54:15] | |||
- Estimated End Time: [2015-12-08, 16:59:15] | |||
Please wait, the first loop has started... | |||
Pass: [1], speed: [1153.50 MiB/sec]. | |||
Pass: [2], speed: [1116.40 MiB/sec]. | |||
Pass: [3], speed: [1092.20 MiB/sec]. | |||
Pass: [4], speed: [1115.70 MiB/sec]. | |||
Pass: [5], speed: [1104.20 MiB/sec]. | |||
Average write speed: [1116.40 MiB/sec]. | |||
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60 | |||
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop. | |||
Virtual disk properties: | |||
- Strip Size: ....... [128 KiB] | |||
- Disk Cache: ....... [Disabled] | |||
- Active Operations: [None] | |||
- Encryption: ....... [FDE] | |||
- Write Cache: ...... [WriteBack] | |||
- Start Time: ....... [2015-12-08, 16:54:16] | |||
- Estimated End Time: [2015-12-08, 16:59:16] | |||
Please wait, the first loop has started... | |||
Pass: [1], speed: [1130.60 MiB/sec]. | |||
Pass: [2], speed: [1117.10 MiB/sec]. | |||
Pass: [3], speed: [1091.20 MiB/sec]. | |||
Pass: [4], speed: [1108.90 MiB/sec]. | |||
Pass: [5], speed: [1109.40 MiB/sec]. | |||
Average write speed: [1111.44 MiB/sec]. | |||
</syntaxhighlight></div></div> | </syntaxhighlight></div></div> | ||
|- | |- | ||
|style="text-align:center; vertical-align:middle;" class="code"| | |style="text-align:center; vertical-align:middle;" class="code"| | ||
|style="text-align:center; vertical-align:middle;"| 256 | |style="text-align:center; vertical-align:middle;"| 256 KiB | ||
|style="text-align:center; vertical-align:middle;"| Disabled | |style="text-align:center; vertical-align:middle;"| Disabled | ||
|style="text-align:center; vertical-align:middle;"| Direct | |style="text-align:center; vertical-align:middle;"| Direct | ||
|style="text-align:center; vertical-align:middle;"| Notable improvement, also default value | |style="text-align:center; vertical-align:middle;"| Notable improvement, also default value | ||
|<div class="toccolours mw-collapsible mw-collapsed"> | |<div class="toccolours mw-collapsible mw-collapsed"> | ||
<div class="mw-collapsible-content"><syntaxhighlight lang=" | <div class="mw-collapsible-content"><syntaxhighlight lang="text"> | ||
</syntaxhighlight></div></div> | </syntaxhighlight></div></div> | ||
Line 121: | Line 164: | ||
|style="text-align:center; vertical-align:middle;"| | |style="text-align:center; vertical-align:middle;"| | ||
|<div class="toccolours mw-collapsible mw-collapsed"> | |<div class="toccolours mw-collapsible mw-collapsed"> | ||
<div class="mw-collapsible-content"><syntaxhighlight lang=" | <div class="mw-collapsible-content"><syntaxhighlight lang="text"> | ||
</syntaxhighlight></div></div> | </syntaxhighlight></div></div> |
Revision as of 17:07, 8 December 2015
AN!Wiki :: How To :: AN!Cluster Tutorial 2 - Disk Performance Tuning |
The goal of this tutorial is to show the process of tuning an Anvil! to maximize storage performance for a given workload.
Warning: This is NOT complete and is largely a dumping ground of raw data. It's probably useless to most people at this time. |
The Workload
Our client specified the following 'fio' test as the simulated workload to optimize:
PATH="/data0/test"
fio --name certification --filename $PATH --ioengine libaio --direct 1 --bs 4M --rw randwrite --size 2g --iodepth 64 --numjobs 4 --runtime 30 --time_based --group_reporting
The '$PATH' will, eventually, be an xfs partition in a server hosted on an Anvil!.
The Setup
Each node is built on two Fujitsu RX2540 M1 nodes. The storage configuration is;
- 24x 1.8 TB 10krpm self-encrypting SAS drives.
- RAID level 6 using 22 drives in the array and 2 hot-spares.
- AVAGO MegaRAID 9361-8i Controller with 1 GiB of FBWC and SafeStore drive encryption management.
- DRBD v8.4.6.
- 4 KiB Native sector size drives using UEFI boot.
This configuration is designed for maximum availability and security. Performance is critical, but secondary to these priorities. As such, this tutorial will likely fall short of the maximum performance possible.
The Optimization Process
Tuning will be done in stages;
- Optimize local storage performance.
- Optimize network performance.
- Optimize DRBD performance (testing on the raw /dev/drbd0 device).
- Optimize clustered LVM performance.
- Optimize raw virtio block device performance.
- Optimize xfs performance inside the server.
Optimize
Local Storage
Local storage optimization requires tuning the RAID controller and virtual disk properties. Write-back caching is used and cached data is protected by flash-backed storage on the controller.
These tests are performed with no background operating in progress, optimal array and full disk encryption enabled.
Recorded test result is from 5 loops, each running for 60 seconds.
Call: /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Write Speed | Strip Size | Disk Cache | IO Policy | Note | Raw output |
---|---|---|---|---|---|
1136.36 MiB/s | 64 KiB | Disabled | Direct | Base-line | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [64 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Start Time: ....... [2015-12-08, 15:58:42]
- Estimated End Time: [2015-12-08, 16:03:42]
Please wait, the first loop has started...
Pass: [1], speed: [1149.50 MiB/sec].
Pass: [2], speed: [1133.90 MiB/sec].
Pass: [3], speed: [1158.80 MiB/sec].
Pass: [4], speed: [1141.30 MiB/sec].
Pass: [5], speed: [1153.60 MiB/sec].
Average write speed: [1147.42 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [64 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Start Time: ....... [2015-12-08, 15:58:43]
- Estimated End Time: [2015-12-08, 16:03:43]
Please wait, the first loop has started...
Pass: [1], speed: [1119.20 MiB/sec].
Pass: [2], speed: [1130.20 MiB/sec].
Pass: [3], speed: [1156.70 MiB/sec].
Pass: [4], speed: [1126.40 MiB/sec].
Pass: [5], speed: [1149.30 MiB/sec].
Average write speed: [1136.36 MiB/sec].
|
1111.44 MiB/s | 128 KiB | Disabled | Direct | No appreciable difference over 64 KiB | [root@node1 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [128 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Start Time: ....... [2015-12-08, 16:54:15]
- Estimated End Time: [2015-12-08, 16:59:15]
Please wait, the first loop has started...
Pass: [1], speed: [1153.50 MiB/sec].
Pass: [2], speed: [1116.40 MiB/sec].
Pass: [3], speed: [1092.20 MiB/sec].
Pass: [4], speed: [1115.70 MiB/sec].
Pass: [5], speed: [1104.20 MiB/sec].
Average write speed: [1116.40 MiB/sec].
[root@node2 ~]# /sbin/dashboard/hap-fio-tester --location /dev/sda5 --loops 5 --runtime 60
Testing: [/dev/sda5], averaging over: [5] loop(s), running for: [60] seconds per loop.
Virtual disk properties:
- Strip Size: ....... [128 KiB]
- Disk Cache: ....... [Disabled]
- Active Operations: [None]
- Encryption: ....... [FDE]
- Write Cache: ...... [WriteBack]
- Start Time: ....... [2015-12-08, 16:54:16]
- Estimated End Time: [2015-12-08, 16:59:16]
Please wait, the first loop has started...
Pass: [1], speed: [1130.60 MiB/sec].
Pass: [2], speed: [1117.10 MiB/sec].
Pass: [3], speed: [1091.20 MiB/sec].
Pass: [4], speed: [1108.90 MiB/sec].
Pass: [5], speed: [1109.40 MiB/sec].
Average write speed: [1111.44 MiB/sec].
|
256 KiB | Disabled | Direct | Notable improvement, also default value | ||
Any questions, feedback, advice, complaints or meanderings are welcome. | |||
Alteeve's Niche! | Enterprise Support: Alteeve Support |
Community Support | |
© Alteeve's Niche! Inc. 1997-2024 | Anvil! "Intelligent Availability®" Platform | ||
legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions. |