Inconsistent fio results on similar VPSes on the same node
我今天在同一个节点上，在类似规格的 VPS 上运行 yabs 时，发现 fio 测试结果存在一些不一致。请参见以下三个示例。
I am seeing some inconsistency on fio tests in yabs run today on similar spec VPSes on the same node. Please see three examples below.
我不清楚发生了什么，是否与我碰巧测试的任何单个 VPS 有关，是否与其他 VPS 或某些节点进程在特定时间使用高文件 I/O 有关，或者可能是其他原因。
I'm unclear on what's happening, whether it's related to any single VPS that I happen to be testing, whether it's related to some other VPS or some node process using high file I/O at certain times, or maybe something else.
我一直在观察 iotop -b 3 -o。目前为止，没有明显的发现。
I've been watching iotop -b 3 -o a little. So far, no obvious insight.
有想法吗？谢谢！
Ideas? Thanks!
fio Disk Speed Tests (Mixed R/W 50/50) (Partition /dev/vda1):
---------------------------------
Block Size | 4k (IOPS) | 64k (IOPS)
------ | --- ---- | ---- ----
Read | 193.28 MB/s (48.3k) | 1.78 GB/s (27.8k)
Write | 193.79 MB/s (48.4k) | 1.79 GB/s (28.0k)
Total | 387.07 MB/s (96.7k) | 3.57 GB/s (55.9k)
| |
Block Size | 512k (IOPS) | 1m (IOPS)
------ | --- ---- | ---- ----
Read | 2.12 GB/s (4.1k) | 2.18 GB/s (2.1k)
Write | 2.23 GB/s (4.3k) | 2.33 GB/s (2.2k)
Total | 4.35 GB/s (8.5k) | 4.51 GB/s (4.4k)
fio Disk Speed Tests (Mixed R/W 50/50) (Partition /dev/vda1):
---------------------------------
Block Size | 4k (IOPS) | 64k (IOPS)
------ | --- ---- | ---- ----
Read | 193.53 MB/s (48.3k) | 1.95 GB/s (30.4k)
Write | 194.04 MB/s (48.5k) | 1.96 GB/s (30.6k)
Total | 387.57 MB/s (96.8k) | 3.91 GB/s (61.1k)
| |
Block Size | 512k (IOPS) | 1m (IOPS)
------ | --- ---- | ---- ----
Read | 957.00 KB/s (1) | 18.49 MB/s (18)
Write | 1.12 MB/s (2) | 20.26 MB/s (19)
Total | 2.07 MB/s (3) | 38.75 MB/s (37)
fio Disk Speed Tests (Mixed R/W 50/50) (Partition /dev/vda1):
---------------------------------
Block Size | 4k (IOPS) | 64k (IOPS)
------ | --- ---- | ---- ----
Read | 16.51 MB/s (4.1k) | 1.84 GB/s (28.7k)
Write | 16.52 MB/s (4.1k) | 1.85 GB/s (28.9k)
Total | 33.03 MB/s (8.2k) | 3.69 GB/s (57.7k)
| |
Block Size | 512k (IOPS) | 1m (IOPS)
------ | --- ---- | ---- ----
Read | 2.11 GB/s (4.1k) | 2.19 GB/s (2.1k)
Write | 2.22 GB/s (4.3k) | 2.34 GB/s (2.2k)
Total | 4.34 GB/s (8.4k) | 4.53 GB/s (4.4k)
除非能提供虚拟化环境信息，否则无法判断...
但的确有一些特殊情况会造成这样的问题，比如vps被分配到了有坏道的磁盘里...
no idea
老哥好像是隔壁L论坛送免费鸡的那位
@virusinstant #3
你的想法太棒了！所有这些 VPS 都分配到同一组 8 块磁盘，并采用 RAID 10 模式。有一个硬件 RAID 控制器，它将 RAID 阵列的状态描述为“最佳”。
Your idea is excellent! All these VPSes are assigned to the same set of 8 disks running RAID 10. There is a hardware RAID controller, which describes the state of the RAID array as "Optimal."
谢谢你的绝妙想法！
Thanks for your ecellent idea!
Sure! Here's the explanation in English:
1. Hardware Resource Contention:
Reason: When multiple virtual machines (VMs) are running on the same physical node, they compete for CPU, memory, and network resources. This can lead to fluctuations in performance, especially during I/O-intensive operations like those tested with
fio.Solution:
Monitor system resource usage using tools like
htoportopto check if other processes are consuming CPU or memory.You can also use
vmstat,iostat, ordstatto track I/O activity and see if resource contention is happening.If possible, reduce the load on the VPS or allocate more resources (CPU, RAM) to improve consistency.
2. Virtualization Layer Impact:
Reason: The disk I/O performance can vary significantly depending on the virtualization technology used (e.g., KVM, OpenVZ, Docker, etc.). Virtualization adds an additional layer that abstracts hardware access, and not all virtualized environments handle disk I/O in the same way. As a result, disk performance can be slower in certain virtualized environments.
Solution:
Compare the performance across different virtualization technologies. For example, if you're running on KVM, test the same workload in an OpenVZ or Docker environment to see if the disk I/O performance is consistent.
If possible, test using a non-virtualized environment (bare-metal server) to see if the performance disparity persists.
Ensure that the VPS is using optimal disk I/O drivers and settings for the chosen virtualization platform.
3. File System Type:
Reason: Different file systems (e.g., ext4, XFS, ZFS) have different performance characteristics. Some file systems may be better optimized for certain types of operations, like sequential writes, while others may perform better with random reads or writes. If the two VPSs are using different file systems, this could explain the discrepancies in
fiotest results.Solution:
Check the file system types used on both VPSs using the
df -Torlsblk -fcommands.If they are different, consider standardizing the file system across all VPS instances, ideally using a high-performance file system like
ext4orXFS.If performance consistency is crucial, consider benchmarking the file systems individually and choosing the one best suited for your workload.
The sudden drop in performance during the second test might just be because the RAID card's RAM cache is full.
I don't know why. Hope you can solve it.