Introduction
Sometimes we need to know the current status of the storage usage of a service in order to find any possible bottleneck in storage side. This tutorial describes how to measure current IOPS
and Throughput
on your server.
Max IOPS
and Throughput
of storage:
In this section, we want to estimate the maximum IOPS
and Throughput
of storage.
dd
command
dd if=/dev/zero of=/tmp/disk_test_dd.file bs=100M count=1 oflag=dsync
- By running this command,
dd
will read from/dev/zero
(a stream of null bytes) and write 100 megabytes of data to the file/tmp/disk_test_dd.file
. The write operation will be synchronized and physically written to the disk beforedd
exits, thanks to theoflag=dsync
option. This command is often used to test the disk write performance or to create files filled with null bytes for various purposes.
Result:
1+0 records in
1+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.22367 s, 469 MB/s
- As you can see
Throughput
of the disk was469 MB/s
fio
command (Recommended)
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test_disk_fio --filename=test_disk_fio --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
- By running this command, you will initiate a
fio
test that performs random read-write I/O operations with a block size of 4 kilobytes, using a 4-gigabyte test file/device. The test will use thelibaio
I/O engine with direct I/O enabled. The I/O depth is set to 64, and the reads-to-writes ratio is 75:25. The test results will provide performance metrics and insights into the disk's I/O capabilities under these conditions.
Result:
test_disk_fio: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=128MiB/s,w=41.0MiB/s][r=32.8k,w=10.7k IOPS][eta 00m:00s]
test_disk_fio: (groupid=0, jobs=1): err= 0: pid=68138: Mon May 15 10:52:23 2023
read: IOPS=37.1k, BW=145MiB/s (152MB/s)(3070MiB/21170msec)
bw ( KiB/s): min=105184, max=246976, per=100.00%, avg=148570.26, stdev=22072.40, samples=42
iops : min=26296, max=61744, avg=37142.55, stdev=5518.10, samples=42
write: IOPS=12.4k, BW=48.5MiB/s (50.8MB/s)(1026MiB/21170msec)
bw ( KiB/s): min=36384, max=82448, per=100.00%, avg=49650.48, stdev=7312.82, samples=42
iops : min= 9096, max=20612, avg=12412.60, stdev=1828.21, samples=42
cpu : usr=4.81%, sys=32.05%, ctx=72759, majf=0, minf=153
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwt: total=785920,262656,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=3070MiB (3219MB), run=21170-21170msec
WRITE: bw=48.5MiB/s (50.8MB/s), 48.5MiB/s-48.5MiB/s (50.8MB/s-50.8MB/s), io=1026MiB (1076MB), run=21170-21170msec
Disk stats (read/write):
sda: ios=785117/262419, merge=0/226, ticks=1027296/246228, in_queue=1273668, util=99.60%
As you see
fio
provides more detailed resultMean
IOPS
for read wasΒ37.1k
and for write was12.4k
Mean
Throughput
for read was152MB/s
and for write was50.8MB/s
Current IOPS
and Throughput
of storage:
In this section, we want to estimate current IOPS
and Throughput
of storage.
With iostat
iostat -xdmb 60 1440
- By running this command,
iostat
will start monitoring and displaying disk statistics for all disk partitions every 60 seconds. The statistics will include information about disk utilization, wait time, service time, and the timestamp when the statistics were collected. The monitoring will continue for approximately 1440 minutes (around 24 hours).
Note:
You can run iostat -xdmb 60 1440 >> iostat_res.txt
to redirect results to a file.
Result:
05/13/2023 08:22:19 PM
Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.60 0.00 0.00 0.00
sda 444.91 84.11 26980.23 2880.85 3.43 63.74 0.76 43.11 0.31 1.74 0.10 60.64 34.25 0.15 8.08
sda1 0.00 0.00 0.09 0.00 0.00 0.00 11.23 0.00 0.22 12.28 0.00 41.62 0.50 0.13 0.00
sda2 444.91 84.11 26980.14 2880.85 3.43 63.74 0.76 43.11 0.31 1.74 0.10 60.64 34.25 0.15 8.08
sdb 897.80 739.67 96125.22 35384.91 41.59 944.47 4.43 56.08 0.02 0.03 0.03 107.07 47.84 0.03 5.62
- You can see the detail of each column inΒ
iostat
Doc - At
05/13/2023 08:22:19 PM
,sda
has444.91
IOPS
for reading and84.11
for write. (Fromr/s
andw/s
columns) - As same,
sda
has26980.23 kB/s
Throughput
for reading and2880.85 kB/s
for write. (FromrkB/s
andwkB/s
columns)
Note:
This is a result of single 60 seconds. If you enter the command same as above, you will see 1440 tables like this after 1 day.
Analyze Output in jupyter notebook
[Optional]
In case you have some servers, and you need to analyze disk activity on them, You can use provided Jupyter Notebook to get more insight into the disks on your servers.
You can see and download the Jupyter Notebook from this link
Conclusion
With provided commands and corresponding results, you can estimate needs of current service and see whether there is a bottleneck regarding to I/O or not.
References
- Calculate
IOPS
in linux - Understanding I/O
- AlsoΒ ChatGPTΒ :)
Top comments (0)