How fast are your disks? Find out the open source way, with fio

 

https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/

Storage benchmarking—much like Wi-Fi benchmarking—is a widely misunderstood black art. Admins and enthusiasts have for decades been tempted to just "get the big number" by reading or writing a large amount of data to a disk, getting a figure in MB/sec, and calling it a day. Unfortunately, the actual workload of a typical disk doesn't look like that—and that "simple speed test" doesn't reproduce a lot of the bottlenecks that slow down disk access in real-world systems.

The most realistic way to test and benchmark disks is, of course, to just use them and see what happens. Unfortunately, that's neither very repeatable, nor is it simple to analyze. So we do want an artificial benchmarking tool—but we want one that we can use intelligently to test storage systems across realistic scenarios that model our day-to-day usage well. Fortunately, we don't have to invent such a tool—there's already a free and open source software tool called fio, and it's even cross-platform!

 

We're going to walk you through some simple but effective uses of fio on Windows, Mac, and Linux computers—but before we do that, let's talk a little bit about storage systems from a more basic perspective.

Throughput, latency, IOPS and cache

Throughput

Throughput, measured most commonly in storage systems in MB/sec, is the most commonly used way to talk about storage performance. There are several choke points in a storage system for throughput—first and foremost, there's the speed of the physical medium itself. If you've got a single head on a conventional rust disk spinning at 7200RPM, the rate you can get data on or off that disk will be limited by the number of physical sectors/blocks passing beneath the head. You're also limited by the bandwidth of your controller and cabling—for example, modern SATA links typically operate at 6Gbps, while modern SAS links can operate up to 22.5Gbps.

Things get a little extra complicated here, because we're mixing units—notice the big B in MB/sec, and the small b in Gbps. That's the difference between bytes and bits. You divide Gbps by 8 to get GB/sec, then multiply by 1024 to get MB/sec. So a SATA-3 6Gbps link can theoretically move up to 768MB/sec. You can't actually move data across the SATA or SAS bus at the full theoretical link speed, but you can get fairly close. It's also worth noting that most SATA controllers won't move much more data than a single link can manage, even with many disks connected to the controller—so it's common to see even lots of very fast solid state drives in an array bottlenecking at around 700 MB/sec.

Latency

Latency is the flip side of the same performance coin. Where throughput refers to how many bytes of data per second you can move on or off the disk, latency—most commonly measured in milliseconds—refers to the amount of time it takes to read or write a single block. Most of the worst storage bottlenecks are latency issues that affect throughput, not the other way around.

In conventional spinning rust disks, there are two major sources of latency: rotational latency, and seek latency. The seek latency is how long it takes to move the mechanical arm the disk head is mounted on to the correct track on disk. Once the head has moved to the correct track, the drive then has to wait for the correct sector to rotate beneath the head—that's the rotational latency. The combination of seek and rotational latency usually adds up to somewhere between 15ms and 25ms.

You can see how latency affects throughput by thought experiment. If we have a reasonably fast spinning disk with a maximum throughput of 180MB/sec and a total access latency of 16ms, and we present it with a maximally fragmented workload—meaning that no two blocks have been written/are being written in sequential order—we can do a little math to come up with that throughput. Assuming 4KB physical blocks on disk, 4KB per seek divided by 0.016 seconds per seek = only 250KB/sec. Ouch!

IOPS

Short for Input/Output Operations Per Second, IOPS is the metric of measurement you'll most commonly hear real storage engineers discussing. It means exactly what it sounds like—how many different operations can a disk service? In much the same way, "throughput" usually refers to the maximal throughput of a disk, with very large and possibly sequential reads or writes, IOPS usually refers to the maximal number of operations a disk can service on the low end—4K random reads and writes.

Solid state disks don't suffer from seek or rotational latency, but 4K random Input/Output (I/O) does still present them with problems. Under the hood, a consumer SSD isn't really a single "disk"—it's a RAID array in a little sealed box, with its own complex controller inside the disk itself managing reads and writes. The SSD controller itself tries to stripe writes across multiple channels of physical flash media in parallel—and, if the user is lucky, the writes which got striped out evenly across those channels will also be read the same way, maximizing throughput.

When a solid state disk is presented with a 4K random I/O workload, if it can't figure out some way to aggregate and parallelize the requests, it will end up bottlenecking at much lower speeds, dictated by how quickly a single cell of flash media can read or write a block of data. The impact isn't as dramatic as it would be on a rust disk, but it's still significant—where a rust disk capable of 180MB/sec of throughput might plummet to 250KB/sec of 4K random I/O, a SSD capable of 500MB/sec could drop to around 40MB/sec.

Although you can discuss throughput in terms of 4K random I/O, and IOPS in terms of sequential 1MB I/O, that's not how each term is typically used. You should generally expect throughput to be discussed in terms of how much data a disk moves under optimal conditions, and IOPS in terms of the "low end grunt" the disk is capable of even under the worst workload. For typical desktop PC use, IOPS is far more important than throughput—because there's lots of that slow 4K random I/O, and it slows the whole system down when it happens.

Cache

As we've seen above, non-optimized workloads hurt performance, and hurt them badly. Thankfully for users, decades of research and development have presented us with all manner of tricks to keep from exploring the worst performance characteristics of our storage—especially rust storage. Operating systems use both read caches and write buffers to minimize the number of seeks necessary in operation and avoid the need to keep reading frequently needed blocks from storage over and over.

Write buffers allow the operating system to store up lots of small I/O requests and commit them to disk in large batches. One megabyte is a very small amount of data, but it still comes out to 256 4KB blocks—and if you must write each of those blocks out with individual operations, you might tie up your disk's entire service capacity for a full second. On the other hand, if you can aggregate those 256 blocks in a write buffer and then flush them out in a single operation, you avoid all that access latency, and the same amount of data can be saved in a hundredth of a second or less. This aggregation can also greatly help with read speeds later. If most of the same blocks need to be read as a group later, the drive can avoid seeking between them since they were all written as a group in the first place.

Read cache keeps the system from having to tie up storage with unnecessary repeated requests for the same blocks over and over again. If your operating system has plenty of RAM available, each time it reads data from disk, it keeps a copy of it lying around in memory. If another program asks for the same blocks later, the operating system can service that request directly from the cache—which keeps the drive's limited resources available for either read or write requests, which must hit the actual disk.

 

FURTHER READING

SSDs are on track to get bigger and cheaper thanks to PLC technology

Some models of SSD have an additional non-volatile write cache on the disk itself, made of a faster and more expensive type of flash media. For example, a TLC or QLC (Quad Layer Cell) SSD might have a few gigabytes of MLC (Multi-Layer Cell) media to use as a buffer for writes; this enables the SSD to keep up with the writes demanded by a typical desktop workload using the faster MLC buffer—but if presented with sustained heavy writes for too long a time, the fast MLC buffer fills, and throughput drops to what the slower TLC or QLC media can manage. This can frequently be a "fall off the cliff" type scenario, since the slower media will typically not only have to sustain ongoing writes, but do so while continuing to stream out the already-accepted writes from the fast MLC buffer.

Modeling storage access realistically

Now that we understand a little about the pain points in a storage system, it's pretty obvious that we shouldn't just use a simple tool like dd to read or write huge chunks of data—and generate huge numbers. Those huge numbers don't really correlate very well with how each disk performs under more realistic workloads—so, we want to generate more realistic access patterns to test with.

This is where fio comes in. Fio is short for Flexible Input/Output tester and can be configured to model nearly any storage workload under the sun. Real storage engineers—at least, the ones who are doing their jobs right—will first analyze the actual storage access patterns of a server or service, then write fio scripts to model those exact patterns. In this way, they can test a disk or array not only for its general performance, but its performance as very specifically applicable to their exact workload.

We're not going to be quite that specific here, but we will use fio to model and report on some key usage patterns common to desktop and server storage. The most important of these is 4K random I/O, which we discussed at length above. 4K random is where the pain lives—it's the reason your nice fast computer with a conventional hard drive suddenly sounds like it's grinding coffee and makes you want to defenestrate it in frustration.

Next, we look at 64K random I/O, in sixteen parallel processes. This is sort of a middle-of-the-road workload for a busy computer—there are a lot of requests for relatively small amounts of data, but there are also lots of parallel processes; on a modern system, that high number of parallel processes is good, because it potentially allows the OS to aggregate lots of small requests into a few larger requests. Although nowhere near as punishing as 4K random I/O, 64K random I/O is enough to significantly slow most storage systems down.

Finally, we look at high-end throughput—some of the biggest numbers you can expect to see out of the system—by way of 1MB random I/O. Technically, you could still get a slightly bigger number by asking fio to generate truly sequential requests—but in the real world, those are vanishingly rare. If your OS needs to write a couple of lines to a system log, or read a few KB of data from a system library, your "sequential" read or write immediately becomes, effectively, 1MB random I/O as it shares time with the other process.

Installing fio

Windows

You can find Windows installers for fio at https://bsdio.com/fio/. Note that you may get Smartscreen warnings when running one of these installers, since they are not digitally signed. These packages are provided by Rebecca Cran and are available without warranty.

Note that Windows has a limited selection of ioengines available, which will inform your selection of command line arguments later. For the most part, Windows users should use --ioengine=windowsaio (Asynchronous Input/Output) with their fio arguments.

Linux / FreeBSD

The instructions for users of Linux and BSD distributions are a little different from one to another, but fio is in nearly all main repositories—so it boils down to <package manager> install fio for the vast majority.

Debian or Ubuntu: sudo apt install fio

FreeBSD: sudo pkg install fio

CentOS (and Red Hat Enterprise Linux) have rather more limited main repositories than most distributions; if you haven't already, you'll need to add the EPEL repository to CentOS/RHEL to get fio.

CentOS/RHEL: sudo yum install epel-release -y ; sudo yum install fio

You get the idea.

MacOS

On a Mac, you'll want to install fio via brew. If you don't already have brew installed, at the Terminal, issue the following command:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

On the one hand, the above is abominable procedure; on the other hand, you can confirm that the script being pulled down tells you everything it's going to do, before it does it, and pauses to allow you to consent to it. If you're sufficiently paranoid, you may wish to download the file, inspect it, and then run it as separate steps instead. Note that the homebrew install script does not need sudo privileges—and will, in fact, refuse to run at all if you try to execute it with sudo.

With Brew installed, you can now install fio easily:

brew install fio

Using fio

Now you can use fio to benchmark storage. First, change directory to the location you actually want to test: if you run fio in your home directory, you'll be testing your computer's internal disk, and if you run it in a directory located on a USB portable disk, you'll be benchmarking that portable disk. Once you've got a command prompt somewhere in the disk you want to test, you're ready to actually run fio.

Baby's first fio run

First, we'll examine the syntax needed for a simple 4K random write test. (Windows users: substitute --ioengine=windowsaio for --ioengine=posixaio in both this and future commands.)

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --numjobs=1 --size=4g --iodepth=1 --runtime=60 --time_based --end_fsync=1

Let's break down what each argument does.

--name= is a required argument, but it's basically human-friendly fluff—fio will create files based on that name to test with, inside the working directory you're currently in.

--ioengine=posixaio sets the mode fio interacts with the filesystem. POSIX is a standard Windows, Macs, Linux, and BSD all understand, so it's great for portability—although inside fio itself, Windows users need to invoke --ioengine=windowsaio, not --ioengine=posixaio, unfortunately. AIO stands for Asynchronous Input Output and means that we can queue up multiple operations to be completed in whatever order the OS decides to complete them. (In this particular example, later arguments effectively nullify this.)

--rw=randwrite means exactly what it looks like it means: we're going to do random write operations to our test files in the current working directory. Other options include seqread, seqwrite, randread, and randrw, all of which should hopefully be fairly self-explanatory.

--bs=4k blocksize 4K. These are very small individual operations. This is where the pain lives; it's hard on the disk, and it also means a ton of extra overhead in the SATA, USB, SAS, SMB, or whatever other command channel lies between us and the disks, since a separate operation has to be commanded for each 4K of data.

--size=4g our test file(s) will be 4GB in size apiece. (We're only creating one, see next argument.)

--numjobs=1 we're only creating a single file, and running a single process commanding operations within that file. If we wanted to simulate multiple parallel processes, we'd do, eg, --numjobs=16, which would create 16 separate test files of --size size, and 16 separate processes operating on them at the same time.

--iodepth=1 this is how deep we're willing to try to stack commands in the OS's queue. Since we set this to 1, this is effectively pretty much the same thing as the sync IO engine—we're only asking for a single operation at a time, and the OS has to acknowledge receipt of every operation we ask for before we can ask for another. (It does not have to satisfy the request itself before we ask it to do more operations, it just has to acknowledge that we actually asked for it.)

--runtime=60 --time_based Run for sixty seconds—and even if we complete sooner, just start over again and keep going until 60 seconds is up.

--end_fsync=1 After all operations have been queued, keep the timer going until the OS reports that the very last one of them has been successfully completed—ie, actually written to disk.

Interpreting fio's output

This is the entire output from the 4K random I/O run on my Ubuntu workstation:

root@banshee:/tmp# fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --runtime=60 --time_based --end_fsync=1
random-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=posixaio
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][eta 00m:00s]                          
random-write: (groupid=0, jobs=1): err= 0: pid=16109: Wed Feb  5 15:09:36 2020
  write: IOPS=32.5k, BW=127MiB/s (133MB/s)(8192MiB/64602msec); 0 zone resets
    slat (nsec): min=250, max=555439, avg=1388.31, stdev=833.19
    clat (nsec): min=90, max=20251k, avg=9642.34, stdev=179381.02
     lat (usec): min=3, max=20252, avg=11.03, stdev=179.39
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    4], 10.00th=[    4], 20.00th=[    5],
     | 30.00th=[    6], 40.00th=[    6], 50.00th=[    7], 60.00th=[    8],
     | 70.00th=[    9], 80.00th=[   10], 90.00th=[   11], 95.00th=[   12],
     | 99.00th=[   17], 99.50th=[   20], 99.90th=[   43], 99.95th=[   77],
     | 99.99th=[12387]
   bw (  KiB/s): min=22256, max=613312, per=100.00%, avg=335527.28, stdev=162778.06, samples=50
   iops        : min= 5564, max=153328, avg=83881.88, stdev=40694.66, samples=50
  lat (nsec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=13.96%, 10=68.85%, 20=16.68%, 50=0.41%
  lat (usec)   : 100=0.04%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=6.35%, sys=11.96%, ctx=2348924, majf=0, minf=48
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2097153,0,1 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=127MiB/s (133MB/s), 127MiB/s-127MiB/s (133MB/s-133MB/s), io=8192MiB (8590MB), run=64602-64602msec

Disk stats (read/write):
    md0: ios=71/749877, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=351/737911, aggrmerge=0/12145, aggrticks=1875/260901, aggrin_queue=30910, aggrutil=83.73%
  sdb: ios=342/737392, merge=0/12663, ticks=1832/241034, in_queue=28672, util=83.35%
  sda: ios=361/738430, merge=0/11628, ticks=1918/280768, in_queue=33148, util=83.73%

This may seem like a lot. It is a lot! But there's only one piece you'll likely care about, in most cases—the line directly under "Run status group 0 (all jobs):" is the one with the aggregate throughput. Fio is capable of running as many wildly different jobs in parallel as you'd like to execute complex workload models. But since we're only running one job group, we've only got one line of aggregates to look through.

Run status group 0 (all jobs):
  WRITE: bw=127MiB/s (133MB/s), 127MiB/s-127MiB/s (133MB/s-133MB/s), io=8192MiB (8590MB), run=64602-64602msec

First, we're seeing output in both MiB/sec and MB/sec. MiB means "mebibytes"—measured in powers of two—where MB means "megabytes," measured in powers of ten. Mebibytes—1024x1024 bytes—are what operating systems and filesystems actually measure data in, so that's the reading you care about.

Run status group 0 (all jobs):
  WRITE: bw=127MiB/s (133MB/s), 127MiB/s-127MiB/s (133MB/s-133MB/s), io=8192MiB (8590MB), run=64602-64602msec

In addition to only having a single job group, we only have a single job in this test—we didn't ask fio to, for example, run sixteen parallel 4K random write processes—so although the second bit shows minimum and maximum range, in this case it's just a repeat of the overall aggregate. If we'd had multiple processes, we'd see the slowest process to the fastest process represented here.

Run status group 0 (all jobs):
  WRITE: bw=127MiB/s (133MB/s), 127MiB/s-127MiB/s (133MB/s-133MB/s), io=8192MiB (8590MB), run=64602-64602msec

Finally, we get the total I/O—8192MiB written to disk, in 64602 milliseconds. Divide 8192MiB by 64.602 seconds, and surprise surprise, you get 126.8MiB/sec—round that up to 127MiB/sec, and that's just what fio told you in the first block of the line for aggregate throughput.

If you're wondering why fio wrote 8192MiB instead of only 4096MiB in this run—despite our --size argument being 4g, and only having one process running—it's because we used --time_based and --runtime=60. And since we're testing on a fast storage medium, we managed to loop through the full write run twice before terminating.

You can cherry pick lots more interesting stats out of the full fio output, including utilization percentages, IOPS per process, and CPU utilization—but for our purposes, we're just going to stick with the aggregate throughput from here on out.

Ars recommended tests

Single 4KiB random write process

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1

This is a single process doing random 4K writes. This is where the pain really, really lives; it's basically the worst possible thing you can ask a disk to do. Where this happens most frequently in real life: copying home directories and dotfiles, manipulating email stuff, some database operations, source code trees.

When I ran this test against the high-performance SSDs in my Ubuntu workstation, they pushed 127MiB/sec. The server just beneath it in the rack only managed 33MiB/sec on its "high-performance" 7200RPM rust disks... but even then, the vast majority of that speed is because the data is being written asynchronously, allowing the operating system to batch it up into larger, more efficient write operations.

If we add the argument --fsync=1, forcing the operating system to perform synchronous writes (calling fsync after each block of data is written) the picture gets much more grim: 2.6MiB/sec on the high-performance SSDs but only 184KiB/sec on the "high-performance" rust. The SSDs were about four times faster than the rust when data was written asynchronously but a whopping fourteen times faster when reduced to the worst-case scenario.

16 parallel 64KiB random write processes

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=64k --size=256m --numjobs=16 --iodepth=16 --runtime=60 --time_based --end_fsync=1

This time, we're creating 16 separate 256MB files (still totaling 4GB, when all put together) and we're issuing 64KB blocksized random write operations. We're doing it with sixteen separate processes running in parallel, and we're queuing up to 16 simultaneous asynchronous ops before we pause and wait for the OS to start acknowledging their receipt.

This is a pretty decent approximation of a significantly busy system. It's not doing any one particularly nasty thing—like running a database engine or copying tons of dotfiles from a user's home directory—but it is coping with a bunch of applications doing moderately demanding stuff all at once.

This is also a pretty good, slightly pessimistic approximation of a busy, multi-user system like a NAS, which needs to handle multiple 1MB operations simultaneously for different users. If several people or processes are trying to read or write big files (photos, movies, whatever) at once, the OS tries to feed them all data simultaneously. This pretty quickly devolves down to a pattern of multiple random small block access. So in addition to "busy desktop with lots of apps," think "busy fileserver with several people actively using it."

You will see a lot more variation in speed as you watch this operation play out on the console. For example, the 4K single process test we tried first wrote a pretty consistent 11MiB/sec on my MacBook Air's internal drive—but this 16-process job fluctuated between about 10MiB/sec and 300MiB/sec during the run, finishing with an average of 126MiB/sec.

Most of the variation you're seeing here is due to the operating system and SSD firmware sometimes being able to aggregate multiple writes. When it manages to aggregate them helpfully, it can write them in a way that allows parallel writes to all the individual physical media stripes inside the SSD. Sometimes, it still ends up having to give up and write to only a single physical media stripe at a time—or a garbage collection or other maintenance operation at the SSD firmware level needs to run briefly in the background, slowing things down.

Single 1MiB random write process

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=1m --size=16g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1

This is pretty close to the best-case scenario for a real-world system doing real-world things. No, it's not quite as fast as a single, truly contiguous write... but the 1MiB blocksize is large enough that it's quite close. Besides, if literally any other disk activity is requested simultaneously with a contiguous write, the "contiguous" write devolves to this level of performance pretty much instantly, so this is a much more realistic test of the upper end of storage performance on a typical system.

You'll see some kooky fluctuations on SSDs when doing this test. This is largely due to the SSD's firmware having better luck or worse luck at any given time, when it's trying to queue operations so that it can write across all physical media stripes cleanly at once. Rust disks will tend to provide a much more consistent, though typically lower, throughput across the run.

You can also see SSD performance fall off a cliff here if you exhaust an onboard write cache—TLC and QLC drives tend to have small write cache areas made of much faster MLC or SLC media. Once those get exhausted, the disk has to drop to writing directly to the much slower TLC/QLC media where the data eventually lands. This is the major difference between, for example, Samsung EVO and Pro SSDs—the EVOs have slow TLC media with a fast MLC cache, where the Pros use the higher-performance, higher-longevity MLC media throughout the entire SSD.

If you have any doubt at all about a TLC or QLC disk's ability to sustain heavy writes, you may want to experimentally extend your time duration here. If you watch the throughput live as the job progresses, you'll see the impact immediately when you run out of cache—what had been a fairly steady, several-hundred-MiB/sec throughput will suddenly plummet to half the speed or less and get considerably less stable as well.

However, you might choose to take the opposite position—you might not expect to do sustained heavy writes very frequently, in which case you actually are more interested in the on-cache behavior. What's important here is that you understand both what you want to test, and how to test it accurately.

Conclusions

Using fio is definitely an exercise for the true nerd (or professional). It won't hold your hand, and although it provides incredibly detailed results, they're not automatically made into pretty graphs for you.

If all of this feels like far too much work, you can also find simpler-to-use graphical tools, such as HD Tune Pro for Windows. HD Tune Pro costs $35, or there's a limited-capability non-Pro version that is free for personal use. It's a good tool, and it'll make shiny graphs—but it's considerably more limited for advanced users, and the price of the make-it-easy user interface is that you're much further removed from the technical reality of what you're doing.

Learning to use fio means really learning the difference between asynchronous and synchronous writes and knowing for absolute certain what it's going to do at a very low level on an individual argument basis. You can't be as certain of what tools like HD Tune Pro are actually doing under the hood—and having to deal with different tools on different operating systems means it's more difficult to directly compare and contrast results as well.