I answered in other thread how exactly integrity check looks like and why the name might be misleading. I'm also in the middle of updating descriptions to be more adequate. Thanks for pointing that out.
Speaking about standard industry metrics.
fio is an established tool, and throughput, IOPS, fsync latency, and percentiles are standard concepts. However, the exact job recipes and the composite “Overall Core” score are project-specific.
The trivial-operation test is also custom: one 4 KiB write plus fsync every 200 ms. The idle window contains at most about 50 operations, so its p99 is effectively the slowest sample’s fio histogram bucket, not a statistically stable population percentile. It should be treated as a small-write durability-latency probe, not a universal application metric.
But after all all tests are in the repository. If they need tweaks, changes I'm open to do so... I started from scratch and did whatever came to my mind. Some tests are added after my initial link here which went mostly unnoticed several weeks ago but I got some requests for more tests which I implemented.
But to sum up. I want this test to be useful so feel free to open PRs with improvements. It's not like I've got some agenda. In fact I wrote here and there on the page that I'm counting on communities of various filesystems to provide improvements, changes etc to make their filesystem shining.
This is personal project made when I realized that multiple-devices benchmarks were almost completely absent. Since I had not access to real hardware I decided to make at least initially everything based on GH runner with all the limitations that came with this approach. I tried to limit these limitations as far as I could. But feel free to submit bugreports, PRs, propositions for improvements.
The current “Integrity” label is broader than the test actually proves, and I’m changing it to “Corruption probe.”
FAIL means that file changed or became unreadable. SURVIVED means only that the file remained readable and hash-identical. It does not prove the entire filesystem was healthy, that every overwritten byte was allocated, or that all 2 GiB were repaired.
For the main linked benchmark there are no dedicated test hypervisors. It's all based on GH runners with all the limitations and quirks that came with it.
Obviously they could and probably were noisy neighbours. I'm not trying to hide that fact.
Initial step for every benchmark is test of underlying device to at least reject completely unlucky cases.
Also after almost 600 runs average is probably more or less correct...
Also take a look at tests on real hardware. There are not many of them but there are some. I pointed to them in my first answer.
I'd actually recommend some excellent books on the "philosophy" of data presentation: The first that comes to mind is "the visual display of quantitative information" by Edward Tufte seems to be freely available online, and the other one on my mind is "how charts lie" by Alberto Cairo (which doesn't seem to be freely accessible)
But if it helps, just some "initial gut feel observations" from me:
* It's definitely not possible to find issue with the the _sheer amount_ of results, but there's just far too much for a human to absorb, all presented at once
* Overall text size is quite small, and difficult to read
* The page doesn't make a strong statement of _what_ is under test: the first words are: "modern-fs-benchmark Multi-device CoW filesystems under workloads classic benchmarks skip" -- which defines the webpage in terms of what it is _not_, without stating what benchmarks are actually present.
* The first line of teh page contains run statistics that probably eithre want to b at the bottom, or just don't need to be in the webpage at all: "latest run 2026-09-18 18:50:45 UTC, kernel 7.0.0-1012-azure, 593 runs recorded · 145 trend points shown"
* A significant proportion of the free text is caveats. There's nothing wrong with being transparent about limitations, but they may be a sign that there might be alternative ways to present the data, or that the data may be flawed (depending on the caveat)
* Theres several categories that I think have been invented for the purpose of collation, but I don't think are defined on the page. I think "Overall Core" and "Core I/O" aren't explained, which means by definition it's impossible for a reader to understand the score table.
* And as we're all aware right now, current Claude models are currently struggling to write coherent English. There's several incoherent sentences on the page. It's a Claude issue.
Speaking as someone who consumes this, I appreciate how it's laid out. As a developer, we can often see at a glance where the bottleneck is if we have enough data laid out - IOW, data overload for you is me feeling like a kid in a candy store.
Sometimes there are ways to make things easier without dumbing them down, but way too many people conflate the two; I get nervous when non engineers say "I've studied this, it should be easy".
ah, yes that's obviously fine; I'm definitely coming fromt he perspective of "bored guy on a Saturday evening who happened to stumble across someone else's wall of charts on HN"
I think the Phoronix test suite does pretty well for the end user who wants something easy to digest; the exact choice of benchmarks is sometimes odd but the harness itself is quite good.
At some point I'd like to get our own automated pts runs going, since Michael is not consistent with what hardware he tests on and he hasn't been consistent with getting them out.
It's not easy to show so many data and make everyone happy about the way it is presented. In fact I'm aiming more at engineers and trying to provide as much info as it is possible to be clear about methodology and everything around it.
But raw data is in JSON files so you can always make PR for creating additional view aimed at philosophers and not engineers ;)
The author of the benchmark here. I went over some comments and I'll try to tackle them here.
I'm pretty clear that GH runner based benchmark is far from perfect due to noisy neighbours etc. Thus every test first is running so called calibration... to reject completely unreliable VMs.
I'm fully aware that this can't completely fix the issue. Can limit it but not fix.
But as of now there are 593 runs recorded so average should still be quite meaningful.
Having that said I'm desperately trying to get REAL hardware to run that benchmark. With some successes ;)
But this new hardware has also so many disks that the plan is to try also more complex, tiered cache topologies. I'm working on it.
I'm happy to answer any other questions, sources of every piece of this benchmark are freely available and I'm not saying they are 100% correct. I'm open to improvements.
I went back and forth with Hetzner a couple times, I think we just got a bad machine :)
I've been saying it for months, but eventually I'm going to move the automated builds off the 48 core monster and we'll be able to use that for automated perf testing too. The machine we just got has spindles for EC perf testing, but the Hetzner monster has very high end enterprise ssdd.
1. Dual Ext4 + external 32GB journal X4 pcie SSD (the prior winner of benchmark surveys)
2. Bare F2FS after a trim and SSD vendor software cache flush operation (it should be slower, but knowing how much slower on identical hardware could be interesting.)
3. DRBD across a 48U 100Gbps host rack (single X4 pcie data drive per host, OS on primary)
4. CephFS across a 48U 100Gbps host rack (single X4 pcie data drive per host, OS on primary)
2. If you have the time, adding XFS + mdraid + dm-integrity [1] (in bitmap mode) as a comparison point against ZFS RAID-Zx might be an interesting data point. That's what I run, personally.
3. Did you give some thoughts to the I/O scheduler choice? Might matter a lot in some cases.
1. XFS reflink is enabled and its reflink/CoW-break measurements do run. The dashboard button currently means “native/full-CoW filesystem family”, not “supports reflink”, but that distinction is not clear from the label. I’ll rename it to “Native CoW” and add a separate reflink-capable filter that includes XFS.
2. The current integrity comparison is XFS on LVM/dm-raid10 with dm-integrity in its default journal mode. It is not mdraid and not bitmap mode, so your suggested stack would be a genuinely different and useful data point. An md RAID5/6 over per-member bitmap-mode dm-integrity comparison against RAID-Z1/Z2 makes sense, with the weaker post-crash bitmap semantics documented.
3. I did not pin or record the scheduler, which is a reproducibility gap. The dedicated SAS machine currently has mq-deadline active on all HDDs and SSDs. I’ll add queue/scheduler metadata to results before considering separate scheduler variants, since it can strongly affect the mixed and latency-sensitive phases.
Love to hear that there will be more real hw tests.
At the moment I am building NAS and used your benchmark for evaluating the filesystems.
I am glad to see that your data roughly matches mine (apart from scrub which on 4x 6tb HDDs took 15 hours for md-raid10 while CoW systems took seconds).
Personally I found that array of HDDs behaves very differently than GH runner (my feeling is that since it runs on same disk you are testing theoretical throughput rather than ability to utilize disks).
My tests gave an idea for following topologies:
* 4 HDDs (for example dm-raid has read balancing optimized specifically for HDDs)
* 5 HDDs (classical raid should see no improvement but btrfs and bcachefs should balance the load)
* 4 SSDs
* 3 HDDs + 1 SSD no tiering
* 2 HDDs + 2 SSD no tiering
* 1 drive 10x larger than others (since how bcachefs and btrfs allocators work)
* nocow
zfs-style scrub only reads allocated blocks and skips unallocated blocks; if the pool is mostly empty it can complete very quickly.
Layered storage systems with a RAID layer that makes N disks look like one big disk generally don't have visibility into which blocks are free and which are allocated so they must "scrub" all the disks on initialization and repair even if only 1% is used.
I have three identical Lenovo SFF PCs with a U.2 SSD in each. I'm currently running them in a Ceph cluster but I'll be tearing that down soon. I could run some benchmarks with three in one box and report back? Would be a one-time thing rather than an on-going commitment though.
Disregard, I just saw you have some RAID10 tests in there so three SSDs won't be enough.
