There are a bunch of reasons, some of which are explained in the description and in the docs. For me, the most important benefit is speed of copying. Note that syq doesn't currently implement rsync's delta merge algorithm, which allows it to avoid copying data that is already in the destination file but at a shifted offset. I might implement this (or an enhanced version of it) in the future. If your workloads have a lot of cases like this then syq might not be better than rsync for you, but I found that for me this rarely came up.
One instance where rsync's delta merge comes up for me: For all of my virtual machines, I build a bootable live ISO so that way I can cleanly swap out the full ISO with a new one when there are software updates. This means that every time I want to change a config file inside the virtual machine, I end up building a new ISO and rsyncing it to my server. rsync speeds this up significantly since most (but not all) of the ISO is unchanged.
Nice example. Rsync might be better here. Although syq will still avoid copying most of the image if changing the config file just modifies a small number of locations and doesn't globally move data around (like prepending a single byte to the ISO)
The iso itself is read-only. When I need to update software I build a whole new iso and replace the existing one. For any folders/files that need persistence across reboots, I store that data separately from the iso, either in a virtual nvme drive or a mounted 9pfs folder from the host. The iso mounts the persistence drive/9pfs via /etc/fstab. So in the end, the software and config files are read-only (until I replace them with a new iso) but my databases and data files are read-write.
Getting into the unnecessary details: in classic live ISO fashion, the root (/) filesystem is tmpfs. Then I have at most 1 persistence drive and/or at most 1 9pfs mount. From there, I use bind mounts to only persist the few folders/files that I want preserved. For example, I'll have my persistence drive mounted at /persist and I'll bind mount "/var/lib/etcd" to "/persist/var/lib/etcd". Then when I reboot, everything outside of "/persist" is wiped out and I'm back to the initial iso + anything in /persist. So it is very similar to running a docker image with a volume mount, or running tails with persistence.
Thanks for the suggestion! I'm not very familiar with btrfs send and btrfs receive. Browsing the documentation, it looks cool, but maybe you could say more about the specific things that make it so much better for your use case? Could any of them be incorporated into a more general tool like syq? Also, it doesn't look like btrfs send helps transfers go fast by parallelizing them or has anything like syq's remote-remote transfer feature, which made me wonder if it would actually make sense to make syq able to work as an authenticated fast transport for a stream generated by btrfs send/receive.
btrfs send is useful for sending incremental updates; this is used a LOT for backups; and it saves a lot of time because it does not need to traverse files/directories that were not changed
You cannot do this without help from the filesystem. That's why rsync itself does not support this, and it has to be done by a specialized btrfs tool in this case.
Browsing the repo, I can see a couple of advantages:
- Single contributor. Software bugs are generally caused by developers writing code. By reducing the number of contributors, syq has cleverly reduced the surface area for defects to sneak in.
- Distribution via shell script. rsync is bundled in most linux distributions, which means you have to deal with annoying software updates from time to time. Syq, OTOH, tells you to pipe curl into bash to execute a shell script, which means one-and-done installation and maintenance.
- UI clarity. If you search StackOverflow for rsync, you'll see thousands of questions asking how to accomplish various tasks with rsync, both straightforward and arcane. By contrast, `syq` isn't even a tag on SO. The obvious conclusion here is that rsync's interface is so byzantine, and its documentation so poor, that users must resort to asking strangers for help, a problem obviously not shared by syq.
- No need to be snarky. It's a new project, while rsync is time tested, and that is certainly an advantage for rsync and all old software.
- You can install by piping curl to bash, using brew, or by compiling yourself. One thing about distribution I would argue syq gets right is that you are never relying on whatever version of syq happens to be installed on a server. The client syq always talks to a server syq tagged with the exact same version, and when your local copy of syq installs a remote copy, it verifies that the remote binary is signed by me.
- I never said or implied that rsync has a byzantine interface or poor documentation. I think rsync is well documented and the interface is overall fine. Syq even has an rsync compatibility mode, so you don't have to learn any new syntax if you don't want. Still, I tried to provide good docs for syq too, and honestly even if there were a gap in the docs, all you have to do these days is ask AI to look at the source code and tell you what to do. I think this also mitigates the trust issue with a new project from a single contributor.
FWIW, I'm only half-joking with my comment: dealing with software updates is an annoying part of life, and IMO rsync's UI is rather complicated (in that it's easy to use the wrong set of flags and do the wrong thing, sometimes in annoyingly subtle ways) and I basically always need to refer to the man pages (or, these days, an LLM, followed closely by the man pages to sanity-check what I'm doing) whenever I'm writing a new incantation of it rather than copying from an existing script or my shell history. I just chafe a little bit at declarations like "better than," since that's clearly not true (in the same way that rsync is not unequivocally "better than" syq), when more precise and less conceited descriptors like "faster than" are sitting right there on your benchmarks page.
Yeah, I get that. I really didn't mean to sound conceited, and I don't think that syq is better than rsync for every use case. I was just trying to convey in a very short space that it is something that rsync users might be interested in because it solves some frustrations with rsync.
It just installs in $HOME/.local/bin by default (and some other parts go in other locations inside your home directory). You don't need superuser access for anything. The first time you connect to a server, it installs its matching counterpart in your home directory on the server.
Let's say this is widely used and let's presume that a specific version has a security issue. With everyone having their own copy of the binary on every host versioned at the time they first ran a command on that host, or any decently sized fleet, it'll take a big effort to track down and ensure all versions have been updated or removed (to be re-generated later).
Frankly, we would re-kick our fleet ahead of schedule rather then try to remediate it more manually. But we re-kick our fleet on a rotating yearly schedule, so it's not too much an effort to re-kick it earlier. It's a completely automated system in place now.
