> But who cares? Why not just let the command figure out the right buffer size automatically?
Well, because it doesn't. At least on the Linux version I used in the past, it defaulted to 512 byte blocks or something similarly small and it commits every block individually, leading to really slow performance with USB sticks with controllers not smart enough to bundle writes together. I wouldn't be surprised if that incurs some heavy write amplification at flash level too. Perhaps it's smart enough to figure a better block size now but this is where that habit comes from.
Another thing, creating sparse files with the seek option (simply put files containing zeros that are not actually written to the disk nor taking space, but do turn up all these zeroes when you read the file). Also something not duplicated with cat or head.
What I like about dd is that it can do pretty much all disk operations in one simple tool. Definitely worth learning all of it IMO.
It uses a syscall per block write (which would slow it down if you use 512 byte blocks instead of 8M for example), but the OS does the file buffering and final writing to the device, unless you pass the fsync or fdatasync options to dd.
Edit:
here's writing to a old and slow stick and you can see that dd is fast and then the OS has to transfer the buffered data onto the stick:
dd if=some-2GB-data status=progress of=/dev/sdX bs=8M
1801256960 bytes (1,8 GB, 1,7 GiB) copied, 7 s, 257 MB/s
0+61631 records in
0+61631 records out
2019556352 bytes (2,0 GB, 1,9 GiB) copied, 378,59 s, 5,3 MB/s
And the stick is placed in a USB 3.2. port on a fast machine ;-0
> Can also do `oflag=direct` and it'll just skip as much of the caching as it can.
Correct, and that's one more point for dd compared to head/tail (which are fine commands by themselves).
But wouldn't help much in my example, where I used an (very) old "high speed usb 2.0 stick" with 4 MByte/s write speed to demonstrate the difference between buffering and actual writing.
Thats when it matters, with cat it would cache the entire write. If you don't want it to cache everything and have no idea how much time is left or how fast it is writing, and then wait an hour for it to unmount instead then cat is fine.
Right, and the parent comment is saying that `cat` doesn't figure it out very well.
I recall that GNU cat used a constant value that was tuned for read performance on common hard drives, giving no consideration to write performance. Looking at the GNU cat sources today, it doesn't look at all how I remember; I'd have to study it a bit to tell you what it does.
Edit: Hrmm, it seems I was mis-remembering. Perhaps I was thinking of the minimum values (32KiB, 64KiB, 128KiB below)? Or perhaps I was thinking of a BSD cat? Anyway:
How GNU cat sizes its buffers, by version:
- >=9.1 (2022-04-15) : Use the `copy_file_range()` syscall and let the kernel figure it out
No, I was talking about dd. As far as I know it does not have any smart tuning of block sizes. It definitely didn't in the past and I doubt it was added.
Not sure what cat does but like I said in my other comment its not really cat itself that does the disk writing in that command but rather the shell redirect. Edit: Nope, I'm wrong there!!
Ah, I misunderstood you. You're correct, `dd`'s default block size was and is 512, which is specified by POSIX.
I interpreted "the command" in the original article as "the command you end up running, whether it be `dd` or `cat` or something else."
But you're mistaken about the shell redirect. In either scenario it's the command (`dd` or `cat`) making the write() syscall to the device. The shell passes the file descriptor to the command, then gets out of the way.
Indeed I was mistaken about the shell, sorry about that. The redirect method I forgot a really long time ago and my memory made it into something it was not.
In that case I guess dd does call a sync() on every output block? Because it's definitely slower and the LED pattern on a USB stick is also much more 'flashy' when using 512 bytes.
For context, choosing the block size has implications on the speed at which data gets written to a USB stick, so not being able to tune that on cat can be a problem.
I guess that made sense when the standard HDD block size was 512b. Then it went to 512/4kb then pure 4kb, not sure what SSD's do maybe also 4kb? Experimenting with speeds and block sizes in the past has shown very quickly diminishing returns increasing the block size. As long as the CPU can keep the queue depth over 1 the device should be flat out.
Generally they do a logocal 4kb, but it's usually physically larger pages of more than 1M per write. a good ssd will helpfully cache and batch up small writes but if it gets it wrong then it'll amplify the wear and kill a drive quicker than needed. That's another reason to do dd with a larger block size, since it'll make it a lot less likely that you write multiple blocks for a single update
Good point. I guess there's so much going on with various types of caching and wear levelling that 'let the device figure it out' is best. And the queue can be on the device now with NVME not on the host so its not a dumb queue any more.
I don't read it that way. He calls out that 'weird' command specifically. But indeed he doesn't specify.
I wonder what cat does in terms of buffers, I kinda doubt it has any special optimisations though I would guess the shell redirect might have. As that's really the thing doing the work there, not cat. Edit: Nope, I'm wrong there!!
Also that command does more than just specify the memory buffer like he says. That's my point, it's useful for tuning which can be super helpful with huge images.
It can also lead to some dangerous gotchas as well when working with files. But with full disk images these don't apply generally.
> though I would guess the shell redirect might have. As that's really the thing doing the work there, not cat.
No? The shell redirection is just
int tmp_fd = open("/dev/sdb", O_WRONLY|O_TRUNC, 0666);
dup2(tmp_fd, STDOUT_FILENO);
close(tmp_fd);
(plus error handling and whatnot)
`cat` ends up with a file descriptor directly to the block device, same as `dd` does; the only difference is whether the `open()` call comes before or after the `execve()` call.
I don't have data at hand but if you choose the right value it's meaningfully better and can also lead to more efficient patterns in bash scripts that are more complicated than `cat in > out`
Unfortunately dd has not just footguns but foot cannons that are amplified by the mistakes people often make with string escaping, odd file names, loops, null checking, and conditionals in bash.
Well, because it doesn't. At least on the Linux version I used in the past, it defaulted to 512 byte blocks or something similarly small and it commits every block individually, leading to really slow performance with USB sticks with controllers not smart enough to bundle writes together. I wouldn't be surprised if that incurs some heavy write amplification at flash level too. Perhaps it's smart enough to figure a better block size now but this is where that habit comes from.
Another thing, creating sparse files with the seek option (simply put files containing zeros that are not actually written to the disk nor taking space, but do turn up all these zeroes when you read the file). Also something not duplicated with cat or head.
What I like about dd is that it can do pretty much all disk operations in one simple tool. Definitely worth learning all of it IMO.