To this day I admire Linux for having the grit and perseverance of maintaining ext2, ext3 and ext4 as separate code bases for so long. Equivalent FreeBSD UFS, UFS with soft updates and UFS with journaling is one code base with each generation as a backward compatible feature. Given FreeBSD's negligible market share perhaps worse is indeed better.
FreeBSD usage is mostly hidden behind very large storage environments. Think PB scale HPC and NetApp (the glorified FreeBSD with some sellable added complexity), all running FreeBSD at it's core.
huh, does that explain why the PS5, in 2023, still gets really upset about having power yanked while it’s not turned all the way off? It yells at you (talk to PG&E, Sony) and does a slow fsck when power is restored. Being stuck with old FreeBSD FFS would explain it. You’d think they could invest in a modern filesystem to make this headache go away.
Good license with a bad product is still bad, so there must be more the "just" the license. For example netflix, they could use linux without any problems (since they don't redistribute the code nor the appliances).
>Plus they also ship devices with Netflix code on them.
Are you talking about open-connect? They don't ship that to "End-users", and you can sell an appliances with GPL software and proprietary code on it..hello Android?? And please stop now, your arguments are like from a 12yo, and i know for a fact that your much much older, also sometimes wiser, i don't know why that is not available for you today.
What BS? The amount of precious upstream changes that FreeBSD gets from all vendors shipping their code?
Android is actually a good example, it is no accident that Google has been cleaning Android from GPL code, leaving only the kernel and since Android 8 only Project Treble drivers (out of kernel) are used.
Fuchsia is even better, only Apache and MIT licenses.
Ironically, I've had far more problems with Linux systems using extfs losing data or requiring very lengthy fsck'ing after hard resets than Windows 9x or even DOS on FAT, despite experiencing more hard resets and routine hangs in my use of the latter.
I suspect the relatively more aggressive write caching of Linux and Unix-likes, combined with their more complex filesystem data structures, leads to a design in which unexpected resets easily cause lots of filesystem corruption. Contrast that with FAT's simplicity, minimal (effectively none) write caching, and the tradition of cutting power as soon as you returned to a DOS prompt. Maybe that's actually the true "worse is better".
Yes, ext2, before journaling was added to extfs was painful, powerloss, or simply switching off a computer without runnng shutdown often resulted in data loss and if you're unlucky enough, fsck threw you in to "please try to fix this, I can't" prompt on the next boot.
"Ironically, I've had far more problems with Linux systems using extfs losing data or requiring very lengthy fsck'ing after hard resets than Windows 9x or even DOS on FAT, despite experiencing more hard resets and routine hangs in my use of the latter."
Your experience does not match the vast majority of users. The only OS I lost data on was on Windows when a drive crashed hard. When the same happened years later on Linux, I lost exactly nothing.
Yeah, I've been using Linux almost exclusively since 1994. I've had power outages and a few crashes but I can't remember ever losing data with ext2.
I don't remember losing a lot of data in DOS/Win3.1 but I rarely used it because it crashed so much.
I did have a lot of keyboard and X11 lockups in the 1990's but after a year I got a dumb terminal for $20 so someone could still check email, read Usenet, etc. while the main display was in use. If the main keyboard or X11 server locked up I could use the terminal to sync the filesystem and reboot the machine cleanly. I remember doing that quite a bit when playing full screen games or anything with SVGAlib
I've had btrfs lose data several years ago from download-> quickly reboot without manual sync, as well as an ext4 system partition corrupted by unintended power loss (only once that I know of, many other times I had no issues).
> would report a successful write to an sd card before it was done. To "help" with speed.
Sadly USB sticks are handled the same way. The write notification and progress bar are long gone but you won't be able to eject that medium for several minutes.
Don't remember if it is the same on windows, since i use it rarely.
That's not the same, GP was talking about the hardware lying, not buggy software abstractions that affects measurements but don't lose data.
If you commit your database journal, you're told it's done and you start overwriting data, but the journal was never persisted... a loss of power will wreck your entire database.
If the syscall meant to flush data to disk doesn't do that due to an implementation detail of a filesystem, I think there's nothing the user can reasonably do to ensure their data is safe.
way back when, unix admins were recommended to enter "sync" twice. this wasn't anything wrong with the program itself, but the time it took to enter the second sync gave the os to finish flushing its buffers to the hard disk.
The first version of the ext filesystem (ext1?) was in 1992. I had some friends who ran Linux then (one of them was a Linux developer) but ext2 was released in 1993. By the time I first installed Linux in 1994 ext2 was already the default and I remember documentation saying NOT to use the original ext
I staied away from XFS (on PC) because my experience was different. Power outage -> replay journal log (no fsck) -> repeat a couple of times -> filesystem corrupted heavily.
> Ironically, I've had far more problems with Linux systems using extfs losing data or requiring very lengthy fsck'ing after hard resets than Windows 9x or even DOS on FAT,
This thread is wild. Are you saying you're still running Windows 98 and DOS machines (and for that matter Linux running a non-journaled ext3/4 device, something which no one has tried in decades)?!
Or is this literally a comment pulled from 1994? People really want to have an argument over the quality of a system in 2023 based on screaming we all did literally 30 years ago. Unbelievable.
Really? As someone who has been using Linux since the 90s, for me the corruptions were extremely rare. Full fsck of course was happening regularly after unclean shutdowns, but I think I encountered only 1 or 2 actual corruptions.
I had it happen many times as well. A power outage or a thrashing system were often accompanied with an fsck and file loss on ext2. This even included system files like libgtk.so because they were open at the moment of crashing. I never understood why reading/opening a file would corrupt it on a crash. I was happy to switch to Reiserfs.
In my own experience, such situations were more likely when the system was already suffering some other issue or failure, like a failing array member or a volume out of space or something more routine, and in the course of troubleshooting, ah shit, now I have a corrupted root to deal with.
Had one app that crashed inexplicably every year at midnight on Halloween. Or at least the two years I put up with it. Had to leave a party to rebuild a server from scratch when one failure turned into lots of failures.
Windows due to power outage and could not recover, yes. Linux, no. As another poster in the thread, your reported experience does not match the vast majority of reality.
> After they discovered this
problem, they added extra capacitors to the power supply, added a power fail interrupt, and taught Irix so that when the power fail interrupt was triggered, it would frantically cancel DMA transfers in order to avoid this problem.
Hilarious, but, I suppose, effective. Perks of controlling the entire vertical...
I like how almost everybody back then earnestly believed that if power dropped, the drive would use its spinning-down motor as a generator to power finishing the current sector write, and then home and lock the head.
I don't know if any drive ever did that. But certainly the drives actual people actually had would not. Power hiccups, and whatever is being written becomes garbage (and maybe a few sectors after it, too), and the heads flop loose.
Maybe a top-quality drive would turn off current to the write head if power sagged, so that the controller going crazy would not have lasting consequences. But certainly cheap drives never had anything that might have cost an extra penny.
Operating system engineers seem to often have a "bug" in their thought process where they believe the models of hardware they use are close to 100% true.
It turns out that they are often almost completely false, so relying on those promises is a bad idea.
Good luck for developping a robust system by considering the execution plateform can introduce bugs anywhere and trying to resist them all. The article makes me think the Linux journaling redudancy robustness was not even really a conscious choice, but more a happy accident coming with a simpler design because less effort cound be put in it.
Now maybe simplicity everywhere is actually beneficial for mean robustness, but this will not be true for all particular cases.
> when power drops, and the voltage levels on the power
rails start drooping, DRAM tends to go insane and starts returning
garbage long before the DMA engine and the hard drive stops
functioning
That doesn't sound right. DRAM can keep its content for a few seconds without power - it's what make Cold boot attacks [1] possible. Also, DRAM has its own buck voltage regulators, usually with hefty inductors, capable of delivering power long after +5V and +12V started to drop.
Having memory return a few bits flipped can have disastrous consequences on a running system. On the other hand having memory with only a few bits flipped is still a gold mine for a cold boot attack.
It could look like the DRAM is the problem if the data bus was the actual problem, as the DMA would request data via the bus. This is just speculation here obviously. But if you have lots of hardware on a tri-state bus most of which is not of "data centre" quality then it wouldn't be unusual for bus arbitration to go out the window in a fluctuating power condition. So from the DMA through to write operation you have data corruption issues, and you'd think it was probably the DRAM but in fact if the bus arbitration is corrupted then you still get corrupted data. I suspect sure that the vast majority of PCs don't have error detection (let alone ECC) on their data buses (even something as simple as a parity bit), but I could be wrong.
> In Linux,
ext2 was incredibly sloppy about how it handled write ordering
--- it didn't do anything at all. But as a consequence we developed
tools that were extremely good to compensate
So worse is not better, then! More like worse needed greater efforts to make decent.
Imagine all that great tooling effort, applied to create excelent tooling for the other so-good and well engineered filesystem, the FFS mentioned in the article.
It's important that people realize here that the software we're talking about is not "Linux" as it exists in the tree today, or even any time over the last 20 years. It's Linux of the mid-90's, when the Zeitgeist of the system was a mad rush to reach feature parity with commercial unix on commodity PC hardware that failed all the time anyway[1].
No one wants to hold up ext2 as a paragon of software quality. But in practice in the mid-90's, it worked just as well as its competitors on the hardware where it was deployed, and quite frankly was much faster. It won for some pretty obvious reasons, and was compatibly replaced over the years with evolved versions that have all the features anyone wanted.
[1] Notably here: at that point it was routine to find IDE drives with large multi-track buffers. Almost none of this hardware did any of the write ordering magic[2] internally, nor did the protocol provide a way to distinguish "on disk" from "in drive RAM". The problem wasn't solvable in the regime where the software was being deployed.
[2] Which IMHO was oversold by the BSD people even at the time. Contrary to the way it's presented, the Berkeley Fast File System was absolutely not reliable in the face of power loss in the way that modern journaled filesystems are. It had to fsck on boot and deal with metadata corruption too, it was just much better about it.
Yeah it seems that the kind of brute force or less than ideal software-engineering style of ext2 was a perfect match for the kind of hardware qualities (i.e. very low ones) that were the target! In the end everything ended up working better, even if due to completely accidental reasons.
Or maybe the decisions were a form of emergent behaviour; the developers that succeeded were the ones whose behaviour best matched the environment. The accidents ARE the reason it worked; i.e. evolution.
Yeah once you start thinking in evolutionary terms you see it everywhere. Like why do we have this discussion via interactive documents? The web has a sort of evolutionary fitness that lucked its way into being a popular app platform.
"Worse is better" has a very specific (counter-intuitive) meaning. It does not purport to be better in the sense you're thinking about it. The "better" is in the sense of "has an easier time gaining widespread adoption because the few things it does promise to do it does very reliably and it is quick to go to market".
"Worse is better" never had such a defensibly coherent meaning. There were gestures toward one, enough to fool people, but it mainly amounted to dressed-up sour grapes: "We would have won if not for those meddling kids!"
The fact is, Unix (and then Linux) was strictly better at the things people wanted to do, full stop.
Um, "worse is better" if your only metrics are things like time to market. How many Linux technologies have not been able to surpass their counterpart technologies after years of iterating? Two are top of mind: epoll and btrfs.
Sometimes designing a system is the best approach. Sometimes iterating on a "worse is better" technology is the right approach. There is no single answer.
I think we need to be more rational about the real reasons Linux is/was so successful. Yes, sometimes it was about "worse is better" but certainly not the majority of the time. My guess is -- the reason Linux was so successful can be summarized much more easily as: free as in beer, unpretentious (does not require adopting some high UNIX religion), appealing licensing (I don't love the GPL/licensing religion but lots of people do!), and it was very, very lucky.
> I think we need to be more rational about the real reasons Linux is/was so successful.
I always think there were two reasons, but one of them didn't matter. The first reason, that didn't matter, is that Linux was free. That didn't matter because BSD was also free. The second reason, and the main reason, is that it had the force of popularity, or verve, behind it by a fanatical group known colloquially as the Penquinistas. By and large I believe they were young developers that often crashed their laptops, and they liked Linux because it booted faster. I honestly believe Linux taking over the datacenter in 2012ish was entirely arbitrary. Booting faster is fundamentally a silly metric in the datacenter. But the takeaway is the change to Linux was driven by developers, not systems administrators, who for the most part were caught with their pants down and complained that Linux wasn't ready (to success in 2006 with IBM, which rolled back their global Linux deployment).
In the first half of the 1990's BSD was sued by Unix Systems Labs, who had bought the rights to AT&T Unix, claiming copyright infringement [0]. It took a couple of years for everything to be dismissed and settled.
"Ultimately, the legal drama did not undercut programmers’ ability to use or redistribute BSD. However, it did stunt adoption of the operating system by creating doubts about BSD’s legal future. As a result, it arguably forged an opening that allowed Linux to gain ground despite being developed primarily by an undergraduate in his Helsinki apartment, rather than a team of professional computer scientists at a major American university."
I've always heard that this lawsuit was significant, ruining the momentum that BSD had at the time, momentum that ended up shifting to linux.
> The second reason, and the main reason, is that it had the force of popularity, or verve, behind it by a fanatical group known colloquially as the Penquinistas.
Wholeheartedly agree with this, maybe not all your particulars, but yeah agree.
> sometimes it was about "worse is better" but certainly not the majority of the time.
Linux had a heyday with this that matched the growing computer and technology market of the time. The state of the art just happened to be moving fast enough to justify this approach.
> How many Linux technologies have not been able to surpass their counterpart technologies after years of iterating? Two are top of mind: epoll and btrfs.
I'm saying epoll is an example of a technology which has not been able to surpass its Solaris or Windows equivalents.
Seems like there's a semantic problem here. epoll won. It clearly "surpassed" its competitors by most definitions. Arguments against it aren't grounded in epoll's capability or performance. They're about aesthetics. People used to kqueues think epoll is ugly. Well, OK. I think that's fair. But being beautiful isn't sufficient to declare a software feature as having "surpassed" another, again by conventional definitions. It's just a way to win an argument.
Really the epoll nonsense is a perfect microcosm of exactly this kind of argument. For 30 years the BSD folks have been making fundamentally aesthetic arguments into a market that demands practical solutions. And... losing, kinda.
FWIW: I think the btrfs argument is rather stronger on a technical level. The problem there is that people stopped caring about local storage filesystems about 10-15 years ago. Local devices are a solved problem, and all the innovation in storage technology is happening in the cloud today, in front of software layers that are only vaguely understood as "filesystems" in the Unix sense.
I agree there is a semantic distinction to be made re: epoll, but also an unstated premise: Worse is better when you can iterate to good enough.
What is good enough? Well I think you may forget another metric: developer experience and ease of building a correct product. Say you're building a web framework or server, how long is that going to take do that correctly with epoll compared to the alternatives?
> Say you're building a web framework or server, how long is that going to take do that correctly with epoll compared to the alternatives.
This is still making too much of the subject.
Every major server framework was epoll-capable within a few months of the feature landing. Again, epoll won. It's fine. It's more than fine, it's actually pretty great. It's not hard to use, at all (at least relative to other technologies in the same space). It's just not as subjectively pretty as kqueues.
epoll didn't win? Linux won. Which was one point of my post (that you're nitpicking). Linux wins for reasons which mostly have nothing to do with design philosophy ("worse is better"), or technical superiority.
I still think you're being way too general here by deploying hyperbole like "technical superiority" or "surpass" to subjects that are just about aesthetics. Adapting it to your frame here, my point was something to the effect of: if kqueues were clearly superior technically, then applications[1] which rely heavily on widely-multiplexed kernel notifications would naturally have migrated to the superior solution. (Ditto for zfs/btrfs, etc...)
And... there really are none. For all the yelling, in fact epoll works great. It still works great. Claiming something is "technically superior" to a product that clearly works great in the market is a red flag that you're probably not arguing about technology.
[1] Which are MANY! The whole cloud runs on epoll!
> if kqueues were clearly superior technically, then applications which rely heavily on widely-multiplexed kernel notifications would naturally have migrated to the superior solution.
I get it -- you don't like the way I phrased my point. More power to you. But you keep shadowboxing a straw man.
My point was: It doesn't matter if kqueue is clearly superior technically or aesthetically or otherwise, Linux (not epoll) wins for reasons that have nothing to do with that.
In order to actually make your point re: aesthetics, you have to make a much stronger case, which is that epoll actually is at least the technical equal of kqueue, not merely your feeling that apps would have naturally migrated, etc. Instead of -- people chose Linux (again not epoll) because of ... inertia, herd mentality, knowledge base, app availability, etc.
> you have to make a much stronger case, which is that epoll actually is at least the technical equal of kqueue
It is, though. Period, full stop. There are no event handling problems solvable with kqueue that cannot be implemented in epoll with code of equal or higher performance and equivalent complexity. There's not one you can point to in practice. Not one. So... doesn't the burden of proof go the other way?
Again, I repeat: you're still trying to have an aesthetic argument (in this form: that Linux won for reasons it "doesn't deserve" or something), but you're using the language of unambiguous technical determinism. And, as BSD has discovered over three decades of this nonsense, that doesn't get you anything but an echo chamber.
What are we arguing about again? You simply disagree as to whether epoll is the equal of kqueue? Who cares?
> It is, though. Period, full stop.
Yeah I didn't ask for you not to explain yourself.
> you're still trying to have an aesthetic argument (in this form: that Linux won for reasons it "doesn't deserve" or something)
No I am not. I don't care either way whether Linux deserved it or not. I am only saying, at the time it mattered, technical merit wasn't the reason Linux beat all the other Unixes, because obviously it wasn't. This truth is at least interesting to consider. Now, reread my initial comment, and ask whether your nitpicking matters one iota?
Just because you think my affection for kqueue is an aesthetic rather than technical choice (because broken semantics aren't technical, okay whatever) doesn't matter to the point of the comment at all.
You bike shedded my comment. I agreed with your nitpicky point, but said there may be other concerns, and then its 5 comments about how I'm still wrong about something-I-know-no-what? I guess my aesthetic sense? JFC. You've never made an affirmative argument! It's all sniping from the sidelines.
> How many Linux technologies have not been able to surpass their counterpart technologies after years of iterating? Two are top of mind: epoll and btrfs.
Apples and oranges. No one is required to use btrfs, it is one of many options for filesystems. In my opinion, btrfs is dangerously close to a failed filesystem design for a variety of reasons, but no one needs to use it. That is a good thing - some designs succeed and some not so much, and the successful ones survive and prosper, no need to change operating systems or anything like that.
FFS/FFS2 on OpenBSD is definitely one of the system's weaknesses. In addition to being kinda slow it's also a bit unnerving that the file system invalidates relatively easily. While it doesn't happen regularly for me, and despite never having suffered actual data loss, it's always unnerving sitting through a forced fsck session.
It's not survivorship bias when you do selection. ext2 was not the only FS type available on Linux. As always with Linux, there was this wide variety of more or less off-the-wall choices. ext2 and it's lineage stuck with us because people started thinking that it was the one least likely to eat your data. The fact that ext2 happened upon a technical decision that worked well with cheap hardware was luck, the fact that ext2 ended up as the "default" file system on Linux was not.
No, but the "worse" technical decision survived because it turned out to be "better" out of sheer luck. If it hadn't, they would have found a fix for ext2 or we would have gotten ext3 sooner, or we would have gotten a different default FS. (Survival of the "bad" decision, not the FS itself, at least not necessarily.)
Intentionally exploiting selection is sort of the point of "worse is better" as a design strategy. By promising fewer things in the design, you can get software in front of users faster, and you find out sooner what works well and what does not.
Unix/Linux is successful because for decades its apparently been ok to ignore the gaping security hole specifically left out of the cloning of Multics. Without a way to run untrusted executables, it us completely unfit for the current 24/7 connectivity and mobile code environment we find ourselves inhabiting. We've patched around it with VMs and Containers, but we all just need a way to hand a file to an untrusted program, and that's the balls part of Multics that Unix cut away.
If you deal with many small files on Windows, you will observe much better performances in WSL2/ext4 than windows with NTFS. Even though the ext4 filesystem is stored as a file inside a NTFS partition.
If you use NodeJS for example, it’s a very obvious speed difference.
Is it NTFS or is it the NT I/O stack? Because the latter is so impressively complex I’m not sure if it’s actually possible for it to do fast metadata operations.
(I remember Raymond Chen writing there was a special metadata caching layer in the NT FAT driver because otherwise it would be unbearably slow. Linux has a general-purpose dentry cache playing a similar role—does modern Windows have anything similar for NTFS?)
Practical File System Design:The Be File System covers the designs of other file systems briefly and goes from FFS to XFS, NTFS, EXT2 (book was written in the late 90s)
"We covered the grandfather of most modern file systems, BSD FFS; the
fast and unsafe grandchild, ext2; the odd-ball cousin, HFS; the burly nephew,
XFS; and the blue-suited distant relative, NTFS. Each of these file systems
has its own characteristics and target audiences. BSD FFS set the standard
for file systems for approximately 10 years. Linux ext2 broke all the rules
regarding safety and also blew the doors off the performance of its predeces-
sors. HFS addressed the needs of the GUI of the Macintosh although design
decisions made in 1984 seem foolhardy in our current enlightened day. The
aim of XFS is squarely on large systems offering huge disk arrays. NTFS is
a good, solid modern design that offers many interesting and sophisticated
features and fits well into the overall structure of Windows NT."
OpenBSD it's the best case of doing thinks right. Stuff is too crappy and insecure to run, such as BT or Linux_compat(4)? Out. Laptop Fn keys for brightness? OOTB. Even with fvwm.
Hyperbola GNU /Linux tries to do the same as a Parabola forked distro.
