This actually made the system boot but there are some leftovers being installed on first boot that I've been unable to disable that also causes the system to be unable to boot.
So now, the machine is running but as soon as it is restarted we have to re-image the disk, go through the process of manually removing patches, and then pray that we don't have a power shortage as we'd have to do everything yet again on next boot.
I'm not convinced that this patch will solve the issue either, because if this updates requires a reboot the fix won't be installed if we can't boot. I might try to install this update from the recovery console see if that works.
There is a very fundamental difference between how Unix and Windows view open files:
On Windows, once the file is open, it is that filename that is open; You can't rename or delete it; Therefore, if you want to replace a DLL (or any other file) that is in use, you have to kill any program that uses it before you can do that; And if it's a fundamental library everything uses (USER32.DLL COMCTL.DLL etc), the only effectively reliable way to do that is reboot.
On Unix, once you have a handle=descriptor of the file, the name is irrelevant; You can delete or rename the file, The descriptor still refers to the file that was opened. Thus, you can just replace any system file; Existing programs will keep using the old one until they close/restart, new programs will get the new file.
What this means is, that even though you don't NEED to restart anything for most upgrades in Unix/Linux, you're still running the old version until you restart the program that uses it. Most upgrade procedures will restart relevant daemons or user programs, or notify that you should, (e.g. Debian and Ubuntu do).
You always need a reboot to upgrade a kernel (kernel-splice and friends not withstanding), but otherwise it is enough in Unixes to restart the affected programs.
> On Windows, once the file is open, it is that filename that is open; You can't rename or delete it;
This is wrong... there's no clear-cut thing like the "file name" or "file stream" that you can specify as "in-use". It depends on the specifics of how the file is opened; often you can rename but not delete files that are open. Some (but AFAIK not all) in-use DLLs are like this. They can be renamed but not deleted. And then there's FILE_SHARE_DELETE which allows deletion, but then the handle starts returning errors when the file is deleted (as opposed to keeping the file "alive").
To make it even more confusing, you can pretty much always even create hardlinks to files that are "in use", but once you do that the new name cannot be deleted unless the old name can also be deleted (i.e. they follow the same rules). This should also make it clear that it's not the "name" that's in use, but the "actual file" (whatever that means... on NTFS I'd suppose it corresponds to the "file record" in the MFT).
The rule that Windows always abides by is that everything that takes up space on the disk must be reachable via some path. So you can't delete in-use files entirely because then they would be allocated disk space but unreachable via any path.
> What this means is, that even though you don't NEED to restart anything for most upgrades in Unix/Linux, you're still running the old version until you restart the program that uses it.
What I expect it also means is that you'll get inconsistencies when doing inter-process communication, since they'll be using different libraries with potential mismatches. Is this correct? Because it seems to me that the Windows method might be less flexible but is likely to be more stable, since there's a single coherent global view of the file system at any given time.
Yes, in principle what you've said about the Unix approach here is correct, if you upgrade one half of a system and not the other half and now they're talking different protocols, that might not work.
But keep in mind that if your system can't cope with this what you've done there is engineer in unreliability, you've made a system that's deliberately not very robust, unless it's very, very tightly integrated (e.g. two sub-routines inside the same running program) the cost savings had better be _enormous_ or what you're doing is just amplifying a problem and giving it to somebody else, like "solving" a city's waste problem by just dumping all the raw sewage into a neighbouring city's rivers.
Now, the "you can't delete things because then the disk space is unreachable" argument makes plenty of sense for, say, FAT, a filesystem from the 1980s.
But (present year argument) this is 2018. Everybody's main file systems are journalled. Sure enough, both systems _can_ write a record to the journal which will cause the blocks to be freed on replay and then remove that journal entry if the blocks actually get freed up before then. The difference is that Windows doesn't bother doing this.
> Now, the "you can't delete things because then the disk space is unreachable" argument makes plenty of sense for, say, FAT, a filesystem from the 1980s.
Unix semantics were IIRC in place as far back as v7 (1979), possibly earlier - granted, a PDP disk from that time was bigger (~10-100MB) than the corresponding PC disk from a few years later (~1-10MB), but an appeal to technological progress in this particular example case is a moot point.
> But keep in mind that if your system can't cope with this what you've done there is engineer in unreliability
It's weird that you're blaming my operating system's problems on me. "My system" is something a ton of other people wrote, and this is the case for pretty much every user of every OS. I'm not engineering anything into (or out of) my system so I don't get the "you've made a system that [basically, sucks]" comments.
> [other arguments]
I wasn't trying to go down this rabbit hole of Linux-bashing (I was just trying to present it as as objective of a flexibility-vs.-reliability trade-off as I could), but given the barrage of comments I've been receiving: I don't know about you, but it happens more often than I would like that I update Linux (Ubuntu) and, lo and behold, I can't really use any programs until I reboot. Sometimes the window rendering gets messed up, sometimes I get random error pop-ups, sometimes stuff just doesn't run. I don't get why it happens in every instance, and there might be lots of different reasons in different instances. IPC mismatch is my best guess for a significant fraction of the incidents. All I know is it happens and it's less stable than what you (or I) would hope or expect. Yet from everyone's comments here I'm guessing I must be the only one who encounters this. Sad for me, but I'm happy for you guys I guess.
> ...but it happens more often than I would like that I update Linux (Ubuntu) and, lo and behold, I can't really use any programs until I reboot...
Ubuntu developer here. This doesn't happen to me in practice. Most updates don't cause system instability. I rarely reboot.
Firefox is the most noticeable thing. After updating Firefox (usually it's a security update), Firefox often starts misbehaving until restarted. But I am very rarely forced to restart the login session or the entire system. I should, to get updates to actually take effect, but as a developer I'm usually aware of the specifics, so I can afford to be more selective than the average user.
Are you sure you aren't comparing apples to oranges here, and are actually complaining about the stability of updates while running the development release, which involves ABIs changing and so forth?
Development release? Currently I'm on 16.04, and I've never been on a development release of anything on Ubuntu. I'm just describing the behavior I usually see in practice (which it seems someone attributed to "D-BUS" [1]). Obviously the logon session doesn't get messed up if all I'm updating is something irrelevant like Firefox, but if I update stuff that would actually affect system components then there's a good chance I'll have to reboot after the update or I'll start seeing weird behavior. This has generally been my experience ever since... any Ubuntu version, really. It's almost ironic that the most robust thing to update in practice is the OS kernel.
All I can say is that, based on everything I know, that's not the current experience of the majority of users, so it doesn't seem fair for you to generalize this to some architectural problem. I don't know if you unknowingly have some edge case setup or what.
Are you reading the same comments I'm writing? I was literally point-by-point saying the opposite of what you seem to have read me writing:
> You: All I can say is that, based on everything I know, that's not the current experience of the majority of users
>> Me: Yet from everyone's comments here I'm guessing I must be the only one who encounters this.
???
> You: It doesn't seem fair for you to generalize this to some architectural problem.
>> Me: I don't get why it happens in every instance, and there might be lots of different reasons in different instances. IPC mismatch is my best guess for a significant fraction of the incidents.
I have been running Ubuntu since 2004, and except for Firefox which tends to destabilize on update, I’ve observed this twice in 14 years; I update weekly or more often, and reboot every few months (usually on a kernel update I want to take hold)
Maybe it's because you update often so there are fewer changes in between? I update far less frequently.. it's not my primary OS so it's not like I'm even on it every day (or week). I use it whenever I need to.
Have you filed any bug report or point to one? All i have seen is a lot of handwaving about "IPC mismatch", things will not fix itself unless people actively report/help fix issues.
Here, on Arch, Firefox updates don't cause me any grief. Only time I've ever need to reboot is after a kernel or DKMS module update.
For systemd updates, I can just reload it. For the likes of core components like bash, and major DE updates, I can just lazily use loginctl to terminate all of my sessions, and start fresh.
I'm not sure why Firefox would be causing instability until you restart (reboot?), though.
Firefox with e10s enabled (all current releases) detects version differences between parent process and and a child process started at a later point in time. Until recently it aborted the entire browser when that happened. I think now they have some logic that tries to keep running with the already open processes and abandoning the incompatible child.
Ideally they'd just prefork a template process for children and open fds for everything they need, that way such a detection wouldn't be necessary.
Or you could explicitly design the package so that your pre/postinstall scripts ensure that you install to a separate directory, and rename-replace the old directory, so you can’t get half-finished updates.
Regarding the rest, if your code has incompatible API breaks between two patch or minor version changes, you’ll need to rethink your development model.
Ubuntu user here. Ubuntu is less stable than my second girlfriend, and she tried to stab me once.
Lately, every time my co-worker has updated Ubuntu, it has broken his system. He's like my canary in the coalmine. I wait for his system to not fall over before I will update mine.
Maybe its time to consider an OS with better maintainers, like Debian. I've had less issues on unstable/sid over the past few years than I had on the last Ubuntu LTS release (which was what spurred me to Debian). On my other machine, Debian Stretch (and Jessie prior to upgrading) have treated me well, there just isn't breakage when upgrading to the latest stable release or when applying security patches.
I chose Ubuntu because it was more widely supported by 3rd party software vendors and support companies than Debian. But this doesn't matter, because I still ran into hardware and software compatibility issues, and Ubuntu is more up to date than Debian, meaning Debian would have been even more broken by default.
I don't know of a single Linux distro that works out of the box with my laptops. Maybe if I bought a $2,000 laptop that shipped with Linux it would work. It would still be a pain in the ass to update, though.
I kind of hate Linux as a desktop now. I've been using it as such for 14 years, and it's only gotten worse.
I had very similar reasons for starting with Ubuntu, but when it came right down to it, all the software that I thought would only work on Ubuntu works just fine on Debian.
Hardware support wise, newer kernels generally come to Debian sooner too, as the latest stable kernel generally gets into Sid a week or so after release, then added to backports for Debian Stable after a few weeks. Currently you can nab 4.14 from backports on Debian Stable, and 4.15 should be coming down the pike shortly (seeing as its just a few days old).
Depends what you need ofcourse; a lot of people buy the newest and fastest but do not need it. Most (90%+) of my dev work works fine on an X220 which I can pick up for $80, has stellar linux support and still really good (14+ hour) battery life. Depends on the use case ofcourse, but when I see what most people around me do on their 2k+ laptops, they could have saved most of that. Also, Ubuntu Unity is just not very good; but Ubuntu or Debian with i3 are perfect. Cannot imagine a better desktop.
This is unfortunate. I've been using Linux and suffered, for the lack of a better word, with its warts since 2002.
There was a period between 2013-2016 where Linux was great as my main operating system. It was more stable than OS X and was much better for development.
Is hardware support your main issue with desktop Linux?
No, it's mostly software, but hardware is a big problem.
The software (especially Ubuntu's desktop) is lacking basic features from even 10 years ago. Maybe there's a way to get it to do what it used to do, but I can't figure it out, and I'm not going to research for two days to figure it out. I just live with a lack of functionality until I can replace this thing.
Not only that, but things are more complicated, with more subsystems applying more constraints (in the name of compatibility, or security, or whatever) that I never asked for and that constantly gets in my way. Just trying to control sound output and volume gives me headaches. Trying to get a new piece of software to work requires working out why some shitty subsystem is not letting the software work, even though it is installed correctly. Or whining about security problems. You installed the software, Ubuntu, don't fucking whine to me that there's a SELinux violation when I open my browser!
Hardware is a big problem because modern software requires more and more memory and compute cycles. All of my old, stable laptops can no longer perform the web browsing workloads they used to. Browsers just crash from lack of memory, or churn from too much processing. If you don't use modern browsers, pages just won't load.
Aside from the computing power issue, drivers are garbage. Ignoring the fact that some installers simply don't support the most modern hard disks, and UEFI stupidity, I can't get video to work half the time. When I can, there are artifacts everywhere, and I have to research for three days straight to decipher what mystical combination of graphics driver and firmware and display server and display configuration will give me working graphics. Virtually every new laptop for several years uses hybrid graphics, and you can't opt-out or you get artifacts or crashing. Even my wifi card causes corruption and system crashing, which I can barely control if I turn off all the features of the driver and set it to the lowest speed! Wifi!!! How do you screw that up, seriously?
Modern Linux is just a pain in the ass and I'm way too old to spend my life trying to make it work.
> I update Linux (Ubuntu) and, lo and behold, I can't really use any programs until I reboot
Which is almost true. In fact, you were unable to use programs that changed runtime dependencies or conflicted with current user sessions, init processes or kernel modules. You can often use other programs, but not ones that in any way touched the ones you upgraded, for one reason or another.
If you have to upgrade, say, a command line utility, that almost always doesn't require rebooting. If you have to upgrade a GUI app, or a tool that depends on some bastardized unholy subsystem designed to "secure desktop sessions", that may very well require relinquishing the session and restarting it. If you have to upgrade a tool used by your desktop (and if you have a complex desktop, that is literally thousands of programs), it's the same story, though you may even need to restart your desktop session manager or even your display server.
Then there's system init processes, kernel modules, firmware, system daemons and the like. You can reload those without rebooting, but it's certainly not easy - you will probably have to change to runlevel 1, which kills almost everything running. You can reload the kernel without rebooting, too - very handy for live patching - but really, why the hell would anyone want to do this unless they were afraid to power off their system?
So, technically, rebooting is not required to update in many cases in Linux, just like in Windows. But it is definitely the simplest and most reliable way.
> If you have to upgrade a GUI app, or a tool that depends on some bastardized unholy subsystem designed to "secure desktop sessions", that may very well require relinquishing the session and restarting it. If you have to upgrade a tool used by your desktop (and if you have a complex desktop, that is literally thousands of programs), it's the same story, though you may even need to restart your desktop session manager or even your display server.
Thanks, I'm glad at least one person agrees I'm not hallucinating. The vast majority of people here are telling me I'm basically the only one this happens to.
I've long since abandoned "bare metal" Linux in favor of VirtualBox and Windows for my home machine and VirtualBox and macOS on my laptop.
Monday's I merge last week's snapshot, take a new one, and run all my updates. Then I do my dev work in my VM. Before I head out on trips, I just ship the entire machine over the network to my MacBook Pro.
This is mostly because have you literally ever tried to install any Linux on laptops? It's always a Russian roulette with those $+#&ing Broadcom wireless chipsets. >.<
So you're not hallucinating. Linux as a desktop/laptop had a sweet spot from like...2012-ish till 2016. Then 802.11ac went mainstream so Broadcom released new chipsets and graphics cards had a whole thing with new drivers and Ubuntu's packagers (the people) lost their mind or something.
Nothing feels right, at least in Ubuntu/Arch land right now.
> I don't know about you, but it happens more often than I would like that I update Linux (Ubuntu) and, lo and behold, I can't really use any programs until I reboot. Sometimes the window rendering gets messed up, sometimes I get random error pop-ups, sometimes stuff just doesn't run.
This is less of an issue with Linux, per se, and more to do with proprietary video drivers.
I have multiple systems in my home with various GPUs. The systems running Intel and AMD GPUs with open source drivers don't have this problem. The two desktops with Nvidia GPUs have this problem whenever the Nvidia driver is updated.
I also had the same exact problem with my AMD system back when it was running the proprietary fglrx driver.
>This is less of an issue with Linux, per se, and more to do with proprietary video drivers.
Actually, it IS a problem with Linux. I don't get this behavior on my Windows or OSX machines where NVIDIA has been reliably (modulo obvious caveats) shipping "evil proprietary" drivers for a decade.
Linux is great, but it doesn't need to be coddled.
It's weird that you're blaming my operating system's problems on me.
No one is blaming you specifically, it is a common way of saying, “if you write operating systems, and you do $THING, you will get $RESULT.” Common, but wrong, which is why your high school English teacher will ding you for phrasing something that way.
Why would Linux need ‘defending’ for superior flexibility? The fact that files work like this is an advantage, not a disadvantage. I have never seen the flaw you’ve pointed out actually occurring in practice.
Well, it's not always an advantage. It's just the consequences of a different locking philosophy.
Windows patches are a much bigger pain in the ass to deal with on a month-to-month basis, but Linux patches can really bite you.
Example 1:
Say I have an application 1 that uses shared library X, and a application 2 that spawns an external process every 5 minutes that uses library X and communicates in some way with application 1. Now let's say that library X v2.0 and v2.1 are incompatible, and I need to apply an update.
On Windows, if I update this program, it will keep running until the system is rebooted. Updates, although they take significant time due to restarts, are essentially atomic. The update either applies to the entire system or none of the system. The system will continue to function in the unpatched state until after it reboots.
On Linux, it's possible for application 1 to continue to run with v2.0 of the shared library, while application 2 will load v2.1, and suddenly your applications stop working. You have to know that your security update is going to cause this breaking change and you need to deal with it immediately after applying the update.
Example 2:
A patch is released which, unbeknownst to you, causes your system to be configured in a non-bootable state.
On Windows, you'll find out immediately that your patch broke the system. It's likely (but not certain) to reboot again, roll back the patch, and return to the pre-patched state. In any event, you will know that the breaking patch is one that was in the most recently applied batch.
On Linux, you may not reboot for months. There may be dozens or hundreds of updates applied before you reboot your system and find that it's not in a bootable state, and you'll have no idea which patch has caused your issue. If you want your system in a known-working state, you'll have to restore it prior to the last system reboot. And God help you if you made any configuration changes or updates to applications that are not in your distro's repository.
No lie. After all nothing is stopping you from updating once every tuesday and rebooting after updates. You just wont have to do it 8 times in succession or stop in the middle of doing useful work to do so.
I just don't update nvidia or my kernel automatically and magically I only have to reboot less than once a month and always on my schedule.
I have! We had a log shipping daemon that wasn't always releasing its file handles properly and kept taking out applications due to out of spacing the box. That said, I drastically prefer the Unix behaviour.
It is a common tactic of Linux evangelists to state that they never have the problems you're experiencing and thereby disregard any criticism. You'll probably also get variants of "you're using the wrong distribution".
> What I expect it also means is that you'll get inconsistencies when doing inter-process communication, since they'll be using different libraries with potential mismatches. Is this correct?
Only if the libraries that use IPC have changed their wire format between versions, which would be a pretty bad practice, so I wouldn't expect that to happen often (if ever).
If something that's already running has its data files moved around or changed sufficiently, and it later tries to open (that is, the app was running but the data file wasn't open when the upgrade happened) what it thinks is an old data file, but is either new and different or just missing, that could cause problems.
> Because it seems to me that the Windows method might be less flexible but is likely to be more stable, since there's a single coherent global view of the file system at any given time.
In practice I've never had an issue with this (nearly 20 years using various Linux desktop and server distros). Upgrade-in-place is generally the norm, and most people will only reboot if there's a kernel update or an update to the init system.
*nixes have a system for handing off to the newer version such as kpatch and kgraft.
kgraft for example swaps off each syscall for a process while it is not being used. This lets the OS slowly transfer to the new kernel as it is running.
kpatch does it all in one go but locks up the system for a few milliseconds.
The version that is currently merged into 4.0+ kernels is a hybrid of the two developed by the authors of both systems.
Runtime kernel patching is a new thing. "*nixes" do not generally have these systems. Some proprietary technologies exist which enable specific platforms to use runtime kernel patching exist.
> What I expect it also means is that you'll get inconsistencies when doing inter-process communication, since they'll be using different libraries with potential mismatches.
In theory, but Linux systems tend to do very little IPC other than X11, pipelines, and IP-based communication, where the protocols tend to support running with different versions.
In practice you can achieve multi-year uptimes with systems until you get a mandatory kernel security update.
How can you ave a multi-year uptime unless you willfully ignore kernel security updates? In this day and age, year-long uptimes are an anti-pattern (if only because you cannot be sure whether your services are actually reboot-safe).
It's easy. You gather information about what the risks and hazards are for each vulnerability and then pragmatically decide whether there are any unacceptable risks after you mitigate with other layers of security.
It's a really common engineering task to do this and I'm not at all surprised that someone trying to maintain uptime would do so. Honestly it's more mature than updating every time because each change also introduces more potential for regression. If your goal is to run a stable system you want to avoid this unless the risk is outweighed.
But with "yum check-update" or the equivalent apt-get incantation saying you have dozens of security updates every week or two, reading the release notes for all of them and deciding which ones can be skipped safely in your environment is too much work. Far easier to just apply all updates every two weeks or monthly or whatever your schedule is, and then reboot.
Fully agree here; a lot (most?) of patches and updates are simply not exploitable in the respective server use case, so why should I incur risk of downtime to apply it?
If my system is closed to the public world, has a tiny amount of external services, and I am aware of the specific bug delta since system release and what mitigations may or may not be required, I can leave it running as long as I choose to accept the risk. Cute phrases like 'pattern' and 'anti-pattern' are rules of thumb, not absolute truths.
Kernel Live Patching (KLP) has been in mainline since 4.4. I've used it to patch various flaws in my Linux distribution since rebooting the running cluster is more tedious.
X11 (or other window-related tooling) was exactly what I was thinking of actually, because every time I do a major Linux (Ubuntu) update I can't really launch programs and use my computer normally until I reboot. It always gets finicky and IPC mismatch is the best explanation I can think of.
Are you sure this isn't more the Desktop Environment/Display Manager than X11? Or otherwise something to to with your use case?
I've primarily been using AwesomeWM for the last few years and occasionally XFCE (both on ArchLinux) and I cant recall ever experiencing what you describe.
I mean, OK, but if Windows's GUI (or PowerShell, or whatever) crashed or misbehaved upon update, would you be satisfied with "that's win32k/DWM/whatever being a poorly implemented mess; everything underneath should be solid"?
No. D-BUS is a travesty and blight upon the Linux desktop, and with systemd, every Linux system. It's the most fragile and nasty IPC system I've encountered. There are several better alternatives implemented by competent people, so there's really no excuse for its many defects.
Roger, I think we both know D-Bus has been rock stable at for the last 7 years, or possibly more, and Simon McVittie, its current maintainer, is a highly skilled and competent engineer.
While I find the situation different with systemd maintainers, whose communication style used to be questionable too often, and their software had some nasty bugs, I must admit that despite those problems they've also built software, which is nevertheless reliable, even though it took them a lot of time to come there.
This, honestly, is the most disappointing statement I read from you in the recent couple of years. And I'm saying this as a person who used to respect you a lot. I find your lack of respect together with the willingness to spread lies like this quite appalling.
> What I expect it also means is that you'll get inconsistencies when doing inter-process communication, since they'll be using different libraries with potential mismatches. Is this correct?
At the first glance this is true, but you can guard against this in several ways. If your process only forks children then it already inherits the loaded libraries from the parent as part of the forked address space. Alternatively you can pass open file descriptors between processes. Another option is to use file-system snapshots, at least if the filesystem supports them.
Yet another option is to not replace individual files but complete directories and swap them out via RENAME_EXCHANGE (an atomic swap, available since kernel 3.15). As long as the process keeps a handle on its original working directory it can keep working with the old version even if it has been replaced with a new one.
Some of those approaches are tricky, but if you want to guard against such inconsistencies at least it is possible. And if your IPC interfaces provide a stable API it shouldn't be necessary.
> And then there's FILE_SHARE_DELETE which allows deletion
That has some issues when the file is mmaped. If I recall correctly you can't replace it as long as a mapping is open.
> What I expect it also means is that you'll get inconsistencies when doing inter-process communication, since they'll be using different libraries with potential mismatches.
While this is true, I've never seen this to be a problem. If two programs use IPC, they usually use either stable, or compatible protocol.
To make things even more complicated, you can have two programs, either each in it's own container, or statically linked, or with their private bundles of libraries, doing IPC and then they are free to have different versions of the underlying libraries, while the users still expect them to work fine.
In principle, yes, IPC can fail between different versions of the same software. However, the chances that communication will fail between new version and some other utility are IMO much higher. A surprise comm failure between two copies of the same software (even different revisions) usually makes developers look pretty bad.
Some versions are known to be incompatible and most Linux distributions do a very good job of recommending and doing a restart of affected services in a way transparent to users. I have been running Linux and home and at work for years, almost never restart those workstations and, as far as I can tell, never had problems from piecemeal upgrades. My 2c.
Here is the simplest way I can put it: When you delete a file in NT, any NtCreateFile() on its name will fail with STATUS_DELETE_PENDING until the last handle is closed.[1] Unix will remove the name for this case and the name is re-usable for any number of unrelated future files.
[1] Note that is not the same as your "must be reachable via some path". It is literally inaccessible by name after delete. Try to access by name and you get STATUS_DELETE_PENDING. This is unrelated to the other misfeature of being able to block deletes by not including FILE_SHARE_DELETE.
"Reachable" doesn't mean "openable". Reachable just means there is a path that the system identifies the file with. There are files you cannot open but which are nevertheless reachable by path. Lots of reasons can exist for this and a pending delete is just one of them. Others can include having wrong permissions or being special files (e.g. hiberfil.sys or even $MFTMirr).
I would be kind of surprised if "the system" cares much about the name of a delete pending file. NT philosophy is to discard the name as soon as possible and work with handles. I was under the impression that ntfs.sys only has this behavior because older filesystems led everybody to expect it.
Well if you look at the scenario you described, I don't believe the parent folder can be deleted while the child is pending deletion. And if the system crashes, I'd expect the file to be there (but haven't tested). So the path components do have to be kept around somewhere...
It's true that NT won't let you remove a directory if a child has a handle open. But I suspect you are getting the reasoning backwards. The directory is not empty as long as that delete pending file is there. Remove this ill-conceived implementation detail (and it is that) then this and other problems go away.
There is also an API that retrieves a filename from a handle. I don't think it guarantees the name be usable though.
It's easy to imagine a system that works the way I would have it, because it exists: Unix. You can unlink and keep descriptors open. NT is very close to being there too, except for these goofy quirks which are kind of artificial.
this has led to some interesting observations for me in linux when I've had really large log files that were still in use and were "deleted" but the file was still in use. (I think cat /dev/nul > file will do this). Tools like du now cannot find where the disk usage actually is. Only on restart of the app does usage show correctly again. Kinda hard to troubleshoot if you were not aware this was what happened.
I agree that this is a drawback or a common gotcha for the Unix behavior which would be more user visible with the NT behavior, but to anyone advocating the Windows way I would ask: is it worth getting this fringe detail "right" by making every unlink(x); open(x, O_CREAT ...); into a risky behavior that may randomly fail depending on what another process is doing to x? On Windows, I have seen this type of pattern, a common one because most people aren't aware of this corner case, be the cause of seemingly random failures that would be rather inexplicable to most programmers. (Often the program holding x open is an AV product scanning it for viruses, meaning that any given user system might have a flurry of race condition causing filesystem activities that may or may not conflict with your process.)
> So you can't delete in-use files entirely because then they would be allocated disk space but unreachable via any path.
Isn't there a $MFT\\{file entry number} virtual directory that gives an alternate access path to each file? Wouldn't that qualify as "a way to access the file?"
Also, you might say that in practice Linux abides by the same rule - the old file can be referenced through some /proc/$pid/fd/$fd entry.
>What I expect it also means is that you'll get inconsistencies when doing inter-process communication, since they'll be using different libraries with potential mismatches.
That's why you should restart those programs that were using the library. You can find this out via `lsof`.
Really? Somehow procure a list of all libraries that were updated in a system update, go through each one, find out which program was using it, and kill that program? Every single time I update? You can't be serious.
In a case where one could get some sort of inconsistency because of different library versions, you restart the applications. That’s the point, that this can be handled by restarting the applications and not the entire operating system.
Yes, there are many ways to determine that on the command line. As far as a GUI, I couldn’t say as that isn’t how I do system administration.
I’m not sure what scenario you are envisioning. Usually upgrades are handled via the distribution and its package manager, and the maintainers take care of library issues. It’s not like windows where you go all over downloading packages from websites and installing them over each other.
Yes, but for some reason you want a GUI alert to tell you important things, which is a foreign concept to me. What I envision is knowing what is running on your server and updating programs purposefully, with knowledge of how they interact and what problems version inconsistency could cause.
You're the one proposing the enumeration of affected programs and services and restarting them as a solution!! How is using a GUI a foreign concept? Are you debating the merits of a GUI in 2018?
In any case, my assumption here is we're trying to help the user and give them an easy way to know what to do, instead of leaving their software in an undefined state.
Heck, on Windows I couldn't even rename audio/video files when playing them in an app, but I can on macOS, without anything crashing (except some stubborn Windows-logic apps like VLC which will fail to replay something from its playlist that has since been renamed, but at least on macOS it will still allow you to rename or move the files while they're being played.)
It's small details like these that make so much difference in daily convenience.
I think a general purpose UX should err on the side of "make it difficult for non-technical users to make mistakes"
Renaming/Deleting files in use is one of those things that us nerds like to complain about, but it makes sense when you think of an accountant that has an open spreadsheet and accidentally deletes a folder with that file. For average non-technical people (on any OS) I would say it makes sense to block that file from being deleted.
I see you’ve never actually experienced it? It is actually more intuitive for the average user, as the file name is updated across every application immediately. In fact, you can actually change the file name from the top of the window directly.
I have experienced this many times in MacOS after deleting a file/folder and replacing it with some other version, like one downloaded from an email. After a while, I realize that I'm working from ~/.Trash and the attachment I just sent didn't include the changes I had been making for the last hour.
I've had this happen in bash also, where I modify some script in an external editor then try to run it, only to realize that I'm running from the trash, even though the bash prompt naively tells me I'm in ~/SomethingNotTrashFolder.
Intuitive would be "Hey, this file you're working on was just moved to the trash. There's a 99.9999% chance you don't want to do this." rather than hoping the filepath is visible, and noticed by dumb chance, in the title bar, since not many people periodically glance at the file they have open to verify it's still the file they want open.
No idea, if the OS design plays into this or if it's just a application design convention, but on desktop Linux how it often works (for example with KDE programs) is that if a program has a file open which was moved (including moved to the trash), then the program will pop open a little persistent notification with a button offering to save the content that you still have in the program to the location where the file used to be, effectively allowing you to recover from such mistakes without hindering you from moving/deleting the file.
This doesn't fully explain why a reboot is not required on Linux. If a *nix operating system updates sysfile1.so and sysfile2.so in the way you describe, then there will be some time where the filename sysfile1.so refers to the new version of that file while sysfile2.so refers to the old version. A program that is started in this brief window will get mixed versions of these libraries. It is unlikely that all combinations of versions of libraries have been tested together, so you could end up running with untested and possibly incompatible versions of libraries.
> This doesn't fully explain why a reboot is not required on Linux.
Of course there is a theoretical possibility that this will happen; however, in practice, updates (especially security updates) on Linux happen with ABI compatible libraries. E.g. on debian/ubuntu
apt-get update && apt-get upgrade
Will generally only do ABI compatible updates, without installing additional packages (you need 'dist-upgrade' or 'full-upgrade' for that).
Some updates will go as far as to prevent a program restart while updating (by temporarily making the executable unavailable).
Firefox on Ubuntu is an outlier - an update will replace it with one that isn't ABI compatible. It detects this and encourages you to restart it.
All in all, it's not that a reboot is never required for linux theoretically - it is that practically, you MUST reboot only for a kernel update, and may occasionally need to restart other programs that have been updated (but are rarely forced too).
This generally should never happen, Linux distributions don’t wholesale replace shared objects with ABI incompatible versions - soname’s exist to protect against this very issue.
I had a program with a rarely reported bug that turned out to be lazy loading of .so files that was this bug. Switched to eager loading and it went away.
> On Windows, once the file is open, it is that filename that is open; You can't rename or delete it
It's simple for any application to open a file in Windows such that it will allow a rename or delete while open - set the FILE_SHARE_DELETE bit on the dwShareMode arg of the win32 CreateFile() function. In .NET, the same behaviour is exposed by File.Open / FileShare.Delete.
That is incorrect. It just depends on Windows how you call the Win32 API and what parameters you specify. Many options there - in the end it's just an object in the NT kernel space.
What is really irritating is when for example an update that only changes mshtml.dll requires a reboot because a program unnecessarily depends on it. These are not as common as it used to be though.
> Speaking of which, why do so many things require reboot to update on Windows?
Can't speak for everyone else, but Windows fully supports shared file-access which prevents the kind of file-locks which causes reboot requirements.
The problem is that the default file-share permissions in the common Windows APIs (unless you want to get verbose in your code) is that the opening process demands exclusive access and locking to the underlying file for the lifetime of that file-handle.
So unless the programmer takes the time to research that 1. these file-share permissions exists, 2. which permissions are appropriate for the use-cases they have in their code, and 3. how to apply these more lenient permissions in their code...
Unless all that, you get Windows programs which creates exclusive file-locks, which again causes reboot-requirements upon upgrades. Not surprising really.
In Linux/UNIX, the default seems to be the other way around: Full-sharing, unless locked down, and people seem prepared to write defensive code to lock-down only upon need, or have code prepare for worst-case scenarios.
Windows executables are opened with mandatory exclusive locking. So you can't overwrite a program or its DLLs while any instances of it are running. If a DLL is widely used, that makes it essentially impossible to update while the system is in use.
> Speaking of which, why do so many things require reboot to update on Windows?
We are getting there on Linux too - with atomic or image based updates of the underlying system. On servers you will (or already) have A/B partitions (or ostrees), on mobiles and IoT too, some desktops (looking at Fedora) also prefer reboot-update-reboot cycle, to prevent things like killing X while doing your update and leaving your machine in inconsistent state.
macOS also does system updates with reboot, for the same reasons.
It's a miracle that reboot-less updates mostly work in Linux. You need to restart services, etc. to make sure they have the latest libs. Gnome3+systemd does that now:
>macOS also does system updates with reboot, for the same reasons.
And speaking from experience, the recent macOS way of applying an update is absolutely insane - on my Macbook Pro (with the stock SSD with 3GB/sec read and 2GB/sec of write) a small system update can take 10+ minutes to install
I used to joke that Windows was alone in this issue, my work laptop being a prime example, but even Apple tends to towards reboots more often than not and especially as of late. Fortunately both can do it during slow periods; as in over night; and make updates nearly invisible to users.
So that's a fine question to ask, and you've received many fascinating answers, but can I just suggest that this case - that is, applying patches that relate to the security of your processor cache - is a very fine reason for requiring a reboot, since it will ensure that your processor cache starts out fresh and all behaviors that cause data to be placed there are correctly following the patched behavior.
The main reason is that in Windows executable files and dynamic libraries (.exe and .dll) are locked while a process is using them, while in other systems, e.g. Linux, you can delete them from disk. The only absolute need for reboot should be an OS kernel update (there are cases where a kernel could be updated/patched without a reboot).
I think a better question would be: why does Windows need multiple successive reboots?
Too often my experience can be resumed as: update-reboot-reboot-update continuing-reboot... ad nauseam.
At least on *nix, even when you need a reboot, once is enough.
To force a reinitialisation of all security contexts. Same reason that many websites make you log in again immediately after changing your password (which interestingly Windows doesn’t)
Historically (Windows 95 and earlier) reboots were required to reload DLLs and so on but that’s not really true anymore. Still a lot of installers and docs say to reboot when it’s not really necessary as a holdover from then
I was under the impression that the reason is what u/beagle3 mentions (in a sibling comment to yours): open system files. I'm curious to see your comment on what he describes, as what you mention (reloading some security context) does not seem to be the whole truth. That websites make one log in again after changing your password has nothing to do with this.
That websites make one log in again after changing your password has nothing to do with this.
No it is exactly the same principle: something has changed therefore invalidate all existing contexts. Far less error prone than trying to recompute them, what happens e.g. if a resource has already been accessed in a context that is now denied? Security 101.
I don't see how changing my password changes a "security context". I don't suddenly get more or fewer permissions.
As for logging other places out, that's a design choice. People change password either because they routinely change theirs (they either need to or choose to), or because of a (suspected) compromise. In the latter case you'll probably want to log everyone else out (though, who says you're logging out the attacker and not the legitimate user?) and in the former case you shouldn't (otherwise changing your password becomes annoying and avoided). The interface for changing the password could have a "log out all sessions" checkbox or it could just be a feature separate from changing your password.
No, it's not as simple as you put it. No need to condescendingly pass it off as "security 101".
Same thing happened to me this week-end and we are not alone [1]. The worst is that it's actually the second time on this computer (Kaby Lake i3 on a MB with Intel B250 Chipset). I had the same issue in December last year (exact same behaviour with probably an earlier version of that hotfix).
I'm running with Windows Update service disabled till this is fixed for good !
I have not been able to disable the update service. I'm supposed to be able to, but damn if I don't open my computer in the morning and see everything closed (and lock files all over the place) and all kinds of annoying shit like this.
I actually like Win 10, but it's shit like this that keeps me from becoming a true convert. Oh, for $X00 I can get enterprise update, but IMO that's just Win 10 home being used as ransomware. /rant
This actually made the system boot but there are some leftovers being installed on first boot that I've been unable to disable that also causes the system to be unable to boot.
So now, the machine is running but as soon as it is restarted we have to re-image the disk, go through the process of manually removing patches, and then pray that we don't have a power shortage as we'd have to do everything yet again on next boot.
I'm not convinced that this patch will solve the issue either, because if this updates requires a reboot the fix won't be installed if we can't boot. I might try to install this update from the recovery console see if that works.
Quite frustrating.