I hope that the Rust devs end up doing the "right thing", which from this article seems like going back and fixing Rc, instead of just marking mem::forget as safe. They're still pre-1.0, and anything they do now they will be stuck with for a long, long time, especially if Rust succeeds as much as I hope it does.
Delaying 1.0 by a few weeks may seem like a big deal, but ultimately it is a self-imposed deadline. It's great to have those, but following them dogmatically might not be the best strategy. In cases like this, I generally lean towards slowing down and doing things right: otherwise you will pay the price ten times over later.
(That said, while I understand this issue, I don't know very much about the context in the Rust community, so I'm not actually sure that mem::forget should be unsafe. It was just the impression from the article and from previous Rust code I've read/written.)
There isn't an acceptable way to fix Rc. It wouldn't be hard to require it not to have cycles, but cyclic data is actually a major usecase for Rc. A cycle collector wouldn't fix any of the unsafety here unless it was run on literally every drop, which would be far too slow to be acceptable in a systems language. An opt-in trait to prevent leaking specific types would be practical, but the implementation details aren't settled and it can be done backwards-compatibly, so there's a strong feeling that it shouldn't block the release. Any other attempts to determine whether cyclic data structures always eventually drop their contents enter very-active-research territory and would be more like a ten year than a few week thing.
I don't know enough about how Rc is used to know if this is an acceptable possibility, I just wanted to mention it in case it is possibly helpful.
I developed a reference-counting system in my library "upb" that handles cycles and never leaks objects. It is precise for no-longer-mutable objects (objects which may no longer change the set of other refcounted objects pointed to) and less precise for mutable objects.
The basic idea is to reference-count groups of objects instead of single objects, and define the groups such that no reference cycle spans groups.
If you introduce a link between two refcounted objects A -> B, then A and B's groups are merged. Now refs/unrefs of A or B ref/unref the (merged) group. Nothing in the group is freed until both A and B's refcounts both fall to zero. This conservative group-merging whenever you create a link ensures that no reference cycle spans groups.
This is imprecise and relatively wasteful if the group grows really large. But you can make it totally precise for any subgraph of refcounted objects that you're willing to freeze. At freeze time, compute strongly-connected components, and each SCC becomes a refcounted group. For any frozen subgraphs, the refcounting is totally precise. And unfrozen objects can reference frozen ones (but not the other way around).
If your application ends up freezing most of the graph, this works great. Or if you're ok with the collection being pretty coarse for your groups of objects, it also works great. If you have an application that can't freeze the graph and wants pretty precise collection, this doesn't work so well.
More info about my scheme is in comments in these headers:
btw, I never did this in my library because I didn't need to, but you could also support an operation that computes SCC's to minimize the groups without freezing the graph.
The minimized groups would still merge with other groups if you created links between them. But this would be a way to make the refcounting more precise without strictly requiring that subgraphs become immutable.
There's a reason why finalizers aren't a widely used technique in Java and why Python's cycle detector simply stacks things with destructors in an out-of-the-way location.
Those languages have bigger problems than Rust here. Rust can still recover correct behavior for APIs like scoped because it can ensure (using lifetimes) that the relevant value never escapes the stack frame. In the languages you mention (and most other languages with a "with" or "IDisposable" equivalent), the "close" equivalent, or even the finalizer, can be called on objects that still have valid references to them hidden in another object or thread.
I was actually speaking of the original finalize method it Java[1]. There's no guarantee that it will be called, there's no guarantee when it will be called if it is, and as a result it shouldn't be used to reclaim resources or much of anything, really. Where "should" is so strong in this case that I've never seen it in the wild, even given the N-monkeys code that I have seen.
I'd prefer the `Leak` based solution too, but it's going to cause a _lot_ of churn. 1.0 has already been delayed often, every time it gets planned and then not followed through on because the date is fuzzy and ignorable. This time they've set a concrete no-nonsense date which we should follow through on IMO. We already had a second alpha.
I guess it's better that way than saying something broken is "ok" only because "I promised myself that I'll finish until next Monday". It's more like Ubisoft-style, or something, except deadlines really matter for them, because money depend on it, and they cannot just "move the deadline" because of all the advertising must be timed, people tend to buy games and go to cinema more on specific dates, etc.
For project like Rust deadline doesn't actually matter that much, grasping for it is almost stupid. The only thing that actually can suffer from breaking it is self-esteem, which should't worry a reasonable man much. But magic "1.0" number does matter a bit more than just deadline, because after it there's no "breaking changes".
So I'd be much happier if Rust wouldn't reach 1.0 for the next 2 years, but would actually become satisfying instead. Somehow "non-stable but working" is better suited for making software than "broken and stable". We have plenty of "broken and stable" out there already.
Rust is not seeking pure programming perfection, it is seeking to be useful and to fulfill its goals of safe systems programming, and it is fulfilling these goals with flying colors. There's absolutely no reason for this to push the release date. There was a bug in a stdlib API, and that issue was fixed weeks ago. Trying to pivot the language by taking a fundamental feature back to the drawing board for nebulous benefit would be utter foolishness at this point.
Please see my comment below (https://news.ycombinator.com/item?id=9447938). If this were a problem for Rust's basic memory safety guarantees, we'd certainly consider delaying release. There are also ways to pursue the `Leak` approach in the future backwards-compatibly.
Please see my comment below (https://news.ycombinator.com/item?id=9447938). There is no risk to Rust's basic safety guarantees here, and there are several ways forward (including introducing things like `Leak` backwards-compatibly later on).
It's very disappointing that the Rust community has opted to go back and change the definition of "safety" instead of actually tackling this problem head-on, all due to a self-imposed deadline there is no need to actually meet.
Rust's basic guarantee has been, and continues to be: unless code uses `unsafe`, it is guaranteeed to be memory safe. The issue under discussion doesn't change that -- it's largely about what such `unsafe` code is allowed to do and assume. I've laid this out in a bit more detail in a comment below (https://news.ycombinator.com/item?id=9447938)
There is a general problem with combining destructors and automatic collection of reference cycles, and its one that several languages have unexpectedly found themselves facing.
For example, Java has the function System.runFinalizersOnExit, but if you look at the documentation you'll see that it now says, "Deprecated. This method is inherently unsafe. It may result in finalizers being called on live objects while other threads are concurrently manipulating those objects, resulting in erratic behavior or deadlock." http://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.h...
Boehm's key points are: (1) When a destructor on an object O runs normally, anything that O points to is still alive (because of the reference from O), but when a cycle is collected, not everyone can go first: that is, all but one destructors in the cycle have to run after one or more of their references are dead. It is hard to write destructor code that is safe in all cases. (2) Any destructor that needs to update a concurrently accessed data structure has to take a lock, but destructors on cycles run asynchronously with respect to the rest of the program and so an unlucky timing leads to deadlock.
If this problem was better appreciated then language designers wouldn't go down the rabbit hole of trying to figure out how to make destructors work together with automatic collection of reference cycles, and instead try the alternative approach of providing mechanisms for the program to run destructors synchronously.
So you cannot safely assume your destructors will be called, and you need to proactively take steps like this in case they aren't. Nothing in the natural use of the language cues you to do this, so you need a moral guide like "Effective C++" for Rust.
This was the kind of accidental complexity that Rust had hoped to avoid. It seems like Rust has avoided a lot of it. But I guess it was entirely too optimistic to hope Rust could avoid all of it.
Note that this is only something you really need to think about when writing `unsafe` code, which basically unlocks a whole new set of rules that you need to worry about (zero-sized types in offsets, manual allocations, safety boundaries, GEP/aliasing rules, uninit memory, double destructors, etc). This is a drop in the bucket that anyone venturing to use `unsafe` will have to face in some kind of "guide to unsafety".
The problem is that all of this is safe code. The problem, to my mind, is that you can write mem::forget (as in the code here's safe_forget)* without ever using the unsafe keyword*.
Edit: I was wrong. The problem isn't with forget. This is hairy.
Which, as you can see, uses Rc, which uses unsafe code in a way that leads to the soundness bug. So that's not strictly true, or rather, doesn't really change anything, as it relies on the same bug.
"1. There were other ways to forget even before Rc, at least in some cases. For example, if T:Send holds, you could send the value to a thread that runs an infinite loop, or which is deadlocked on a port.
"2. It is true that one cannot assume that a destructor will run, and hence that forget is not itself unsafe (rather, it is unsafe to write a dtor that must run)...."
And alexcrichton:
"I commented on #24292, but the gist is that there are multiple ways to leak memory today (e.g. #14875 and #16135), so a targeted solution at Rc may not cover all use cases. Although as I mention in #24292 these other bugs can also be considered separate bugs on their own which need to be fixed regardless (but sometimes is quite difficult to do so)."
I especially liked dgrunwald's comment:
"A safe mem::forget has the advantage that it makes it easier to write the counterexample proving thread::JoinGuard unsafe. Safe mem::forget makes it more likely that people will know that destructors are not realiable, so they can avoid repeating this mistake."
But then, "Put a big freaking spike in the middle of the steering wheel and get rid of the airbags and seat belts" has always appealed to me as an automotive safety approach.
I think you need to split the unsoundness bug, which happens because thread::scoped uses unsafe code, from the "you can write mem::forget in safe code" bug, which is arguably not a bug but is at least an enormous footgun waiting to happen.
When the footgun goes off in unsafe code in the standard library, you get use-after-free and memory unsoundness. When the footgun goes off in safe code written by mere humans, you leak arbitrary resources.
The fact that you can write mem::forget in safe code is nothing more than an argument that mem::forget itself should not be marked as unsafe.
The fact remains that as long as you are not using unsafe code yourself, you don't really have to worry about anything. The category of destructors that leave the world in an unsafe state if not run is AFAIK a strict subset of the category of destructors that require unsafe code to be written.
The problem here looks like it is with Rc, which does require unsafe code. I haven't checked, but I don't think the destructors in the thread::scoped issue are unsafe. Even if they are, I can easily imagine another situation where a programmer assuming a safe destructor is always called causes a bug, which is exactly the problem with thread::scoped.
And Rc only looks like it is the problem. Rc is doing exactly what it says it does, as safely as it can. The fact that you can use it to avoid having a destructor called is a side-effect of it doing what it's supposed to do.
If main is safe, mem::forget is safe (remember, Rc is just playing the part of mem::forget), and the destructor for JoinGuard is safe (which it is; I just checked), where is the unsafe code?
`unsafe` is a stateful property, not a local one. The specific invariant here is that you cannot rely on destructors for memory safety, which the old thread::scoped API was doing. The actual unsafe code is actually here: https://github.com/rust-lang/rust/blob/master/src/libstd/thr...
(I myself have been guilty of this same error, of assuming that `unsafe` regardless of its position cannot have far-reaching effects on the overall state of your code.)
The important thing is this: using safe Rust, and using any safe stdlib API, memory unsafety is impossible. This includes stdlib APIs that use `unsafe` internally, such as `Rc`. Proving those interfaces safe is the burden of the Rust developers, and if you can show that there has been an error in these proofs then it is a drop-everything sky-is-falling defcon-1 situation that they must address, up to and including unreleasing the language if no suitable solution came to mind (fortunately, this is quite unlikely).
In the thread spawning. The destructor itself isn't unsafe, but it was being used as a guard to enforce the safety of the threading code.
> I can easily imagine another situation where a programmer assuming a safe destructor is always called causes a bug, which is exactly the problem with thread::scoped.
There's a difference between "a bug" and "memory unsafety". If the bug is that the value the destructor guards isn't released (for example, if a File is leaked, the associated file descriptor will never be flushed and closed), then that's not Rust's problem. The programmer probably wanted that file descriptor to be closed, but there's no memory safety violations going on there, and in fact nothing worse than simply stuffing that File into a global array/hashtable and forgetting about it.
The problem with thread::scoped is actually fairly unique. The destructor is not doing much besides just calling join() on the forked thread. The problem is that the JoinGuard object doesn't actually control any access to anything at all, it exists purely as a proxy to represent any values (particularly stack values) that are borrowed by the forked thread. Because it's only a proxy, leaking the guard does not actually make the guarded values inaccessible.
I can't think of any situation besides multithreading that has this same problem. Every other situation involving a RAII object with a lifetime being leaked (and therefore outliving its lifetime, which is technically a violation of borrowck) by definition makes the object inaccessible, which means the data it's protecting cannot be reached. For example, a struct that contains a &-ref to some value may outlive the referenced value if the struct is leaked, but this is still memory-safe because the reference will never be dereferenced. If the reference can be dereferenced, that means code exists that has visibility on the struct, which means the struct hasn't actually been leaked yet, and as long as it hasn't been leaked it's still safe (e.g. because borrowck will still enforce the lifetimes).
Multithreading is special because the forked thread is what's actually accessing the values, without ever going through the RAII object. This means that borrowck cannot guarantee that the RAII object is still alive (and therefore that the "protected" values are still valid). This was considered to be ok as the RAII object would join the thread (therefore ensuring it had finished executing) before destructing, but the ability to leak objects means the join may not happen.
Ultimately, this means that JoinGuard is the moral equivalent of a Mutex that coexists with its protected data rather than owning it (thereby making it possible to write code that accesses the data without holding the lock). It's a bit better than that, because you have to actually go to some effort to leak the JoinGuard as opposed to just doing it accidentally, but the point is that borrowck cannot guarantee that the code that accesses the data does not outlive the guard.
And all of this comes back to my original point, which is that I can't think of another situation where it's actually unsafe to leak the RAII guard (as opposed to merely being a bug) without there being `unsafe` code in the guard itself. Note that the example given for PPYP, Vec::drain_range, uses `unsafe` code in its destructor.
A quick observation from an outsider: The article goes into a lot of depth, seems highly technical, well crafted and researched but the whole "pre-poop your pants" thing just ruins all that and makes it looks childish. I understand that not everything about development has to be serious, but I also think that aligning form and content is the surest way to make sure your message comes across.
I'll reuse my reply on /r/rust to defend the naming:
> I like the naming and feeling of "Pre-Pooping Your Pants" pattern. We are doing the intentional leak only to be (hopefully) collected later, which may not happen but it still remains memory-safe even in that case. Still this is not very desirable and the name clearly indicates so.
A term for code written in an undesirable way to tie you over in the short-term is a "hack".
Not all hacks are bad. But, if I see "// this is a hack to..." or "# this is a hack to..." or similar in code, that is a signal to me that the developer (1) either didn't have time to perform the right thing but could have with more time (within their lifetime), or (2) could not feasibly perform the right thing under constraints given (within their lifetime).
Over the past ~8 years, "hack"'s use has switched back to a much more positive sense- possibly because of HN, or possibly because people read Eric Raymond http://www.catb.org/esr/faqs/hacker-howto.html, but "hack" != "hacker". The negative sense is still used and understood by most developers, and it comes from two things: "HACK... HACK... HACK...", like the sound of an ax attempting to chop down a tree (brute force), and the other definition of a hack not used as much these days, which is "work for hire especially with mediocre professional standards".
I think we should at least consider PPYP to be a subclass of "hack", or perhaps just use the word "hack" instead of PPYP.
I think Pre-Poop Your Pants is a really clever way to package the core idea. I agree that the article is well crafted, and I don't think it's childish to talk about poop in this context.
You have the right to your own feelings, and I agree that the title may be off putting. The audience for the article is rust core contributors and rust fans, and they're a very informal bunch. If rusters are like similar communities I know, talking about poop makes a blog post more likely to spread.
EDIT: I do not want to argue with Dewey2, so I'm putting this here: the women in the rust community that feel uncomfortable when they are addressed as "guys" have every right to feel that way, and I commend the rust community on making that stand.
There's nothing to suggest all other rust devs like this way of presenting it. I don't. I agree it's off-putting even if the issue itself is interesting.
Since you're so interested in my activity here[1], you may notice that my very first, apparently terrible and flame-inducing post was about me pointing out that an article spelled someone's name wrong[2], modulo transliteration. Now you might argue that I was wrong, but do you really think that was such a terrible, unproductive and antagonistic post? The next mass of downvotes came when I replied sarcastically to someone who wrote that my concerns/arguments were "stupid" and not worthwhile. That provocative post towards me was, of course, not downvoted. My follow-up to my original, downvoted, post was at first well-recieved[3], but then creeped down to 1 again for some reason.
I'm not a victim by any means, and I'm sometimes an asshole. But maybe you should stop taking Karma at face value and stop believing that this community is beyond reproach. If you want to go through my history - which on this account is quite short - at least do it better.
[1] And your profile says nothing about you being a mod.
> Your current posting style didn't work for your last account. It doesn't seem to be working for this account.
Yes, I was an asshole on my "Dewie" account, and recently on Dewie2 as well. But go over the posts I presented again - was I really being an asshole? Or did I just have opinions and priorities that people were in such disagreement with that they chose to use the "I don't like this - Downvote!"-artillery? Come on. At least address my specific points when I go to the trouble of typing them out. Instead of just playing Dr. Phil.
> It doesn't seem to be working for this account.
Yes. Thanks for underlining that for the powers that be. Well, it's not like I was hiding so who am I to complain.
> Do you think this is working for you?
Clearly, my problem is being a glutton for punishment in that I keep coming back to this poisonous community. :) It's not like the assholes on this forum are helping me be less of an asshole. (And no, you're not one of the assholes, just speaking of a subset here.)
Because "guys" is gender discrimination? Give me a break. Might as well call ~everyone who speaks the Spanish language massive sexists while you're at it.
I guess it is if you're oversensitive to everything that might be construed as sexist, through any historical, cultural etc. link. Like some people are oversensitive about bodily functions.
Besides, some groups of people might be put off by such crass analogies. You don't want to exclude those people, do you Ygg2?
Yeah, no, the default male plural pronoun in other languages is a language feature. One doesn't feel like they're being called male when it's used. On the other hand "guys" sounds like you're being called, well, a guy (and English has singular they and many other constructs that can be used in its place).
Usage of "guys" makes many women feel they don't belong. It perpetuates ideas about programming being a boys club that shouldn't exist.
Poop jokes might make some people uncomfortable, but they're not as harmful as "guys".
Plus, if you read kibwen's comment below that, it's not something that's considered a serious offence, but you will be asked to stop. The same would probably go for excessive poop jokes in IRC or whatever.
(Also, it's an unofficial blog post. Not the same standards. There wouldn't be poop jokes on an official blog post, and there wouldn't be much of a ruckus if someone used "guys" in an unofficial one)
I dunno, I'll say "hi guys" to groups of women as well as mixed or male groups. I've also heard girls say "hi guys" to a mixed group.
I've also heard people say things like "hey up A" where A is in
["guys and gals", "guys and girls", "everyone", "mothafuckas", "nerds", "friends", "all you", "people", etc.]
Coming form a very formal understanding of language, I can see how it could be taken as a reference to only the men, but coming from my practical experience, "guys" has been used as a gender neutral way of referencing a group without issue.
Now that I think of it, this really is a context sensitive issue. For example, saying "hi guys" to a group of 100 with only one or two women in it might give some listeners the impression that there are no women, thus perpetuating the "boys club" view, whereas saying "hi guys" to an evenly mixed group might not give the listeners that impression.
> I guess the Rust community isn't so carefree and non-sensitive after all.
Carefree? Mostly. Non-sensitive? Pretty much the opposite. Rust community is extremely sensitive. The rules are lax, but they draw the line at making contributors uncomfortable.
In IRC, the poop jokes and calling people guys, would have the same chance of drawing ire, which isn't huge. The whole "guys" thing is just a smidge more likely to draw ire. However if you are insistent on either accounts expect to get ostracized.
> I guess they don't feel like they belong in any part of mixed gender society then, since it is so widespread.
Well, yes. It's not like women make on average less than their male peers.
-------------------
Let me show an abriged version of what usually happens in IRC.
Anon: Hi guys,
Mod: We prefer the term guys and gals :)
Anon: Hi guys and gals*
Anonhole: Hi guys
Mod: We prefer the term guys and gals :)
Anonhole: I know.
Mod: We have a term of conduct, please read it.
Anonhole: I won't
Mod: *ban* Anonhole
While it is certainly informal to say "guys," I have heard both men and women use it in a definitely gender-neutral sense for my entire life, including in workplaces, and informality is not impolite in an informal setting like an IRC channel. I wasn't sure if I was crazy here so I asked a couple of women I know and they were as surprised as I was. By contrast, nobody uses the word "gals" - it sounds incredibly awkward and lame. Perhaps the use of the word "guys" is regional, like "y'all" (sadly, it seems like English does not have any universally established pronoun for this). It's sad that you are proud of treating people who are new to Rust by permanently excluding people and labeling them as assholes for using idioms which are considered perfectly courteous and gender-neutral in many places. Given that, banning people just for using the word "guys" smacks of finding excuses to harass and ban people more than offering a welcoming community. If this is "what usually happens in IRC" then that might be a clue that there are dialect differences you aren't recognizing?
Insofar as you are representing the Rust community via HN, you've given an overwhelming impression that the Rust community is an unpleasant place where "outsiders" are totally unwelcome. As someone who considered adopting Rust prior to reading your post, that's a problem because with such an immature language I would eventually have to interact with that community and then I would apparently have to deal with people randomly nitpicking which dialect of English I learned and calling me an asshole. No thanks, life is too short.
I'm sorry I gave that impressions, but I'm not really member of Rust community. I'm just a guy writing Rust code and occasionally checking out the channel. I'm much more Java than Rust guy :P
Also you missed the point of that exchange, you are excluded from the Rust community, only if you act against Code of Conduct actively. The titular bad example was given a warning, he disregarded it several times and continued to behave against community wishes.
What do you expect happens once you act against what other members of the club urge you not to do? A single mistake/slip up isn't a problem. Repeated offense and disregard of CoC is.
Usually the custom is to greet each other with "Hi Rustaceans". It's gender neutral and has no negative connotation.
If this gives you impression of a treehouse filled with cultist, who am I to judge.
"Guys" is not gender neutral for everyone. You might find it that way, and I used to, but many women don't.
Re:IRC: You're conflating different things here. The IRC incedent was a sexual comment. Strict nono. "guys" is something that will be frowned upon. Sexual jokes are also a nono. Just today someone made a penis joke in IRC, and was similarly told to stop. Poop jokes -- childish, maybe, and certainly frowned on by some, but not considered harmful.
I don't see anything wrong with arbitrarily deciding. You're free to not play in the sandbox if you don't like the rules. The community likes those rules. And fwiw the rules don't disallow "guys". Again, you'll just be asked politely to not do that.
As I understand, "guys" wasn't a huge problem but the discussion that followed. It got ugly quick. I'm pretty sure even Spanish has the variant of "guys and gals".
What I want is besides the point. It's a club. It has rules. I'm Ok with them.
I usually go with "Hi peeps!" (Yes, I imagine everyone on Rust IRC is a talking marshfellow).
Spanish nouns have two genders, and a corresponding plural for a noun of that genders. When you have a mixed group, both chicos (boys) and chicas (girls) you use the masuline plural (chicos).
I've also noticed that male genitalia are used to mean something positive (es la polla, tener cojones)
and female genitalia are generally used to be negative (un coñazo, coño)
You're okay with the rules because you live in the clubhouse. The fact that you are pulling up the ladder to keep other people out unless they roll over and pee themselves to acknowledge your dominance speaks volumes about what life is like inside the clubhouse.
Nope. I'm just a guy passing through that knows the customs and didn't enter the house with his shoes on (well, there was that one time). Rust community in my experience only ever censored people that were actively rude for a long time - I'm talking months if not years.
Anyway, if Rust community did allow such transgressions, people would be up in arms because they are promoting Linus brand of verbal abuse...
I tend to keep him updated with rust news, so that when the day comes, I've laid enough groundwork to be able to do something in it without having to explain what it is and why etc.
And part of that is showing flaws, and how they're resolved/worked around and so on.
The entire infantile humour / brogrammer / hipster / drama phenomenon. It started in Ruby (_why, Zed Shaw, GitHub, etc) then moved into JS as Node and CoffeeScript became popular with Rails users, now it's in Rust (again via Ruby).
I don't think it's always the same people or that it has anything to do with Ruby itself, it just seems like there's a certain culture that first grew in the Ruby / Rails community.
This post is the first infantile thing in this vein I can remember seeing related to Rust, and, as far as I know, the author has no connection to the Ruby community you describe, they're just choosing poor titles of their own accord (I agree with Steve that the title detracts).
But the problem here is complex and hard to solve. Cycles and memory leaks are nigh impossible to solve completely without sacrificing on that altar ability to create custom data structures or no-GC by default. Which was non-starter for Rust.
Well, I'm still not quite proficient with Rust, so maybe my feelings are misguided, but the whole situation seems really terrifying to me. Like, way too terrifying than that we could just make a point about "not doing like that again" and calmly move on to the 1.0.
After all, why Rust? Because we're all tired of problems with C++ and stuff. And the point isn't really about making a language that would be safe "as often as possible", it would be quite terrible goal, actually. The point is, I believe, to make it nearly impossible to write unsafe code accidentally, without even noticing it.
And again, maybe it's just me, but what I understood from that post is totally non-intuitive to me. I don't really understand what are the guidelines and how many more gotchas like that there could be. "Not to mess with unsafe code" is not really good guideline, because it turns out that it can be not quite obvious that the code is unsafe.
Please see my comment below (https://news.ycombinator.com/item?id=9447938) -- a key point is that, as always, you have to write `unsafe` to get memory unsafety in Rust, period. This is Rust's core guarantee. The issue being reported here is a safety problem caused by a particular use of `unsafe` blocks. The debate is largely about what unsafe blocks should or shouldn't be able to assume.
Well, the thing is, until Rust is formally tested, as in a subset of Rust is proven in CoQ to be safe, you don't have any guarantees.
I assume much of it is safe, with few probably hidden caveats, hidden in more complex logic. In C++ writing code that causes this behavior is trivial, here you need to jump through a lot of hoops. So it offers you comparatively more safety, but this kind of bugs are potential gold mine for crackers.
Anyway core developers are on the scene deciding how to tackle this issue.
Oh, sure, but let's be reasonable. If it's something like the notorious sort problem in Java, I'd say it's ok. It's preferable when things are simple enough that somebody with a good understanding of the whole system can make assumptions about its safety without using CoQ. If things are complicated enough that we actually need Agda/CoQ/Idris to feel safe — seems that things went really wrong at some point.
> but this kind of bugs are potential gold mine for crackers
Exactly! And I'm just saying that a problem which is "not very likely to trigger, but is very hard to find if happens" is far, far more dangerous than a problem which is "easy for a novice to miss, but every experienced developer would notice".
> If things are complicated enough that we actually need Agda/CoQ/Idris to
feel safe — seems that things went really wrong at some point.
I disagree. Complex software/hardware is by its nature full of bugs. What Rust set out to do is complex and it's bound to have bugs, but doesn't mean it's reason to abandon it, no more than we should discard LHC or ITER because they had issues starting up.
Whether it's a wrong invariant on TimSort or a really obscure multi-threading case, or a weird glitch in time library, its still an error. Each one is exploitable.
ALL software could use some coverage/tests and theorem proving. I just wish Rust started with a proven safe subset.
> Exactly! And I'm just saying that a problem which is "not very likely to
trigger, but is very hard to find if happens" is far,
far more dangerous than a problem which is "easy for a
novice to miss, but every experienced developer would notice".
Again, I disagree. The matter of security is one about making the assailant less likely to breach your software. If you take two comparable pieces of code, the one with more holes will take less effort to break.
If your language offers fewer avenues of exploit, makes these exploits more expensive. A rare bug is worth more than a run of the mill bug. Which prohibits the number of assailants which could purchase it, which reduces number of potential attackers.
(
people getting mad about PPYP is really a victory, though; the second last bullet was the most important:
> It's vaguely incoherent and meaningless on its own: this is a property inherited from its connection to the venerable Resource Acquisition Is Initialization (RAII) pattern.
I'm mad about RAII and not going to take it anymore!
)
Putting stuff in a HashMap that you'll never ask for again is a kind of leak. Allocating stuff on the stack and then looping forever is a kind of leak. The case where you put it on the heap and forget to ever free it is a specific kind of leak that people seem particularly terrified of for whatever reason.
I too am particularly terrified of this kind of leak. The reason is that it is the only kind of leak mentioned that is not clearly a programmer mistake.
C++ has a whole class of "mistakes" that one cannot be aware of simply by reading the language specification. Common ones such as writing a constructor and destructor but not a copy-constructor are now compiler warnings, but many will pass under the radar. As a result, to write correct code one needs to be aware of multiple conventions and rules and patterns that are external to the language. When reviewing code one needs to do more than observe that it compiles and accomplishes its stated task, there are probably several organizational standards that need to be satisfied as well to be safe.
This is precisely the kind of thing that Rust was supposed to solve. The goal was to have a language that didn't require an "Effective <xyz>" book to be memorized to do code review. A language where any rule like "You should always free resources in advance in case the destructor doesn't run" is captured statically by a compiler error.
In Rust, you cannot put something on the heap and "forget" to ever free it. Rust doesn't expose any mechanism in safe code to heap-allocate something that won't automatically free it for you when that thing goes out of scope (the primary way to do this is with the Box type).
As evidenced by this whole kerfuffle, you can in fact leak heap-allocated values by constructing an Rc cycle (or taking advantage of various implementation bugs in other things), or by using mem::forget() (which I believe is still marked as unsafe but seems likely to change). But none of these approaches could be characterized as "forget[ting] to ever free it", because they're not a consequence of "oops I forgot to do that".
> A language where any rule like "You should always free resources in advance in case the destructor doesn't run" is captured statically by a compiler error.
This only shows up if you're specifically opting into unsafe code by using the "unsafe" keyword. Rust doesn't (yet) try to ensure safety of unsafe code, because it's unsafe. As long as you don't step into unsafe code, you can ignore this entire article.
Which unsafe code is that? In the article, I don't see any use of "unsafe" in safe_forget[1] or either main[2] in the first part of the article.
[1] Which creates a reference-counted cycle containing the thing-to-be-forgotten, ensuring the finalizer is never called.
[2] The first main simply demonstrates that the destructor is never called, which is ok in itself but causes problems when you're doing smart things in the finalizers, as in the second main, where the code uses the destructors to ensure the threads are terminated and the programmer is assuming the borrow checker is going to warn them if the threads aren't terminated.
Right. That block is unsafe because it's doing unsafe things. (ptr::read and alloc::heap::deallocate)
But the use of Rc is kind of a red herring; if mem::forget is marked safe, then you don't need safe_forget because you can forget things (fail to execute their destructors, specifically) safely anyway.
The problem is that destructors aren't guaranteed; the bugs in the thready thing (and potentially unbounded other things) are symptoms. Drop needs big, red warning signs.
The power of a notation, and a type system, is in what it lets you not think about. The fact that destructors may not be called is, unfortunately, something you have to think about.
Well, it's worth noting that this error happened because unsafe library code relied on a property that wasn't actually true. The language itself is working just fine, Rust cannot and will not prevent all bugs.
Scrapping the guarantee that destructors will be called is tantamount to scrapping RAII. thread::scoped made the assumption that destructors were called before a borrowed value was returned, and it led to a memory-safety error. If you don't use unsafe, you won't get memory-safety errors, but you can get plenty of other nastiness.
The guarantee that destructors are called when values leave scope is fantastic. It means that one can leverage memory-safety into arbitrary-resource-acquisition-safety. It violates the principle of least astonishment in a brutal way if Rust's borrow checker can statically guarantee that a value no longer exists, but can't know whether or not its destructor has been called.
Most important, this API problem does not break Rust's basic safety guarantee. That is, you must write unsafe code to violate memory safety.
If Rust's basic memory safety guarantee were at risk, the core team would absolutely consider delaying the release, or doing whatever else was needed to address it! We must never allow memory safety violations in safe code, no matter how obscure the bug that leads to them.
So this is a question of writing "safe" APIs that hide uses of `unsafe`. What precise assumptions are you allowed to make within such code?
As it is, we are taking this API issue very seriously, and making sure that we explore all of the options available. We have already yanked the affected APIs, while we determine the best way forward.
There are known ways to safely re-introduce the (relatively few) places in the standard library where unsafe code was using this pattern. Gankro talks about one in the post; you can see a proposal here (https://github.com/rust-lang/rfcs/pull/1084) for the `scoped` API.
The basic issue here is about a tension between a couple of different APIs, as Gankro explained in the post. In particular:
- `Rc` has been part of Rust for a long, long time, and is a fundamental systems programming tool.
- APIs like `scoped` and `drain_range` would like to use RAII for ergonomics and consistency, but this is either not possible (`scoped`) or subtle (`drain_range`) given `Rc`. In either case, these APIs use `unsafe` internally, so it's all about what that unsafe code can assume.
Furthermore, there is not a strong consensus about how best to resolve these issues, even if we had all the time in the world. Currently there are at least three camps:
- Stick with the status quo (perhaps marking `mem::forget` safe to reflect that reality). Safe code can assume no leakage, but unsafe code needs to take extra care (and sometimes avoid the RAII pattern). This is already the case for a large number of other properties, as Gankro points out. Leakage in safe code can never lead to memory unsafety; it is just a vanilla bug, and Rust doesn't prevent you from making bugs.
- Introduce something like `Leak`, meaning that you have to explicitly ask for a given type to be guaranteed not to leak through things like `Rc` cycles. While that allows you to write APIs like `scoped` and `drain_range` easily using RAII, you need to be aware of the marker, and make sure to use it for such types. Worse, though, is the interaction with trait objects: depending on the design, you may have to write `Box<MyTrait + Leak>` to be able to store the trait object behind an `Rc`, and that `Leak` bound needs to be present for the entire chain of APIs leading up to that point.
- Restrict `Rc` in some way, perhaps to `'static` data, thereby ruling out (a class of) leaks for all types. It's not completely clear how much fallout this would involve. The compiler currently relies on non-`'static` reference cells, and this change would likely force channels to use a `'static` bound as well, thereby defeating much of the purpose of the `scoped` API in the first place.
Finally, it's worth noting that at least one version of the `Leak` proposal can be added backwards-compatibly, later on, so there is potentially plenty of time to explore that approach if we feel the complexity is worth it.
I believe that Niko Matsakis (also on the core team) is planning to write a blog post explaining all of the above in much greater detail.
Largely unrelated question: In the code about a third of the way into the article, there is,
impl<'a> Drop for Foo<'a> {
fn drop(&mut self) {
*self.0 += 1;
}
}
What does the line "self.0 += 1;" do? Specifically, the ".0"? Foo is a basically a reference to an i32, so I would have thought that would be "self += 1;".
Delaying 1.0 by a few weeks may seem like a big deal, but ultimately it is a self-imposed deadline. It's great to have those, but following them dogmatically might not be the best strategy. In cases like this, I generally lean towards slowing down and doing things right: otherwise you will pay the price ten times over later.
(That said, while I understand this issue, I don't know very much about the context in the Rust community, so I'm not actually sure that mem::forget should be unsafe. It was just the impression from the article and from previous Rust code I've read/written.)