There's a huge difference between "zero testing" and "we didn't have this particular test case covered because the state space is massive". Lets say you even have "100% code coverage", do you really have a unique case for each possible condition in that query? Often times chaining operators like TFA give "deceptive" coverage results because they mark that whole code path as covered, without verifying the state space of the operands is covered fully. You can sometimes "cheat" coverage e.g. by using ternary operator assignment in place of if/else (often by mistake).
In the general case you are right, but in this specific case... this test case should have been covered. Even a rudimentary integration test would have caught this bug.
It sounds like the user story is:
- A user runs a query
- The user does not have any queries left in their plan
- They are billed.
- When the user runs the query again they are not billed a second time.
I'm struggling to imagine how the test would have failed to catch this. Maybe it was unit tested but with a database containing just one user? Maybe they got very unlucky and the correct user was randomly chosen?
Yeah, they might have just had unit tests with mocked db, or there wasn't enough combinatorial variety. It's insufficient to test the lack of double billing - you have to specifically induce a race condition, assert the race occurred, and that the interlock prevented double spend. It's tricky. Personally I'd try to engineer around race condition entirely, like some scheme with tokens and pagination.
I agree, this sort of thing seems like it should be extra-well covered, but, y'know, move fast and double charge folks.
You're demonstrating the opposite point perfectly. That test scenario would NOT have catched the issue.
The problem wasn't that requesters that should've gotten billed didn't.
The problem wasn't even that requesters that shouldn't have gotten billed did.[1]
The problem was that clients OTHER than the requester got billed.
You can, of course, write test cases that check your entire database for unintended state changes, but I struggle to find that a reasonable amount of effort. Especially since you'd have to do that for all code paths. That will very quickly cost a large multiple of the 73k this bug caused.
The adequate monitoring and quick response they did here is probably a very good trade-off. Like it or not, production is ALWAYS your last test. Issues are less costly if you realise that, than if you don't.
[1] Though for this specific bug, that scenario would've failed too, and might've triggered an extra look at the code.
> You can, of course, write test cases that check your entire database for unintended state changes, but I struggle to find that a reasonable amount of effort.
I think you're overthinking this. I'm not suggesting they should have looked for any unintentional state changes, we both agree that is overkill (until you start doing FP).
> Though for this specific bug, that scenario would've failed too, and might've triggered an extra look at the code.
Yes, exactly. For this specific bug even the simplest test would have failed, which would have caused someone to take a look at what was happening. You are correct that if the bug had been more complicated, such as causing both the proper user AND an additional random user to be charged, then it's unlikely a reasonable level of testing would have caught it.
> The adequate monitoring and quick response they did here is probably a very good trade-off.
We agree that there is a trade-off here, and if sacrificing some correctness is what it takes to win you a much higher velocity then they probably made the correct trade off; nobody died as a result of this bug.
But... surely you see there are some cheap steps they could have taken which would have caught this bug? Not all bugs, but this specific bug.
- Write integration tests for important behaviors, such as charging users!
- Make sure those integration tests run in an environment which closely simulates production.
> update your dependencies and ship it based on version numbers alone
The above is zero testing. Update your dependencies and ship it based on version numbers and integration tests is different. In that case as you suggested test coverage may easily have missed something, but there’s moving fast and there’s moving blindly and the second is just wasteful.
Dense in the sense of land to building use, not people per square foot. You have a 3-5,000 square foot home for a family of four on a quarter acre lot.
I'm not sure either, but during my time at Apple (one the richest and most advanced companies in the world) an additional $5-$6k was taken out of my paycheck if I wanted health care for me and my family. It's not like health care is automatically free for the working population.
Apple had full time OSM mappers as well as sponsored weekly mapping meetups where employees could volunteer to help with mapping. I’m not sure if they still do this. The OSM and cartographers I worked with at Apple were all very passionate about helping map remote parts of the world and the benefits mapping brings during disasters.
In the US in the 80’s and 90’s many kids were taught that the psychosis seen in many Vietnam soldiers after they’re return was due to psychedelic drugs and not a failed war, with little support, and lack of effort by the federal government.
I'd like to ask people who were kids in the 80s and 90s if they are aware how many Vietnam soldiers were addicted to heroin upon returning, and what kind of effect that had.
That was the most frustrating part about reading HN while working at Apple. Changing something like the touch-bar or keyboard are things that take enormous amount of redesign and engineering from both the product and manufacturing standpoint which take time.
Oh, dear, engineering people have to do engineering. The horrors.
The "broken" four years of Apple laptops (2016 to 2019) were quite a bit more frustrating to users.
- The keyboard was prone to failure at entering text - "You Had One Job!" This was an amazingly bad reversion to... I don't even know when, actually. People love and hate various styles of laptop keyboards, but it was exceedingly rare to hear that a keyboard fundamentally didn't function as a keyboard after some time of use. Ive's (I assume, given his known preferences) pursuit of Thin Uber Alles led to a fundamentally broken keyboard. Ok, not a huge issue if the keyboard is a cheap and easy fix, but...
- The keyboard was so integrated into the top case that the whole thing was unrepairable without literally replacing the whole top case, track pad, battery, etc. IIRC it was around $700 out of warranty, and while Apple kept extending out the keyboard repair issue window for a while, it doesn't change the fact that it was both disruptive for users and, apparently, quite expensive to Apple.
When I got a lightly used mid-2015 MBP in 2018 or so (oddly, the base model was still being made quite a while after it had been "replaced" in the consumer lineup), I figured it would be my last Apple laptop, because the replacements were clearly broken, and after three or four years of it, it was clear that the direction was set, and that you were typing on it wrong, or something of the sort.
I'm exceedingly glad to see that with the departure of Ive, some engineering sanity has returned to Apple, and the freshly redesigned M1 {Max,Pro} laptops seem to be a reversion to "That Which Works." A more standard keyboard actuation, and actual ports on the side. Woah...
Unfortunately, that said, I'm no longer using Apple products at the moment because the whole CSAM thing, on top of bowing to China regarding iCloud, and the questionable labor ethics involved have driven me off. I'm glad to see they're addressing repairability and such, but it was painful enough to rip myself free of that ecosystem (I'm currently using a Flip IV phone, a PineBook Pro, and some Kobos as my general use hardware - yes, they all have a lot of sharp edges) that I don't want to really dive back in unless I'm confident I won't have to exit it again in the near future. With the on-device scanning, in particular, "Well... we're delaying it... for a while..." is a very different claim from "Yeah, sorry, that was a bad idea and we're not going to do it." The second would be useful, the first implies that they're waiting until either a few more issues are resolved, or until people simply forget about the objections. Or it could imply that they're planning on the second, but just don't want to say it for some reason. I have no way of knowing.
It's been interesting, though. I so very badly want one of the M1 Max laptops, as it's literally everything I was looking for in a laptop, just... anymore, I'm too hesitant about Apple to actually buy one. And the alternatives for little ARM laptops all mostly suck... oh well. I didn't need to do high performance compute anyway.
I swear my Pinebook Pro has drawn more blood than any other computer device I've ever opened. You'd think I'd learn to be more careful after the third time the bottom shell sliced my finger open.
You can have working deep sleep, or audio that resumes after sleep. I've got a kernel patch that improves the state of that (the audio codec literally has no sleep/wake function in the 5.7/5.8 kernel), but I haven't applied it to the 5.8 kernel I run, so audio is... just broken.
Wifi works. Except when it doesn't. There's an issue with the firmware involving country codes and some 5Ghz frequencies, and I've not had the patience to track it down. Sometimes after sleep, wifi is just gone until a reboot.
The Kobos are fine, other than some random lag and reboots if you ask them to do too much. Large PDFs will choke an Aura badly.
The Flip IV... works, mostly, if you're not too picky, and don't care about things like seeing who all is in a group text. It's not unusable, but neither is it nice.
Etc.
The PBP has some legitimately sharp edges physically, though, too.
Because that's not how these systems work. Did the new keyboard allow for a flatter design to allow for more battery life?
Apple has always tried to improve computing. I've had MacBooks from the nineties and the keyboards on those are large and heavy. Is that where they should have stop with laptop design?
Did the new keyboard allow for a flatter design to allow for more battery life?
All it did was to increase Apple's repair cost because they had to support repairing these keyboards for many years.
> Apple has always tried to improve computing.
Then why go backwards with a bad keyboard, removing the Escape Key etc. Apple is not a scrappy startup where they don't have a team to test and give feedback.
