There is this ex-employee, telling some interesting stories. And regarding the downgrade... he was the one who did it for reasons he explains. I want to see if you are talking about different case or what.
> Question: There's the story online of that hacker who was pulling software images off through the door Ethernet port and found that his car's firmware was remotely downgraded after he uncovered and posted the first references to the P100 models.
> Answer: yup, i'm the guy that installed the older versions. this was a marketing mistake really. if i recall correctly, he ended up getting a marketing car or his car got tagged in the update system as a trusted car and he ended up getting pre-release stuff. this happened from time to time - sometimes marketing would sell off a car and the poo poo erp system wouldn't record the change. that car would then get prerelease and sometimes very broken firmware. i seem to recall another case where we just forgot to remove the prerelease materials from the official build, so all you had to do was look around.
Uh, that still doesn't paint Tesla in a good light, so this guy bought a car running special experimental firmware by mistake? What if there was a bug and he crashed?
then the firmware is downgraded again before anyone gets sued/fired or the hackers who hacked the manufacturer cover their tracks. duh.
one big reason why i'm not particularly thrilled at the prospect of one day owning a car that is actively managed/controllable by the manufacturer.
in the days of the audi 5000 sudden acceleration syndrome, they could take a production car and study it.
how would a third party/government agency trustlessly investigate "brake lose control" when the manufacturer has root and remote access on the vehicle?
> not particularly thrilled at the prospect of one day owning a car that is actively managed/controllable by the manufacturer
Ditto.
So many examples of companies shoving half-baked software updates down our throats... and worse... companies with half-baked security that are compromised with no consequences/little liability and effect the consumer.
i'd love to see any computer system that can provide a crytographically tamper proof log that comes with cryptographic proofs that:
a) a specific set of source code was used to generate the logs... so cryptographic security from source text files through compilation to execution on the hardware.
b) that specific set of software was run on a specific machine at a specific time.
it's hard... but beyond that, pretty much all audit logs are just security theater anyhow. sometimes can be mitigated by sending them to multiple places, but they're still meaningless unless you have a proof of what exactly generated them.
I don’t think they need to be tamper proof, but I do think they need to be held criminally liable if they are tampered with.
Volkswagen tampered with regulated measurements and got away with it for a while. But it was discovered and they paid a huge price for it.
With that precedent, it should create incentive to deal with mistakes transparently.
It does also mean transportation authorities will need to create some amount of leeway for mistakes. But I think there has been plenty of that given to traditional auto manufacturers.
that sort of inches at my point. how exactly would this criminal liability work? it would be comprised of two classes of evidence:
1) unassured/untrusted digital evidence that can be fabricated to say anything, leaving no trace of its previous state. (unless digital forensics has advanced well beyond what i know to be possible)
2) personal testimony that can also be fabricated.
i suppose what's new here is that in a digital world, bits can be set however one likes with no trace of their previous state, where physical evidence left behind physical manifestations of itself.
imagine rootkits that seek not to steal or hide their existence, but rather exist to plant fabricated evidence or hide actual evidence. in today's untrusted computer systems, this is all possible, and terrifying.
It’s weird, and almost a bit funny. For a brief moment.
The first smashing can be felt more than heard, a subsonic strike something like a vast drumhead being struck with a metre-wide mallet, but so quick, you barely even notice it until it’s over. The second one, however, isn’t far behind, and it’s a bit louder. That second thump gives away its location — whatever it was seems to be happening quite close by — in the direction of the parking structure.
At just this moment a car cruises through the pick-up zone at full speed, barreling along at least 100 kmh. It’s only because of some very fast reactions that no one gets hurt as it passes by. As it zooms past, you notice there’s no one behind the wheel.
Before you have any time to process that, another huge thump nearby causes a section of the barrier wall of an upper floor of the concrete parking structure to shear off. A pile of rubble falls to the ground not very far away from you.
But you know eventually someone will do a big over the air firmware update and brick a couple of hundred cars. This is almost certain to happen eventually.
There has already been a proof-of-concept hack of a vehicle: Hackers Remotely Attack a Jeep on the Highway. [0]
(Incidentally, I applaud the top comment at [0]. Everyone involved in recklessly demonstrating the hack on public roads, should have faced serious consequences. As far as I can tell though, nothing came from the police report.)
It's only if you aren't logged in to an account that word filters are applied (and some other things e.g. there are a bunch of subforums not visible to outsiders).
They know if they tried a lawsuit they risk their entire business model of after sales updates have to change. Many countries -including the entirety of the EU- require a car to get a new type approval/certificate of conformity if you change a car significantly. When Tesla significantly changes a car (install a completely new self-driving system for example) every single Tesla is instantly uncertified and illegal on roads if Tesla were forced to follow the letter of the law. At some point Tesla will come up against this in court but so far we haven't really seen much mention of it[1]. Going to court over said firmware though and it will very likely happen or if someone gets killed because of an accident caused by something newly implemented (like FSD updates after type approval) then Tesla is on the hook for this accident. They are playing with fire and treading carefully.
> Many countries -including the entirety of the EU- require a car to get a new type approval/certificate of conformity if you change a car significantly.
Do you have more information on this? The way you describe sounds like it's targeted at aftermarket mods more than software updates.
First thought about something similar would be the 737 MAX and the MCAS fiasco. I remember the whole "new type certification" was floated as solution to something like this from happening again.
Obviously different case, but I think the same principle applies here. A software update that significantly changes the performance characteristics should be recertified and not grandfathered in.
> Interestingly, some of these geofences do not seem to have a clear connection to SpaceX. While we will not disclose these locations here, I will say that the SNOW_RANCH looks like a nice location to play with development hardware.
Most likely these are testing locations. Possibly even second homes of testers & engineers. After all, this is a product that has very different operating parameters depending on location.
There's a "Snow Ranch" that is a working cattle ranch near Stockton California, where the owners allow model rocket launches. http://www.lunar.org/events.shtml
I wonder if that's what he's alluding to. I don't see an explicit connection to SpaceX, but it seems to fit.
I haven't been to snow ranch in years, but early Spacex investor Steve Jurvetson was a regular at the high power amateur rocket launches there. Probably other Spacex folks there as well.
Launches have all been shutdown for COVID reasons, and even before that, not infrequently cancelled due to fire risk.
Using ECC is very strange in this context. eMMC storage already provides for data correctness (and wear leveling of flash). If flash is corrupted to a point where internal error correction cannot compensate for it, eMMC will return no data, simply returning an error. This means that the additional level of error correction that they added will never ever be used.
Perhaps earlier revisions of this used raw NAND? Either that, or somebody got overzealous without thinking through.
Is the internal error correction done on the eMMC module? Perhaps they don't trust that, or are being wary of bit errors on the data lines.
It seems like a fair bit of effort for a device like this - I wonder if the same u-boot codebase is used on devices in space which have higher reliability requirements.
The spec is quite clear. Read shall either produce the same data as was written or return an error. I haven't ever seen an eMMC device that didn't comply with this part of the spec given how obviously bad that would end.
It is equally likely to happen during other parts of the process. The colonel is only a few megabytes. The file system is much larger and read many times. And then what?
Maybe they're worried about bit flips before the data gets to emmc?
They probably aren't doing it, but you could always interleave the data across the emmc blocks. A lost emmc block would therefore corrupt single-digit bytes across many ECC'ed blocks, rather than many bytes within a couple ECC'd blocks. 32 bytes of Reed Solomon should be able to correct up to 16 bytes of corruption. CDs do the same trick to be robust to scratches.
If I had to guess, it's likely shared code from SpaceX's other projects like Falcon 9, Dragon, and the Starlink satellites themselves, where bit flips can be much more common.
Been meaning to do this myself! Great to see it. :)
> While we would have to perform some more tests it appears that a full trusted boot chain (TF-A) is implemented from the early stage ROM bootloader all the way down to the Linux operating system.
This unfortunately means it will likely be somewhat difficult (or infeasible) to reflash it with a custom firmware that uses
actual GPS location for targeting of satellites but reports a couple km offset to the telemetry service APIs to keep my residence address somewhat private from my ISP.
It's a bummer they didn't share the dumps. It always bothers me when researchers act all coy about their results. Now I have to get my hands on a dish myself and do what they didn't (namely, actually publish the data).
In doing so they put themselves in a precarious legal position. The author has given you all they legally are allowed to. If you want a copy of the firmware, you now have all of the information you need to obtain your own copy.
Do you normally have the ability to hide your address from your ISP? As the first owner of my house, I needed to have them physically come out to run a new line that didn't exist before, and there is obviously no way to have a cable run to your house without telling the cable provider where your house is.
My terrestrial ISP fails open on identity check/verification, so I was able to give them a brand new alias and answer "none of the above" to all of the public records verification questions, and simply run 100% of the traffic out of the pipe over a VPN via a VPN router. They see nothing but ciphertext to a datacenter. This did require their largest-tier service deposit due to null credit history, but such is life.
To answer your question directly, no, they have the service address. But with no other data to link the service address to me, this is okay.
With starlink I am hoping to upgrade the privacy setup to +/-2km location fuzziness. I don't think I'll ever use an ISP without 100% of the last mile
traffic being VPN'd ever again.
How the heck did you manage to purchase the service anonymously? I had trouble even putting in service requests at first because my wife set up the account and didn't add me, so when I called, they wouldn't even talk to me because I couldn't prove I was the account owner. How do they bill you without knowing who you are?
CenturyLink's DSL signup process pushes you towards the "simple credit check" option, but if you refuse enough times they let you sign up for prepaid and do absolutely zero validation of any of the information you supply.
You can also sign up for a "business prepaid" account and they won't even ask for your name. They also don't ask what type of entity the business is, so there's nothing wrong with just making up a Doing Business As off the top of your head.
Frankly, this is how it ought to be. My personal data are none of their business.
I doubt this will work too well with Starlink in particular. The satellite handoff has to be pretty precise. Even without GPS, the antenna (on one side or the other) can probably measure your location within a kilometer or two, and as the satellites improve (larger satellite-side phased arrays), that will shrink.
Assuming that they're referring to decimal degree notation, and not Degrees-Minutes-Seconds, and that the location is correct, and ignoring the spheroid vs perfect sphere issues: about 11.1 meters (36.4 ft).
What avenues would they have to determine precise location if the dish reported its location 1km off, the delivery of CPE went to a different name/address, the service is paid for with a prepaid card, and 100% of the traffic to/from the device is encrypted at L4 to prevent usefulness of sniffing?
Multilateration. Can’t cheat RF time of arrival/flight. And a StarLink terminal misreporting it’s position and the constellation indicating such an anomaly sounds like a paddlin’ from StarLink.
They know the locations of their satellites extremely precisely, and they know the round-trip time from them to the dish. It basically is a positioning system on it's own.
Sure, but to do this would require firmware support (due to the timing requirements) in the satellites themselves, I believe. I doubt that's happening right now.
As bitrates increase, time of flight becomes more and more of an important quantity. Also, it’s likely Starlink satellites will develop ability to do time of flight for military reasons, ie as a backup to GPS.
The satellites are presumably themselves beamforming their transmissions to the dish to a high angular precision. If the satellites know their attitude precisely via star tracking and IMU, then a single satellite could be enough to geo-locate a dish.
I wouldn't be surprised if SpaceX tracks starlink satellite orbits via GPS, so that may limit their use as a redundant GPS system.
Now I'm curious what their command and control links look like. It's possible and perhaps likely that they can only communicate with any given satellite intermittently, as they fly overhead of ground stations.
That doesn't make any sense. There's WAY more money in commercial satellite internet than there is with niche US military contracts.
I don't understand why people are so committed to assuming there's a nefarious "real motivation" behind Starlink. Global telecomms is on the order of $2 trillion in revenue per year and growing. The US military's GPS budget is about $1.5 billion. So we're talking about addressable markets 3 orders of magnitude different in size.
Have you actually ordered Starlink service yet? I'm on the list having put in a deposit and they expect to start offering service here in September. I had to give them an address so they know where to send the satellite to. That gives them precise location.
Though sure, you could make the purchase using a pre-paid card under a fake identity so they can't associate the address with a person. At least not through their own records. House deeds are public record, so if you own the house, they can figure out who you are by making a public records request, which is generally one way refi spammers find you.
Plus, depending on where you are, the government itself might sell your name and address to third parties. I know Texas does this, which is why I put getting a Texas ID for such a long time and continued using California ID until Texas last year decided you can't vote with an out-of-state ID. So now the DMV is selling my identity and I'm getting a lot more spam.
Agreed. Even if I had a dish (waitlist) the odds of me being able to disassemble and reassemble it with my electronics butterfingers is near 0. I would really love to pick apart the firmware though. I've done similar work on firmware disassembly to find ways into devices.
I was literally just thinking about the inevitability of someone doing this yesterday. I don't have the chops but there would seem to be a bunch of cool possibilities for the dish hardware for the SDR crowd.
Sure, the beam might be greater than 1km FWHM, but you’ll lose gain if you’re off too much, so they may be able to triangulate better than that just off of angle alone. It’s also possible they may do time of flight measurements (as a backup to GPS… which is relevant especially for the military side).
Sure, GPS SDR Sim[1] works just fine. You will want to be in an RF chamber of some kind not only to prevent the terminal from seeing natural GPS signals, but also to prevent you from screwing up the GPS in nearby satnav systems. Also because broadcasting on those bands on public airwaves is illegal as a private citizen.
Of course putting your satellite antenna inside of a RF chamber also prevents it from working, so this may not be a viable long term strategy. Plus the terminal is undoubtedly using the GPS coordinates to calculate the antenna steering profile so you won't be able to lock on if your GPS is wrong. But since all they want to do is enable access to dump the firmware this probably isn't an issue.
From working on hardware with GPS-functionality tacked on before, I can suggest a simpler solution;
1: Find the GPS module, and look up its data sheet.
2: Spoof the data coming out of its IO ports. Cheap GNSS modules that spit out NMEA messages on a serial line are everywhere. (I guess because they're super cheap, and easy to integrate)
Spoofing GPS might be dangerous should the dish detect coarsely its position also from the IP satellite link. If it does, then having the incoming data telling one position and the GPS a very different one, would likely trigger some protection.
I'd have to imagine in this case it's using the GPS location to assist in acquiring and tracking the satellites (though that's entirely a guess based on the "auto-adjusting" that's claimed). Spoofing your GPS location like that may work as far as bypassing the geofence, but you may not get internet at the same time.
Right, if the UT has a mistaken idea of its position, it won't find the satellites that it is looking for in orbit, and simply not work. Alternatively, if it DID find satellites, then it will know at least what cell it is in (how big are these?) regardless of the spoofed GPS fix.
I'm not sure the dish can continue to work if it doesn't have a real GPS lock. That said, this is a mechanism that they found on the dish side in the firmware - firmware that is unencrypted stored on that flash chip - so you can obviously manipulate the firmware side to ignore the debug fuse stuff.
> so you can obviously manipulate the firmware side to ignore the debug fuse stuff.
Might not be as trivial as that makes it sound:
"Continuing through the boot process we can see that U-Boot loads a kernel, ramdisk and Flattened Device Tree (FDT) from a Flattened uImage Tree (FIT) image that is stored on an embedded MultiMediaCard (eMMC).
We can also see that the integrity (SHA256) and authenticity (RSA 2048) of the kernel, ramdisk and FDT is being checked. While we would have to perform some more tests it appears that a full trusted boot chain (TF-A) is implemented from the early stage ROM bootloader all the way down to the Linux operating system."
Yeah. If you can futz with the hardware, pretty much any security can be circumvented.
Desoldering a SOC and replacing it with something similar enough but different in its trusted boot config is somewhat less trivial than "manipulate the firmware" though, at least in my opinion...
I have no info about the UT's use of GPS, but given that you can basically construct a practically-unbreakable geofence around the fact that the UT can only physically see a specific set of satellite(s) at any particular point in time (and space), and you get this property for free given the UT's purpose, I can't imagine that not being leveraged in the design.
It appears that after they discovered the ECC encoding, they simply ignored the ECC data to extract the image. What if some deliberate (correctable) bit errors were scattered through the image? They have the code that implements the ECC algorithm. If I were them, I would have used it and perhaps even submitted a patch to binwalk so it would automatically recognize/decode the image.
Also, now that they have the image, they could try to override the geofence/fuse protections by running it on an SoC without the fuse blown, and a SDR-based GPS spoofer. Seems like a fun endeavor.
I might be mistaken, but the geofencing might be based on the specific satellite you were connecting to, correlated with the time of day. And very likely verified/audit-logged on the satellite side too.
Looks like a phased array [0], which is probably a smart idea with a dish like that. Instead of using a parabolic reflector with a receiver at the focus (like in a normal satellite dish) they use an array of a ton of tiny receivers (each of those tiny ICs would be a driver for a small, on-PCB antenna). Phased arrays (essentially algorithmically delaying the individual radio signals from/to each driver on the scale of fractions of a period of the carrier frequency) let you do really precise beamforming and aiming, but take a lot of processing power and a lot of antennae to be efficient so weren't practical until recently.
It's a phased array, it uses those many little chips to do "beam steering". Check out this video by The Signal Path (a Bell Labs expert!) doing a teardown and explaning parts of it: https://www.youtube.com/watch?v=h6MfM8EFkGg
It is a phased array, the closest thing that exists to it in the rest of the world is the phased array radar in the nose radome of an air superiority fighter jet.
Phased arrays are everywhere. An incredibly common example are many cellular base station antennas. They are fancy but nothing particularly special these days. Active phased arrays, e.g. steerable with either analog phase / delay shifters or with digital front ends are less common but still much more common than the BAE Systems phased arrays in F35's.
There are speaker companies that sell phased array loudspeakers for tuning the sound in e.g. a specific corner of a concert hall, where you can't just put extra speakers anywhere you want.
Hmm, I wonder if locking down the boot chain like this is GPL compliant, since apparently even GPLv2 requires the ability to modify GPL parts on the device.
> That article mentions nothing about code signing
From the article:
"... it appears that a full trusted boot chain (TF-A) is implemented from the early stage ROM bootloader all the way down to the Linux operating system."
> the GPL 2 doesn't have anything that would prevent Tivoization.
From the GPLv2:
"... plus the scripts used to control compilation and installation of the executable."
From the Software Freedom Conservancy blog post about the GPLv2 installation requirements:
"GPLv2 assures, to the purchaser of an embedded product, their absolute right to receive the information necessary to install a modified version of the GPLv2'd works."
Of course, that doesn't mean that the proprietary software must continue to function after modifying the GPL software, apparently even the GPLv3 allows what Tivo did. From a presentation about GPLv3 on cars:
"Ironically, even if Linux were GPLv3, Tivo’s method of crypto-
lock-down would likely comply with GPLv3."
"Tivoization" is a word because GPL2 does not say you must be given signing keys that allow your modified image to be accepted and executed by the device, these are not covered by "scripts used to control compilation and installation of the executable".
GPLv2 requires the practical ability to modify GPL software on the device, whether or not signing keys are mentioned explicitly is irrelevant. I suggest you re-read the Conservancy blog post I linked above and scan through the GPLv3 in cars presentation.
I knew the GPLv3 Installation Instructions requirements were narrow (enough that I don't think it's actually a barrier to, say, bundling GPL code with an iOS app), but "anti-TiVo so weak it wouldn't stop TiVo" is a TIL.
TBH despite being a GPL zealot I think it is a fairly reasonable compromise to allow proprietary software to delete itself or stop itself from working if a user updates the GPL software. OTOH, the LGPL does not allow that, you have to allow re-linking with the proprietary code, so there is precedent the other way too.
IIRC the issue with GPL and iOS was that the GPL was/is incompatible with the App Store ToS.
Tesla responded by disabling the car's ethernet port, downgrading the firmware, and preventing the car from receiving future upgrades to software.