Ken, do you think in some future it might be feasible for a hobbyist (even if just a very advanced one like you) to do some sort of precise x-ray imaging that would obviate the need to destructively dismantle the chip? For a chip of that vintage, I mean.

Obviously that's not an issue for 8086 or 6502, since there are more than plenty around. But if there were ever for example an engineering sample appearing, it would be incredibly interesting to know what might have changed. But if it's the only one, dissecting it could go very wrong and you lose both the chip and the insight it could have given.[1]

Also in terms of footnotes, I always meant to ask: I think they make sense as footnotes, but unlike footnotes in a book or paper (or in this short comment), I cannot just let my eyes jump down and back up, which interrupts flow a little. I've seen at least one website having footnotes on the side, i.e. in the margin next to the text that they apply to. Maybe with a little JS or CSS to fully unveil then. Would that work?

[1] Case in point, from that very errata, I don't know how rare 8086s with (C)1978 are, but it's conceivable they could be rare enough that dissolving them to compare the bugfix area isn't desirable.

I don't know about a hobbyist, but are you talking about something along the lines of "Ptychographic X-ray Laminography"? [0] [1]

[0] https://spectrum.ieee.org/xray-tech-lays-chip-secrets-bare

[1] https://www.nature.com/articles/s41928-019-0309-z

They were using 6.2keV x-rays from the Swiss Light Source, a multi-billion Euro scientific installation. 6.2keV isn't especially energetic by x-ray standards, (a tungsten target x-ray tube will do ten times that) so either they needed monochromacy or the high flux you can only get from a building-sized synchrotron. Given that the paper says they poured 76 million Grays of ionizing radiation into an area of 90,000 cubic micrometers over the course of 60 hours suggests the latter. (A fatal dose of whole-body radiation to a human is about 5 Grays. This is not a tomography technique that will ever be applied to a living subject, though there is some interesting things no doubt being done right now to frozen bacteria or viruses.)

"Though the group tested the technique on a chip made using a 16-nanometer process technology, it will be able to comfortably handle those made using the new 7-nanometer process technology, where the minimum distance between metal lines is around 35 to 40 nanometers."

The discernible feature size required for imaging CPUs of 6502 or even 8086 vintage is much lower than that. The latter was apparently made with a 3 µm process, so barring any differences in process naming since, that's about 3 orders of magnitude less.

Plus, as Ken's article says, the 8086 has only one metal layer instead of a dozen.

So my (still ignorant) hope is that you maybe don't need a synchrotron for that.

Although it's unclear whether the very first revisions of the 8088 (not 8086) with this bug ended up in IBM PCs, since that would be a few years before its introduction, the original PC and successors have the ability to disable NMI in the external logic via an I/O port.

Even if you could disable the NMI with external circuitry, that workaround quickly becomes untenable, especially when it's not fully synchronous.

Psrset was the command, there is very little info on it since HP gave up on HPUX.. I thought it was a good solution for some problems.

They even sent a bunch of us to a red hat kernel internals class to see if Linux had something comparable.

https://www.unix.com/man-page/hpux/1m/psrset/

https://wiki.osdev.org/NMI

Thre's also another set of NMI control bits in port 61h on an AT. It's worth noting that the port 70h and 61h controls are still there in the 2022 Intel 600-series chipsets, almost 40 years later; look at pages "959" and "960" of this: https://cdrdv2.intel.com/v1/dl/getContent/710279?fileName=71...

I get it. And i think i might know why: we are all smart here. Smart enough to do outstandingly well at high school. As time goes on: university and career the selection bias reveals itself. We are now with millions of peers and statistically some of them will be well ahead by some measure. In short: the feeling is to be expected, and it is by design of modern life and high interconnectedness

sometimes in the too often contrarian comment sections, it’s jarring to see someone’s comment recognize the human being on the other end. it’s inspiring.

i’m not the op, but your comment was kinda rad. thanks.

But the previous post, about the bootstrap drivers gave me a chance. I had an hour to spare, the size of the circuit was small enough, so I just stared at it on my cell phone, scrolling back and forth, back and forth. It took a while, but slowly I started picking up some basic understanding. Big shoutout to Ken, BTW, he did everything to make it easy for mortals to follow along.

I am still clearly at the beginner level, and surely can't redo that post on my own, but there is a path forward to learn this.

If I'd redo that post, I'd advise to have pen and paper ready, and print out the circuit a few times to doodle upon them. And have lots of time to focus.

In the series he builds all the things more or less from the group up. And if that isn't enough he in another video series builds his breadboard VGA graphics card

kens: were you able to identify the original bug? I understand the errata well enough, it implies that the processor state is indeterminate between the segment assignment and subsequent update to SP. But this isn't a 2/386 with segment selectors; on the 8086, SS and SP are just plain registers. At least conceptually, whenever you access memory the processor does an implicit shift/add with the relevant segment; there's no intermediate state.

Unless there is? Was there a special optimization that pre-baked the segment offset into SP, maybe? Would love to hear about anything you saw.

In the overwhelmingly common situation where your stack started with an SP of 0, you just assign SP back to 0 before setting SS and nothing can go wrong (you are, after all, about to move off this stack and don't care about its contents). Even the weirdest setups can be handled with a quick thunk through an SP that overlaps in the same 64k region as the target.

It's a clumsy interface but hardly something that Intel has traditionally shied away from (I mean, good grief, fast forward a half decade and they'll have inflicted protected mode transitions on us). I'm just surprised they felt this was worth a hardware patch.

I guess I was hoping for something juicier.

Consider the case when the MSDOS command interpreter is handing off execution to a user program. It can't just set SP to 0 first (well, really 0xFFFF since the stack grows down), because it's somewhere several frames deep in its own execution stack, an execution stack which will hopefully resume when the user program exits.

Programs using the 'tiny' memory model, like all COM files, have SS == CS == DS so new_ss:old_sp could easily clobber the code or data from the loaded file.

Edit, just because all the folks jumping in to downvote and tell me I'm wrong are getting under my skin, this (untested, obviously) sequence should work to safely set SP to zero in an interrupt-safe way, all it requires is that you have enough space under your current stack to align SP down to a 16 bit boundary and fit 16 bytes and one interrupt frame below that.

Again, you wouldn't use this in production OS code, you'd just make sure there was a frame of space at the top of your stack or be otherwise able to recover from an interrupt. But you absolutely can do this.

     mov %sp, %ax  // Align SP down to a segment boundary
     and %ax, ~0xf
     mov %ax, %sp
    loop:          // Iteratively:
     mov %ss, %ax  // + decrement the segment (safe)
     dec %ax
     mov %ax, %ss
     mov %sp, %ax  // + increment the stack pointer by the same amount (safe)
     add %ax, 16
     mov %ax, %sp
     jnz loop      // + Until SP is zero (and points to the same spot!)

That’s a lot to put up with, and in my mind qualifies as a design bug with no palpable software workaround. Not great for wanting folks to use your brand new CPU.

The protected mode transitions that you mentioned are much more palpable because they only happen in very bespoke parts of the OS kernel, usually only in bootstrap in any non-ancient OS.

Uh, yes, yes it is. That's the point. Go check the 80286 PRM for examples where the same hardware vendor inflicted far worse protocols on their users only a few years after the hack we're looking at.

The subject under discussion isn't "whether or not the 8086 implemented a good stack switching interface". Clearly it did not. It's "whether the 8086 stack switching interface as designed was so irreparably broken that it required a hardware patch[1] to fix". I think the case there is pretty weak, honestly, given the state of the industry at the time. And I found it surprising that Intel would have bothered.

[1] And increase interrupt latency!

And then the 286 came along at a time where the x86 architecture was already enormously successful, part of the ubiquitous IBM PC. The 8086 on the other hand was a much simpler, completely new CPU competing with many others. There aren’t many warts in the 8086 itself, whose data sheet is a few pages instead of a book (segmentation was a plus at the time because it allowed a large address space and some easy multitasking without relocation in such a simple CPU), and still it actually only won out against other CPUs for marginal reasons. If 68k had been ready for the PC, that would likely have been it.

Intel had no capital at the time to inflict such an insanity to people. And why is patching the mask in a later revision of the chip such a big deal anyway?[1]

Just look at the solution you proposed. Always gradually adjusting SS and SP in a loop until SP reaches 0, just because an NMI may hit. That’s untenable.

[1] Interrupt latency is only slightly increased on MOV/POP SS. That shouldn’t happen so much that the much more often CLI/STI blocks with many instructions in them aren’t a bigger concern. And the 8086 wasn’t exactly low latency anyway.

Motorola, when faced with a critical bug that would affect OS implementors in a way they could technically work around (c.f. Apollo, who used two chips) but didn't want to, didn't even bother to fix the thing and made them buy the upgraded version (obviously in some sense the '010 was the equivalent "fix it in the mask" thing). They were selling chips that couldn't restart a hardware fault well into the late 80's.

> Just look at the solution you proposed.

That wasn't the solution I proposed. I proposed just having the stack reserve an interrupt frame at the top and setting SP to zero before changing SS. The code I showed was just proof that even putative users[1] who couldn't change the OS would be able to work around it.

[1] There were none. This was several years before MS-DOS; no one was shipping 8086 hardware in any significant quantities, certainly not to a market with binary software compatibility concerns.

Like what?

> Motorola, when faced with a critical bug [...] (c.f. Apollo, who used two chips)

I am well aware of the Apollo two chip hack, but I was not aware that this was considered a "critical bug". I thought the original 68000 just did not support virtual memory with paging, and the 68010 introduced that. Given that you needed an external MMU if you wanted to do any kind of virtual memory, which most clients did not care about, and that the 8086 and many if not most contemporaries in that class did not support any kind of virtual memory either, that seemed like a reasonable limitation to me.

The kicker is, I actually own a Siemens PC-D, a marvelous machine that has an 80186 (the differences to an 8086 don't matter here), and an actual external MMU made up of discrete 74 series logic chips. It runs SINIX, and has full memory protection through that MMU.

And yet it does not support paging either, because by the time the external circuitry recognized a fault, the CPU had already moved on further. The only thing SINIX did was kill the faulting process.

> while forgetting the interrupt state bug that forced the introduction of the 68010

Are we actually talking about the same thing then? The Apollo hack I know was because after a bus error, not an interrupt, you could not restart the faulting instruction, presumably because by the time the fault was recognized, the CPU had already continued executing an indeterminate amount. This does not matter for regular interrupts, since they are not synchronized to running code and do not restart instructions. It would have been a pretty useless chip otherwise, and yet it thrived even after its descendant was introduced.

The point is, it seemed to me Apollo really wanted to use the 68000 for something it did not support (yet), and other users of the 68000 were not affected (most did not even have the MMU). It was a marvelous, unique hack, that I've actually never seen fully documented or confirmed anywhere (please let me know if you have). The PC would have been amongst the ones not affected, and the chip even spun off variants long after the 68010 was released, not having paged virtual memory just wasn't a dealbreaker for most.

> That wasn't the solution I proposed. I proposed just having the stack reserve an interrupt frame at the top and setting SP to zero before changing SS. The code I showed was just proof that even putative users[1] who couldn't change the OS would be able to work around it.

Yes, you proposed every single piece of code switching a stack, not just OS code, to do some careful dance whenever it switches the stack, on top of making sure that every stack in existence has the proper amount of unused space at its base to support that.

And then all interrupting code would have to immediately switch to its own stack (which, while required today, wasn't necessary back then, mainly because all code ran with all privileges and plenty of stack). While taking care that all registers can be properly restored on the way back, of course. Or only the NMI handler, but then you would have to additionally have to save and restore the interrupt flag around your stack switching.

But okay, let's agree to disagree then. You say there is a reasonable software workaround, I say there is not, for what to me was a pretty clear design bug. Albeit one that was also easily fixed. In my mind, Intel did the sane thing back then by simply patching it up. Can't say that it hurt them.

> I thought the original 68000 just did not support virtual memory with paging, and the 68010 introduced that

The 68000 "did not support virtual memory" because instructions that caused a synchronous[1] bus exception could not be restarted. It was a bug with the state pushed on the stack (though I forget the details). In fact it was a bug[2] almost exactly like the one that produced the stack mess we're talking about. And Motorola never bothered to fix it. Intel could likewise have announced "you can only assign to SS when SP is zero" and refused, with little change in device success. And I find it surprising that they didn't.

[1] Because technically the bus isn't synchronous on a 68k, thus the obvious-in-hindsight design thinko. See [2].

[2] A design error following from the failure of the engineers to anticipate the interaction between two obvious-seeming features (processor use of the stack for interrupts, and stack segments in the former, hardware bus errors being used to report synchronous instruction results in the latter).

I've been privy to such discussions myself, albeit in modern times. I'm familiar with silicon engineers and kernel engineers coming together to discuss a CPU bug or design mishap, and coming up with all sorts of crazy workarounds. Until it's eventually decided that this is just too much to inflict, and it's fixed.

Since you are interested in the 68000 thing, I think I do know the details and can explain, because I have before looked at the difference between the 68000 and the 68010 that makes paging possible. Let's look at the 68010 manual here: http://www.bitsavers.org/components/motorola/68000/MC68010_6...

(Notice how it already says "VIRTUAL MEMORY MICROPROCESSORS" on the front while the 68000 manual in the same directory does not, though I suppose it could be a good marketing move if there was an actual unintended bug, and Motorola decided to advertise the fix as a new CPU?)

Page 5-12: "Bus error exceptions occur when external logic terminates a bus cycle with a bus error signal. Whether the processor was doing instruction or exception processing, that processing is terminated, and the processor immediately begins exception processing."

It specifies external logic, which would be the MMU in the interesting cases. Everyone not using an MMU did not care, and bus errors were still valuable with or without MMU to signal illegal accesses and crash the program that caused it. The full problem is elaborated on the next page: "The value of the saved program counter does not necessarily point to the instruction that was executing when the bus error occurred, but may be advanced by up to five words. This is due to the prefetch mechanism on the MC68010 that always fetches a new instruction word as each previously fetched instruction word is used (see 7.1.2 Instruction Prefetch)."

The 68010 fixed it by adding enough (opaque, undocumented) internal state for the CPU on the stack to effectively undo what it did so far and restart the instruction, that state is shown (without detail, and subject to change between revisions) on the diagram.

The 68000 manual simply says: "Although this information is not sufficient in general to effect full recovery from the bus error, it does allow software diagnosis" (which should include a multitasking OS killing the task and proceeding), and none of the internal state exists on the stack.

But you may still well be right that Motorola did intend for bus errors to be restartable and failed. They would not say so explicitly in the manual of course.

But from what I can see this still only affected systems with an MMU, and then only if they wanted to do full virtual memory including paging. SINIX on the PC-D did not, they were content with swapping and merely detecting illegal accesses, like other UNIXes at the time. The 8086 problem affected everyone, including in application code.

If you know more than what I've cobbled together and can elaborate, I'd be extremely interested in it, because the Apollo DN100 hack always fascinated me to no end (as should be obvious).

It isn't. It's a rare case of a software bug being worked around in hardware. That alone makes it fascinating.

The natural way of handling it in software would be to disable interrupts temporarily while changing the stack segment and pointer. You could just say that software developers should do that, but there's a performance cost, and more importantly: The rare random failures from the software already out there that didn't do this, would give your precious new CPU a reputation for being unstable. Better to just make it work.

(Some machines using these processors have hardware to externally gate the NMI input to the processor, but disabling the NMI through these methods defeats the whole purpose of the NMI, which is being able to interrupt the processor even when it has disabled interrupts.)

The root cause was effectively a bug in the programming model, not in the implementation -- there was no way to atomically relocate the stack (by updating SS and SP). The processor state wasn't indeterminate between the two instructions that updated SS and SP, but the behavior resulting from an interrupt at that point would almost certainly be unintended, since the CPU would push state onto a stack in the wrong location, potentially overwriting other stack frames, and the architecture as originally designed provided no way to prevent this.

Therefore, the CPU has consistent state, but an extra fix was still necessary just to prevent interrupts after a mov/pop ss.

So, as I understand it, the problem isn't that code running in the interrupt handler might push data to a bad place; the problem is that the the processor itself would do so when pushing the return address before branching to the interrupt handler.

I want to emphasize this point: when handling an interrupt, the x86 processor itself pushes the return address and the flags into the stack. Other processor architectures store the return address and the processor state word into dedicated registers instead, and it's code running in the interrupt handler which saves the state in the stack; that would easily allow fixing the problem in software, instead of requiring it to be fixed in hardware (a simple software fix would be to use a dedicated save area or interrupt stack, separate from the normal stack; IIRC, it's that approach which later x86 processors followed, with a separate interrupt stack).

Some detail here: https://www.kernel.org/doc/html/latest/x86/kernel-stacks.htm...

Is there a software that builds/reverses circuity from die photos or this is all manual work?

The slides are a great summary: http://www.visual6502.org/docs/6502_in_action_14_web.pdf

https://twitter.com/kenshirriff/status/1596622754593259526

I prefer the challenge of mentally stitching together individual sentences interspersed by ads.

It's a very good starting point, getting a summary of the topic, before diving into a longer blog post knowing what to expect and with some knowledge you probably didn't have before.

The way I'm learning best is having some skeleton that can be fleshed out. A concise Twitter threads builds such a skeleton if I didn't have it already.

Like, even your comment would need to be split into 2 twits

Then again I read pretty fast so someone writing slightly longer sentence than absolutely necessary doesn't really bother me in the first place

Following that argument you can't ever build up proper knowledge unless you get a deep introduction into a topic.

> And format of Twitter is abhorrent.

I disagree but you need to write to its format. Just writing a blog post and then splitting it up for cramming as much as possible into a tweet, possibly even breaking within a sentence, yes, that is abhorrent.

> Like, even your comment would need to be split into 2 twits

3 even if you preserve the paragraphs I used.

> Then again I read pretty fast so someone writing slightly longer sentence than absolutely necessary doesn't really bother me in the first place

It's not about reading slow, it's about getting small pieces of information. Tweets usually don't have the fluff that many blog posts do. Or even articles that take up one third of the whole piece to describe a person in a coffee shop, drinking a specific brand of coffee to illustrate some superficial point that has a vague connection to the actual topic of the article.

It's the same reason why many people go to the Hacker News comments before clicking the article. :)

let's recognize subjectivity a bit here. it's different, you may like it more -- that's great , but prove 'better' a bit better.

i've said this dozens of times, and i'm sure people are getting tired of the sentiment, but Twitter is a worse reading experience than all but the very worst blog formats for long-format content.

ads, forced long-form formatting that breaks flow constantly, absolutely unrelated commentary half of the time (from people that have absolutely no clout or experience with whatever subject), and the forced MeTooism/Whataboutism inherent with social media platforms and their trolling audience.

It gets a lot of things right with communication, and many people love reading long-form things on Twitter -- but really i'm just asking you to convince me here ; there is very little 'better' from what I see : help me to realize the offerings.

tl;dr : old man says 'I just don't get IT.' w.r.t. long-form tweeting.

It's hard for an outside observer to judge which one you mean, and just in case you meant the advocating one, people might feel to discuss to prevent damage.

It's easy for you to clarify with "specifically for me" or something similar.

And there even is something that I think is generally better, namely that I've seen this blog (and others) attract long form comments from people who were also there, and elaborate some more historical details in them. On Twitter it's mostly condensed to the bare minimum, which you can see on this very Twitter thread! So losing the blog would be bad.

Yeah, you can see that by just looking at the comments for the first tweet there. A few ones are maybe interesting nuggets of information (but a comment on the blog or on HN might have elaborated more), the rest is... exactly what you say.

I suppose you’d have to re-test everything with either a new layout or this fix, so no real cost to save there?

Sort of. From what I've gathered, at this point Intel was still doing the actual layout with rubylith, yes. But in small modular sections. The sheets were then digitized and stitched into the final whole in software; there wasn't literally a giant 8086 rubylith sheet pieced together by hand, unlike just a few years before with the 8080. But the logic and circuits etc. were on paper, and there was no computer model capable of going from that to layout. The computerized mask was little more than an digital image. So a hardware patch it would have to be, unless you want to redo a lot.

Soon, designers would start creating and editing such masks directly in CAD software. But those were just giant (for the time) images, really, with specialized editors. Significant software introspection and abstraction handling them came later. I don't think the modern approach of full synthesis from a specification, really came into use until the late 80s.

I found a paper on this 'Coping with the Complexity of Microprocessor Design at Intel – A CAD History' https://www.researchgate.net/profile/Avinoam-Kolodny/publica...

>In the 80286 design, the blocks of RTL were manually translated into the schematic design of gates and transistors which were manually entered in the schematic capture system which generated netlists of the design.

>Albert proposed to support the research at U.C. Berkeley, introduce the use of multi-level logic synthesis and automatic layout for the control logic of the 386, and to set up an internal group to implement the plan, albeit Alberto pointed out that multi-level synthesis had not been released even internally to other research groups in U.C. Berkeley.

>Only the I/O ring, the data and address path, the microcode array and three large PLAs were not taken through the synthesis tool chain on the 386. While there were many early skeptics, the results spoke for themselves. With layout of standard cell blocks automatically generated, the layout and circuit designers could myopically focus on the highly optimized blocks like the datapath and I/O ring where their creativity could yield much greater impact

>486 design:

    A fully automated translation from RTL to layout (we called it RLS: RTL to Layout Synthesis)
    No manual schematic design (direct synthesis of gate-level netlists from RTL, without graphical schematics of the circuits)
    Multi-level logic synthesis for the control functions
    Automated gate sizing and optimization
    Inclusion of parasitic elements estimation
    Full chip layout and floor planning tools

One motivator is if the fixes are simple enough, the modifications can be done such they only affect the top-most metal layer, so there is no need to remanufacture the masks for other layers, saving time and $$$.

It's striking how many of the early microprocessors shipped with bugs. The 6502 and 8086 both did and it was an even bigger problem for processors like the Z8000 and 32016 where the bugs really helped to hinder their adoption.

And the problem of bugs was a motivation for both the Berkeley RISC team and for Acorn ARM team when choosing RISC.

The many bugs (usually from 20 to 100) of each CPU model are enumerated in errata documents with euphemistic names like "Specification Update" of Intel and "Revision Guide" of AMD. Some processor vendors have the stupid policy of providing the errata list only under NDA.

Many of the known bugs have workarounds provided by microcode updates, a few are scheduled to be fixed in a later CPU revision, some affect only privileged code and it is expected that the operating system kernels will include workarounds that are described in the errata document, and many bugs have the resolution "Won't fix", because it is considered that they either affect things that are not essential for the correct execution of a program, e.g. the values of the performance counters or the speed of execution, or because it is considered that those bugs happen only in very peculiar circumstances that are unlikely to occur in any normal program.

I recommend the reading of some of the "Specification Update" documents of Intel, to understand the difficulties of designing a bug-free CPU, even if much of the important information about the specific circumstances that cause the bugs is usually omitted.

Are there any good documentary style videos that dive into some of such details of chips and microprocessor architectures?

Lo and behold... looks like Ken just changed his domain name.

Amazing work, as always, Ken.