IBM Z systems are always fascinating to a computer engineer. I was fortunate at my first job to be able to access a z10 brand new, and was tasked with installing, configuring, and trying out several POCs with z/OS, DB2 on z/OS, WebSphere Application server on z/OS, CICS transaction server, z/VM operating system, running 1000's of linux images on top of z/VM, making them communicate with each other and z/OS using Hipersockets. It was pure fun.
The cool moments I distinctly remember were runing the z/OS installation binaries from tape drives, and swapping in and out - DASDs(Hard disks), and perhaps the coolest was being able to hot plug in/out CPU blades from the cabinet while the system is live.
It's pity that thing is so expensive, and out of reach of the common folks.
For those not in the know, and curious.
Based on my knowledge from 2011, CPs are the general purpose CPUs that run z/OS - the main OS companies typically run on System Z.
There were special CPUs(may be microcoded so), called IFL that were allowed to run z/VM and Linux.
I don't know how things have evolved in the last 7 years :-)
The bulk of my mainframe exposure was with z/VM, but you're right, those systems are fascinating. I've never understood why IBM doesn't provide hobbyist licenses of z/OS and z/VM for individuals to learn on. Of course, the software would be run on Hercules [1] emulator and not a real mainframe. Even so, it would be a great avenue for people to learn about the mainframe software environment and maybe even help with the mainframe skills shortage.
Things were different in the 1970s. And perhaps that explains why IBM is more secretive now.
The Itel AS/5 was a clone of the IBM System 370 Model 158, a very popular mainframe for its time. How was Itel able to clone it? Simple. Just use the hardware schematics!
ALD's were Automated Logic Diagrams. IBM sold these.
They were complete schematics for the S-370.
EXSYSCO basically duplicated the logic of the S-370.
I had personal experience with the AS/5. It came supplied with IBM schematics and microcode listings. These were kept in giant binders in the machine room.
When the machine booted up, a Z80 based processor that Itel designed was used to load IBM microcode into the clone mainframe.
Oh, and the application software that ran on that 1970s machine will still run on IBMs latest mainframes.
When IBM introduced the PC in 1981, hardware schematics and BIOS listings were also available for purchase.
So I joined IBM in 2005 working on firmware for a project called eCLipz, which eventually produced the POWER6 family of systems. We've just shipped POWER9 which is in the #1 and #2 supercomputers in the world, Summit and Sierra. [1]
The "i/p/z" part of the project name is important, because it represents:
i - the lineage of the AS/400, which was a 'minicomputer' design from the 80's designed for multiple users, which came up with the idea of "wizards" for administration tasks before there was even a name for a "wizard." The "i" means "integrated," since IBM i is meant to require very little administration. [2] [4]
p - the lineage of the RS/6000, which begat AIX UNIX. The "p" indicated "performance." The box I worked on shipped at 5 GHz in 2008 (IIRC).
z - the lineage of the System/360 mainframe which the above article discusses. The "z" indicated "zero downtime."
The control structure and power and cooling infrastructure for the systems used common elements with that project. The hardware and firmware underneath the operating systems for i and p converged into Power systems, which eventually produced a variety of projects in the open source community (OpenCAPI, OpenPOWER itself, OpenBMC), and of course runs bi-endian to support things like little-endian Ubuntu or the classic Big-endian AIX or i/OS on the same processor core.
Interesting trivia - In IBM systems (Mainframe or Power), the cold boot of the processor from an unpowered state is called an "Initial Program Load," or IPL. The IPL lingo dates back to System/360 (see its Wikipedia), and survives to this day in really interesting places in the wild. Take the OpenBMC project, which aims to create a 100% open source software stack for those sideband management processors found in servers (and increasingly in other devices too). This github bug from a few months ago which complains about a bug causing "IPL" problems. [3] ;-)
> i - the lineage of the AS/400, which was a 'minicomputer' design from the 70's designed for multiple users, which came up with the idea of "wizards" for administration tasks before there was even a name for a "wizard." The "i" means "integrated," since IBM i is meant to require very little administration. [2]
That specific system was introduced in the 1980s, but it's even more interesting than that, because it's the last remnant of the Future Systems project which did date to the 1970s:
> The IBM Future System (FS) project was a design effort to succeed the IBM S/370 architecture that was adopted company wide in 1971. FS was based on a single-level store with automatic data management. The project is reputed to have cost more than $1B when it was cancelled in February 1975, but the single-level store idea was adopted by System/38, AS/400, and iSeries.
It had interesting effects on IBM:
> Most corrosive of all, the old IBM candor died with F/S. Top management, particularly Opel, reacted defensively as F/S headed toward a debacle. The IBM culture that Watson had built was a harsh one, but it encouraged dissent and open controversy. But because of the heavy investment of face by the top management, F/S took years to kill, although its wrongheadedness was obvious from the very outset. "For the first time, during F/S, outspoken criticism became politically dangerous," recalls a former top executive."
I have friends who still say their program "abended".
Just a minor nitpick - the green screens we see today are also often descendants of the 3270 family as well as the 5250's that started life with the System/34 and found their way into the AS/400 and iSeries.
I haven't ever really worked on Mainframe as such, but I _am_ in the IBM/ERP world, and, well, a program Abends, that's just what it does when it goes South.
Not even sure what else to call it - failed? Errorred out? ...Segfaulted its guts on the ground? :->
(edit: I suppose I should also fess up that we call virtual machines "LPARs" and physical machines "frames" :P )
Thanks for this shot of nostalgia that is rushing through my brain. Yes, I remember IPL from z/OS, working with RACF (Resource Access Control Facility), LPAR (Logical Partitions), and the complexity of Parallel Sysplex (clustering) all leading up to running COBOL and PL/I applications mostly for banks.
The TN3270 terminals, and the wild hacks of writing Assembler macros to translate those TN3270 inputs and outputs to a webservice :-)
I have never worked with the Power systems personally, but heard a lot of good things about it.
It always felt like IBM enterprise stuff that $$$$ heavy are in a world of its own completely disconnected from people who cant think in that much $$$s.
> It always felt like IBM enterprise stuff that $$$$ heavy are in a world of its own completely disconnected from people who cant think in that much $$$s.
You're not wrong. Licenses for even the most trivial of software cost literal fortunes.
It's just that these systems process billions of software and run millions of lines written decades ago.
When I first heard about eCLipz, I thought it was pushing towards a single processor architecture for all three series. From today's article, it seems as though z/Series machines still get quite distinct CPUs, though. Do you know to what extend these processors are relatives of the POWER9?
POWER6 and z10 share functional blocks, but the ISAs are pretty different.
I'm not familiar with the z architecture enough to comment with any sense of certainty, and the people I'd normally pester are on vacation, unfortunately.
I do love the lingo from these machines. I remember encountering the need to "vary off" an interface.
The weirdest was the being told I needed to "wand on" the system. I guess the origin was from the AS/400 when they were pushing cards with a barcode as ids, so you would scan your card with a optical wand (pen) to login to your station. Years later I still hear "wand on".
They used to provide a free z Linux guest OS available to anyone. I used it to verify that my open source projects worked on Z. It was RedHat 7.3 I think.
I can run a kubernetes cluster, with 1000 of linux images, have them communicate with each other. Hot swap in and out nodes. Such a thing is not out of reach of common folks if you change your mental model of what a computer is. "The network is the computer"
That is an Apples-to-Oranges comparison. The Z-series already changes your mental model of what a computer is if you are not yet familiar with it. A Kubernetes cluster is not a computer, it is a cluster of computers.
The end result (computations being done) may be the same but the way you get there is radically different between those two setups and you're going to have to do a lot more work on your application on a cluster vs a very large computer with some awesome hardware capabilities thrown in. Not cheap, but for some applications well worth it.
> A Kubernetes cluster is not a computer, it is a cluster of computers.
That statement is true, but the thing it implies isn't. A cluster of computers on a common backplane can certainly present itself as a single (NUMA) computer, with one process scheduler, one filesystem, etc. Most HPC systems are built this way.
As well, when your workloads are, themselves, VMs, there isn't much of a difference between having a Z-series machine as a hypervisor and, say, a vSphere cluster as a hypervisor. Either way, you get a pool of CPU, a pool of RAM, a pool of storage, etc. And either way, you get VM workloads with transparent migration on node hotplug/node failure, etc.
But that comes at the expense of complicated software. Sure, once you've gone through the effort of procuring all that hardware, configuring it to run Kubernetes, reconfiguring your software to run in containers... you're good to go.
Mainframes allow you to easily scale your software vertically (but at the expense of complicated hardware). That might seem silly in today's world, but a lot of the software running on Z was written decades ago and the risk of rewriting it is extremely high.
Coding for a mainframe in the early days was almost regimental in many companies, lots of dotting i's and crossing t's, code sign off, lots of QA and checks.
One example would be a bit of COBOL I did and associated test program, all worked, ticked all the boxes accept there was a single spelling mistake in one comment line, actually I'd missed the full stop of the last sentence.
So I had to make that change and retest it all from scratch again. How many would just leave it, or not fully test every instance and associated program that had in full and document all tests and results compared to expected results today? Yes we still have industries that would do that, airline industry or finance systems, but that was the prevail of all code back then.
But then source code control does not consist of a group of auditors in a separate group physically comparing outputs with previous with a lightbox. So been much progress, though equally lots of automation in which a single flaw can cascade.
Now as for scalability - mainframes lived on what we know as moore's law today more than we do today and iirc the likes of IBM effectively promised it's customers a solid path for tomorrow, today and remember - what we now know as LTE today, was and is the mainframe staple and LTE for them, means a lifetime. But you pay for that level of service, but always have. This enables lots of legacy, well tested and proven code to carry on still being used today.
Equally a relevant anecdote, was working in the 80's for a large company who had iirc a DPS88T (Honeywell Bull), which had a maximum of 4 CPU clusters available. This company was hitting max usage and I identified a program that I could optimise and recover near on a whole CPU's worth of processing back for use. I also mentioned that dispite only officially having 4 CPU clusters available, it was possible to attach a 5th. May mistake as next thing they are onto the supplier demanding a 5th CPU. As for them money solved a problem and whilst a cheaper option was available that would entail change and when it comes to changing code that just works as needed, they are always reluctant to change it when they can carry on just adding more power.
And the mainframe suppliers make their money on supplying more power, year after year, all in a heavily fault tolerant hardware design that goes beyond just having ECC memory and thinking life is good. But mainframe hardware is a well worth looking into as redundancy, legacy handling, robustness is at a whole level many don't appreciate.
The vast majority of mainframe software, especially older software was batch processing oriented - often millions of discreet financial transactions - that can easily be scaled horizontally across multiple CPUs or processing engines.
You can also run Kubernetes on a mainframe now, with ZLinux and the hundreds of precompiled s390X (IBM CPU architecture) docker images: https://hub.docker.com/u/ibmcom/.
This is attractive if you happen to already be a large company and locked into mainframe contracts. Sure, you don't need a mainframe to do this, but it's also nice to not need to throw away your mainframe and buy a bunch of commodity hardware to do this.
Well, it is murky if you start generalising that way. The equivalent of Kubernetes in the Mainframe world would be Parallel Sysplex - where there is a cluster of Mainframe servers that co-ordinate with each other and take over upon failure or over load of other participants of the cluster.
A single system Z is vertically integrated and scaled. Now, I was talking about a single operating system that can handle a hotplug out/in of several of its CPUs without any kind of service degradation.
Sure, I can throw in a few x86 servers, and achieve the similar convenience using say load balancer, floating IP, shared storage, and whatever else is needed for a full switch over. You see where I'm getting at. Complexity that is on YOUR plate.
I'm not denying that Kubernetes gives a taste of such resilience for the common folks, just saying that the IBM system Z's engineered solution is on a different level that is just not what Kubernetes is aimed at.
As a side rant, Kubernetes is a magic layer for most of the users, that it is glorious until they hit the first serious problem that makes them scratch their head and look at their complex black box differently. With Mainframes, the banks just call IBM :-)
At some point in history, I had the misfortune of working with Stratus hardware.
Not quite where you describe the Z system, but it did operate with hot swappable components, and multiple OS's running in a "fault tolerant" manner.
We hit show stopping firmware bugs that managed to take down every OS. The end result was that the three standalone Dells on top of it had better uptime.
I always wondered how the mainframe world ever managed this. When you market is so small (relative to x86) every case is an edge case. Surely these extremely large and expensive devices still had bugs that transcended that redundancy.
Here's a 100,000 foot view of mainframes (based on when I was working with them and on my limited view).
The mainframes that I worked with (not sure if this is true with all) had something called Capacity Upgrade on Demand. This allowed the data center to temporarily turn on extra processors/engines to handle unusually high processing demands. After the crunch is over, the extra processors can be turned off. I'm not sure how the money aspect of this works.
What the x86 world thinks of as a bare-metal box/chassis is what looks like a full sized rack, except it's enclosed in a cabinet. This bare-metal box is carved up into LPARs (logical partitions) using software called PR/SM [1]. I've read a number of times that PR/SM is a stripped-down version of z/VM. I don't know if that's true or not.
IBM has multiple operating systems for mainframes. The most well known is z/OS (formerly known as MVS). There are others like z/TPF and z/VSE (maybe even more?). There is also z/VM.
JCL (Job Control Language) is used by z/OS (possibly z/VSE too, idk) for running "jobs". This is essentially like shell scripts. The job is submitted and then queued for execution. After execution, the results are made available in an output queue. From Linux on x86 point of view, it's like all program output is being redirected to a file.
There are a number of z/OS add-ons that are typically licensed to run on z/OS. One is TSO (Time Sharing Option). I don't know if TSO is separately licensed or not. Basically, it makes a batch environment almost appear to be interactive. Another component commonly licensed is ISPF (I don't know what it stands for). ISPF is basically a menu-based system that is configurable and I think it comes with some helpful built-in utilities.
z/VM's history goes back to the late 60's and early 70's. Supposedly, one of the primary reasons it was developed was to give z/OS system programmers a sandbox for developing and testing changes. Apparently, some mainframe shops also ran MVS and other operating systems as guests under VM in production (this was probably in 80's). This wasn't the only use of VM though. There was CMS that is a single-user OS that was typically used for end-user business users. Since VM with CMS guests were so common, one would often just see "VM/CMS" written. From within CMS, one could issue CMS commands as well as CP commands. CP is "Control Program" and refers to base z/VM (hypervisor).
One can also run z/VM as a guest under z/VM. This is referred to as 2nd-level VM and is used for testing/evaluating new versions of z/VM before installing them into production.
When Linux hit the scene, versions of Linux were built for mainframes and Linux was added to the list of guest OSs that z/VM supported. To be clear, this is real Linux -- not emulated. It's normal Linux, but compiled to mainframe opcodes.
Once you get access to a Linux guest on z/VM, it's really hard to tell that it's different from x86. It's not weird or different. Any linux user would feel just as much as home with Linux under z/VM as with x86.
Parent posted discussion of CPs and IFLs. Back when I was on the scene, there were also zIIPs and zAPPs. I think zIIP was for encryption or compression (maybe both). I believe zAPP was primarily for running Java workloads. It's been a long time, so my recollection may not be 100% accurate.
Probably the biggest shock to a newcomer of z/VM and z/OS is that terminal sessions are block-oriented (3270) and not character-oriented. With 3270 session, you type whatever you need to on a screen and then when a special attention key is pressed, your screen information is passed to the mainframe to process whatever you typed. No vi editing with 3270. Text editing on 3270 is with line-oriented editors like XEdit.
The scripting language for z/OS and z/VM is REXX. It's as high level as Python and is really a nice language once you get used to its quirks.
Here ends my mainframe brain dump. Can't guarantee that it's all correct or even still relevant, but hopefully gives a little bit of the feel for the environment.
I used to work at IBM, making x86-based appliances. Occasionally there was a push to use Power, but x86 single-thread performance was better and cost was lower so it never happened. What would have been nice is if they allowed us to use z's CPU, but in a 1U or 2U appliance. The marketing advantage would be that we could claim to get Z's reliability for our appliance (even if it was slower). I understand they wouldn't make a generic z low end z/OS server (competing with themselves), but this would have been a nice closed-system use case.
BTW, mainframe group takes reliability very seriously. Basically it must not ever corrupt customer (financial) data plus the fit rate must be as good as it can possibly be. I remember when they were making the zBx add-on (an x86 blade center for z which could run Java web apps), they tested the x86 in a cyclotron beam to get an idea of its fit rate- it was quite good (good enough), but not as good as z's CPU.
Where I work, been here for twenty years, and in all that twenty years I have been told we are going to do away with the mainframe which is a Z series. Of course we have been told the same about the i series which is actually larger. What have we done in the last year alone, gone to the Z-14 and a 32-way 870 i.
seriously people underestimate the cost to move off a platform that is working and worse when there are so many touch points no one can seem to get an accurate handle on them all.
IBM does indeed take their reliability seriously, both i and Z are all built around not having any downtime but the Z does do it better than i which still can require it for patches.
i just find it fascinating the work that we use these machines for and how instead of being replaced more work lands on them because of reliability issues of other platforms. That and support requirements of other platforms. The z and i teams are three man admin with a few operators to nod heads... the other platforms seem to have armies
Not having access to a cyclotron beam I would use gen-1 car phones as my go-to test device for hardware reliability in the face of RF interference. That is incredibly hard to defend against already, I can't imagine what kind of countermeasures you'd have to take to get your device to function under the circumstances you sketch.
It can detect, correct and survive normal bit errors of course. But then it can detect, correct and survive power regulator, clock, DRAM, lane, controller and channel failures. RAIM (Redundant Array of Independent Memory) eats up 20% of the memory capacity.
If you want Z CPU, then you might want the memory subsystem as well. If you have them, you probably want the redundant IO as well. Then it's easier to buy the whole thing.
The "baby system" market does exist at IBM, it's just Power-based instead. My personal POWER6 is one of those (an 8203-E4A), and with AIX you can do all the fun stuff like hotswapping PCI cards while the system is up and running. It's a different level than typical commodity boxes.
Seeing z14 at the top of HN makes me really happy :). I actually currently work Z systems as Linux kernel dev. Z processor is a beast and has some great features.
Really enjoyed reading this, even if I didn't understand a lot of the jargon.
I find mainframes fascinating beasts, I don't understand how a programmer would be able to make use of all those "CPs" but it sure looks like a lot of power to use!
It's very exotic, maybe too exotic. I'd imagine hiring junior people and training them up on these things is a lot of effort and probably very hard to hire for - in competition with the companies offering cloud experience etc
I was hired as a junior software engineer for a mainframe company, and it is indeed very hard. zOS was radically different from anything I had done before and it took a long while before i felt comfortable with it.
IBM actually holds a very good week long training you can send junior developers to. It’s basically a boot camp. There were about 10 other developers at it when I went, most of whom were fresh college grads from the big mainframe software companies (BMC, CA, Rocket, etc.).
I’ve since moved on to the world of .net and I don’t miss the mainframe stuff at all.
Edit: I’ll add that a lot of what made it difficult for me was that I was working with assembly language and our product required a lot of knowledge about the OS. Many mainframe programmers are using java and eclipse, which isn’t that different from using java and eclipse to target any other system.
Every time I read tech news on IBM's Z series mainframes I wonder what a minimum viable / reasonable build out of this hardware might be. As a regular human there's no way or reason I'd ever really need an actual mainframe but the hardware itself has fascinated me for a long time.
IIRC most of the 90's AS/400's are of similar size as contemporary servers. Or twice as big when they have "Integrated something whatever Facility", which is simply an x86 server that occupies half of the enclosure.
It's a little sad these machines are so expensive. I can see it from IBM's point of view, there probably is not much money to be made in scaling these machines down, but still.
For IO intensive transaction processing with high security and low downtime requirements these systems are quite sweet.
IBM sells it with moniker "Pervasive Encryption" but if you look under the hood, it is actually impressive and very detailed.
Build in security at the hardware level. Especially the Crypto Express6S card. There is hardware acceleration for everything cryptography (including SHA-3 and SHAKE), tamper sensing, true random number generation. Encrypted paging in z/vm, etc. SSL/TLS can run fast. Security auditing is not an afterthought.
Of course you can add all this in special cards for x86 server racks but the cost is probably the same with lower IO throughput and less integration. Getting the same uptime in production would be significantly more expensive if not impossible.
On unreliable commodity hardware, you would build a cluster and cluster OS / scheduler that would keep your service and jobs running while individual servers died. With a mainframe, you save costs on software development by just assuming the hardware is 100% reliable.
The z14 has a ton of cache and does everything it can for single threaded performance, but for the money you could buy a ton of cores (Intel Xeons or AMD chips) and trounce the mainframe on benchmarks that can use the parallelism.
For the money, you would be able to buy the gear to have a lot of bandwidth between nodes and to the storage.
"trounce uptime by a couple of orders of magnitude"?
Not true. If you view the network as the computer, then the network is now forever winning.
What is Google's or Facebooks uptime? Their computer is their cluster. Those clusters are never going to be powered off ever. They can upgrade and replace the entire cluster without being down. If you have a cluster of 10,000 nodes, turning off 100 or 1000 is not downtime, so longer as it's up and serving. No mainframe ever will run for the next 100 years. It's a possibility that a cluster built today could be running for the next 100yrs.
99.9 percent. Cloud services from Microsoft, Google, Amazon typically have only three nines or less for paying customers. They also seem to have same level of uptime in their own services.
But these are not "firm guarantees". For example Amazon just gives service credits after downtime exceeds what is promised. They "undergo unscheduled outages" from time to time. Whole regions are down at once.
Remember Amazon's Prime Day outages? It's not good for business when tings go down.
Exactly, I think that folks forget that ATMs operate like that, if the network is down ATM doesn't stop working. That's why folk's account can go below $0. Banks have been operating distributed systems with eventual consistency for a very long time.
And even then, what is their non-degraded uptime? There are parts of their infrastructure failing all the time being routed around by incredibly clever software.
And I bet all that costs much more than an IBM. Not a computer, the company.
One of my co-workers manages the SGI UV300 described at http://med.stanford.edu/gbsc/uv300.html (new purchased get hardware badged as the HPE MC990 X). It does fit most definitions of a supercomputer, as it's multiple nodes with interconnects that allow a single OS instance to run across all of them (the OS then sees & manages the separate NUMA nodes).
The group I'm in at NASA Goddard has two UV300s. They have 96 cores each (no hyperthreading) and 6TB of RAM. And they are loud.
The SGI Altix line, which used Itanium 2, could go up to 1024 cores with a single system image, which is just insane. We had one of those at Goddard a decade or so ago, but I didn't have access to it.
I'm fairly sure UV (or some SGI labelling) has gone up to 1024-ish cores in a single system image, but I don't have a reference. (SGI Origin 3000 was spec'ed to 512 cores long ago.)
Speaking of SGI, I was expecting the Cray XC family to be able to do that with the ludicrously fast interconnects they have, but I couldn't find any mention of it.
No, an XC40 runs a kernel on each node, the programming model is MPI, not SSI with NUMA. It runs a minimalist Linux distro called CLE. The nodes are surprisingly small, in terms of memory too, 64 or 128G. Physically they are beautiful to look at, so clean, just a CPU and some memory and nothing extraneous. Cray really should do glass cases.
No, the Ares interconnect is brutally fast but it fundamentally isn’t doing anything as complicated as that. It acts as both storage and networking between the nodes.
The biggest I heard of was built by SGI (although it was just the husk of SGI which was was bought by another company which kept the name), now HPE. It had up to 32 Xeon sockets (so maybe 256 cores?) and up 24 TB of RAM.
You can get 8-way Xeon Platinum with tens of terabytes of memory that'll more closely mimic a mainframe. You probably won't be able to fit more than 3 in a rack and they will be hideously expensive (but, at least, will have hot-swappable storage and power supplies) and will be mostly air-cooled.
I didn't check this out but what made me respect IBM was their cell (PS3) processor. Very well thought design and ahead of time. Shame it was hard to program.
The cool moments I distinctly remember were runing the z/OS installation binaries from tape drives, and swapping in and out - DASDs(Hard disks), and perhaps the coolest was being able to hot plug in/out CPU blades from the cabinet while the system is live.
It's pity that thing is so expensive, and out of reach of the common folks.
For those not in the know, and curious. Based on my knowledge from 2011, CPs are the general purpose CPUs that run z/OS - the main OS companies typically run on System Z. There were special CPUs(may be microcoded so), called IFL that were allowed to run z/VM and Linux. I don't know how things have evolved in the last 7 years :-)