> MS-DOS had an additional memory model known as Tiny, in which the code and data were all combined into a single 64KB segment. This was the memory model required by programs that ended with the .COM extension, and it existed for backward compatibility with CP/M.
The usual "tiny" memory model had all the segment registers ES DS SS CS set to the same value. It was loaded by simply copying the program file into memory and jumping to it.
At Zortech we realized that it wasn't necessary for all the segment registers to be the same - it's just that the program file needed to be less than 64Kb. This meant the code plus the statically initialized data had to be less than 64Kb. SS and DS, though, could be changed to the start of the initialized data, and had access to 64Kb beyond that, enabling significantly larger programs while still using near pointers.
This trick generally works, but is not guaranteed to work by DOS API. COM binary gets 4k paragraph allocation and nothing more, the fact that this allocation essentially always is at the bottom edge of free conventional memory is not even an implementation detail, it is consequence of everybody trying to be frugal with their memory allocations. ie. if there was TSR that allocates and then frees 64k region, you will get exactly this region and this trick will break and you will overwrite who knows what.
As I recall an operating system call was used to extend the memory, so it should never have broken unless a broken TSR was used. There weren't any issues with it that I know of, used it all the time for many years.
To be honest, I remember something different. Dos was originally a single process OS. So memory was allocated like a stack, at least for a .COM.
Some detail:All memory above your starting address is yours. You call int 12 to find out how much base memory is installed, which in later computers was either 640K or 639K if the bios wanted a bit or storage.
If you want to start another program, you first resize your part of the memory with INT 21 (ah=4D? Its too long ago, don't quote me on this)
Later, there came TSRs, EMS/UMBs/high memory/... to muck this scheme up.
A TSR was not allowed to allocate memory after it terminated, exactly because it runs code in another process while it should have been terminated. It had to always stay under all other programs, memory address wise.
It was not allowed to do much, in fact, because DOS was not reentrant, and running int21h while running int21h had a good chance of making DOS overwrite its own variables.
UPDATE: It was quite hard to find reference information about all this. I vaguely remember reading it in one of the Peter Norton books at the time, but I can't really find them. The most relevant thing I can still find on todays internet is this:
... the operating system simply makes the worst-case assumption and gives .COM programs everything that is available. If a .COM program wants to use the EXEC function to invoke another process, it must first shrink down its memory allocation ...
DOS was much more of a glorified process loader than an operating system as we think of one today. It had no knowledge of memory models, very limited knowledge of protected mode, and no means to constrain application behavior.
"DOS" memory models were styles of programing rather than features the DOS kernel understood, much less supported. The DOS API always used far pointers for addressing and had no notion of which memory model a calling process used.
As far as protected mode is concerned, DOS has very limited knowledge of protected mode. Any DOS application could switch to protected mode at any time. In earlier versions of DOS, this was done by setting up the processor hardware directly. In version 4 and beyond, DOS software entered protected mode using VCPI or DPMI functionality through emm386.
Any DOS process which went into protected mode essentially became the running operating system and was on its own and had no support from the DOS kernel. Technologies were eventually developed to allow protected mode software to call the DOS kernel and BIOS--typically used for filesystem access and video mode setup--but the protected mode code was ultimately in control of the computer. DOS was a second-class citizen at best.
> The DOS API always used far pointers for addressing and had no notion of which memory model a calling process used
Ha, yeah it's all coming back to me.
> Any DOS process which went into protected mode essentially became the running operating system and was on its own and had no support from the DOS kernel. Technologies were eventually developed to allow protected mode software to call the DOS kernel and BIOS--typically used for filesystem access and video mode setup--but the protected mode code was ultimately in control of the computer.
I don't know what it is, but it really felt like a superpower after setup and then flipping CR0. You knew that every single bit of code running on that box after that was hand crafted by you. Unless you go embedded, gone are those days. Computers are too complex these days to know what every single think is doing... we've all now been delegated to mere mortals :(
Those are really two different worlds. "Protected mode" is a CPU function, available from the 286 upwards. (The 286 had a major misfeature: you couldn't exit protected mode again!). "Tiny" is a compiler target, and COM is an executable format and loader specification.
To enter protected mode usefully you have to set up your segment descriptors somewhere. "Real" mode has none of that.
Heh, I was about to point to that same paragraph because of how disingenuous the term "backwards compatibility" is. I'm sure the author wasn't intentionally trying to revise history and probably didn't think about the term much, but to set the record straight, DOS wasn't a successor to CP/M, but a "quick and dirty" clone, then later an actual competitor. COM files were included in DOS to make sure programming for it was similar to programing for CP/M.[1]
The story (which I'm sure many of you know) was that in 1979, Seattle Computer was selling an 8086 board, but CP/M didn't run on that CPU yet. So Tim Paterson solved the problem by whipping up QDOS to help sell more boards, cloning the architecture and APIs of CP/M directly from the manuals (without access to the actual source code).[2]
The simplest sort of executable code is a COM file (doesn't get much simpler, actually), so if you're copying the way an OS works, that's pretty much where you'll start. It was about being a familiar paradigm for programmers, not some sort of cross-platform binary compatibility to an out of date OS as insinuated by the term "backwards compatible".
"Intel documented how the original software developer could mechanically translate an 8-bit program into a 16-bit program. Only the developer of the program with possession of the source code could make this translation. I designed DOS so the translated program would work the same as it had with CP/M – translation compatibility. The key to making this work was implementing the CP/M API."
I have a couple of friends who had that original Seattle Computer CPU board and the QDOS that came with it. I was very jealous. The sale to Microsoft came out of the clear blue.
Again, I don't think the author thought about it much. But at the time, CP/M wasn't a legacy system, and COM files were integral to both systems. Imagine if the sentence was something like, "Windows 1.0 used a mouse pointer, which existed to be backwards compatible with the Mac."
Sure, but an earlier system nonetheless. Similarly, I'd describe Windows 95 as backwards compatible with Windows 3, even if one might describe Windows 3.1 as a contemporary system at the time of 95's release.
The author is Raymond Chen, a developer on the Windows team who has been writing well-researched articles about backward compatibility in Windows and MS-DOS for the better part of two decades, so I think it's safe to say he's very well informed on the subject.
But I don't think he's "rewriting history" at all, intentionally or otherwise: QDOS/MS-DOS was originally designed to provide a largely CP/M-compatible runtime environment to ease porting, hence "backward compatibility" — "backward", not because MS-DOS was a "successor" or because CP/M was obsolete, but for the simple reason that this compatibility was both one-way and limited to features found in pre-existing versions of CP/M.
Additionally, you can segregate a large array (that never has a pointer taken to any of its elements) into a separate 64k area pointed to by ES. Since the only way to refer to an element is with ary[i], you always know when to use the ES segment override. Desperate circumstances lead to desperate measures.
>> MS-DOS had an additional memory model known as Tiny, [..]
Which was by far my favourite memory model;-)
Coming from the 8-bit world all this segmentation, memory model, ES: DS: SS: ES:, far pointer, near pointer stuff confused me to no end. Adding to that the memory access scheme of the graphics card (bitplanes!) made it even worse. No wonder I almost transitioned directly from 8-bit to 32-bit Linux. I still somehow think of the 16-bit (PC) era as kind of the dark ages - a lost time I don't miss much - sandwiched between the glory of 8-bit and 32-bit times.
Did that predate TurboC 1.00? I always used the tiny model in that (being an assembly programmer used to flat memory spaces it was the only way to go).
At one point I added the "v" memory model for Zortech C++, which was a virtual memory system for code. Functions would be loaded from disk on demand, and discarded from memory when not needed. The loader would do all the fixups necessary. I was pretty proud of it, it was far better than the usual hierarchical overlay system.
But by then people were using 286 DOS extenders which were better.
Is there a good article on how DOS overlays worked?
Back when I learned (?) C, I never learned about the x86/DOS memory models. I just knew I was always hitting the 64k limit, and fortunately DJGPP existed at the time so I just switched from Quick C/Turbo C to that.
Overlays implied a specific kind of resource management about keeping unused code on the disk and loading it on demand. Doing that required help from linker so they were mostly implemented by the language like Turbo Pascal/Turbo C's "FBOV" overlays.
You could also keep resources in overlays too but they didn't have a resource management mechanism if I recall correctly. You had to implement it yourself.
The near and far pointers bit is also the reason why in the Windows flavor of Systems Hungarian notation, you have a lot of “LP” prefixes; I believe it means Long Pointer and would give you a far pointer. It’s typical to have one pointer typedef for Long Pointer, like LPBYTE, and sometimes one near, like PIMAGE_DOS_HEADER. Of course now it’s all ignored so if you see someone writing LP in modern code chances are it’s just tradition.
Also, today the segment-jumping far call still exists on modern AMD64 machines, and is how you cross “heaven's gate” to go between 32 bit and 64 bit execution.
> Also, today the segment-jumpimg far call still exists on modern AMD64 machines, and is how you cross “heaven's gate” to go between 32 bit and 64 bit execution.
Addendum: this model only exists on Windows. On Unixes, programs are entirely 64-bit or entirely 32-bit, so there's no LDT setup to have a 32-bit segment in 64-bit code. And there's no syscall on x86-64 to let userspace muck with the LDT as there is on x86-32.
It's worth pointing out that, outside the IT industry, very little "professional" software was actually written to use far data pointers as a default type (a long function call, on the other hand, was only a tiny bit slower than a near one -- this was pervasive). The overhead involved in doing the segment dance for every pointer indirection was simply too high to be practical.
Almost all real mode software made use of some home-grown framework for managing large data across a segmented space.
As someone who wrote 16-bit x86 Asm for a long time, if you are writing Asm by hand, the "tiny" model is not as tiny as it may sound at first glance --- 65,536 bytes seems very far when you can write useful applications starting in the dozens of bytes and going up to a few KB. A "Hello World" is around 21 bytes, half of which is the message itself. The majority of the executables that came with MS-DOS 6.22 are under 64K as well, despite being written in C.
Then I look at the Win32 C equivalent with the default MSVC compiler options, and it's over 64KB already...
FWIW: it’s trivial to make minimal binaries, just, nothing optimizes for it for the most part. If you use OpenWatcom v2 with nodefaultlib you can get win32 binaries that are around ~1500 bytes, which is not too shabby when you consider its padded to 512 byte boundaries, and that the PE headers take at least 300 bytes or so if you aren’t overlapping structures.
I have a project that implements a regular expression engine, JSON parser and some game patching code into a DLL that is around 10 KiB (iirc) using OpenWatcom. Of course it’s easy if you’re only targeting Windows and 32 bit but you can do similar if you drop the standard library in MSVC.
Weird I was just thinking about this last night, in terms of how a multicore computer might work.
Imagine a chip with, say, 256 cores, arranged in an 8x8 grid. Each core would have 64k locally, but could call out to other memory regions and retrieve them automagically in less than 2*sqrt(N CPUs) memory clock cycles.
I'd like it to work like DOS, where the unsigned 16 bit pointers would access the local memory of each chip. But also be able to write a program in 32 bit mode where all of the cores can access the global memory space like in a regular computer. A high-memory program like that might reserve the bottom 64k for system code and low memory variables.
Even better, I'd like progressively more powerful versions of this computer, starting with 4 or 8 bit cores, on up through 16 bit, 32 bit and even 64 bit.
Does anyone know a starting point for a memory architecture like this? Does it already exist?
I'm having flashbacks now of programming in Windows 3.1 and the 64k limit for contiguous allocation. We rolled our own Vector template in C++ in order to get around it, but it still made things a pain, like sorting.
And then there was the dreaded GPF... General Protection Fault.
I always considered myself very lucky that I didn't start programming Windows until it went 32-bit. I knew about near and far pointers but never had to worry about them because they weren't relevant to anything I did.
The usual "tiny" memory model had all the segment registers ES DS SS CS set to the same value. It was loaded by simply copying the program file into memory and jumping to it.
At Zortech we realized that it wasn't necessary for all the segment registers to be the same - it's just that the program file needed to be less than 64Kb. This meant the code plus the statically initialized data had to be less than 64Kb. SS and DS, though, could be changed to the start of the initialized data, and had access to 64Kb beyond that, enabling significantly larger programs while still using near pointers.