Normally static recompilation of video games breaks down due to self modifying code. If you're going to have to have go through the motions of JIT anyway (and in places where, if they're going through the motions of self modifying code, it's probably computationally expensive code at that), then you might as well go whole hog. That way you only have to build one kind of compiler, and you can take advantage of profile information for everything.
But the 360's security model specifically out rules self modifying code. And this is enforced by the hypervisor; the system is W xor X, and the transition of a page from W to X has to be accompanied by signatures signed by Microsoft's private key. Nobody, including the kernel, can modify their code.
That might make this one of the first platforms where the static recompilation scheme might make sense in the long term architecture of the emulator.
Self modifying code hasn't been common in video games for 20 years. The main problem is that static recompilation requires accurate alias analysis-- knowing what values a given pointer might have at a particular point of execution. This is especially important for indirect calls through function pointers (C++ virtual functions), where entire chunks of code might be missed.
But you can still have video games implement their own form of dynamically loaded modules, which (to a jit) is the same problem as self modifying code. I know several PS2 games do this.
But it's still something to be aware of, so that you don't have to make major architectural changes to your jit so that one game will work. It's a million and one little things like this that make emulators hard.
Yeah, you'd probably want an indirect table for inter-page calls, with maybe a soft TLB for this translation if the profiler gave you good results. You might be able to bank on the idea that any processor you'd be running the generated code on will have so much better branch prediction and speculative execution than the 360, that you'll be able to fit in the cracks even with this extra overhead.
It always seemed like a poor security choice to me, but some game binaries are inexplicably massive and break up easily depending on what phase of the game that you're on.
Sure, in the PS2/gamecube era custom dynamically loaded code mechanisms were common.
But on modern consoles (360, PS3 and later) the OS bans games from loading/generating code. To load code they must call out to the OS, which loads it, checks the signatures and puts it in write protected memory.
It's this security mechanism which makes static recompilation viable for 360/PS3 games.
I wonder if they did this on purpose? Self modifying code has been known to break emulators. Part of a future-proof scalable platform is to provide a way to run the previous generations games and emulation is a really good way to do that cheaply.
If they can emulate a PowerPC system then they can run AmigaOS and MacOS/MacOSX for PowerMacs to run some legacy programs on Windows, Linux, Modern MacOSX, iOS, etc.
There were rumors that Microsoft would port Windows 10 to the XBox One and then convert XBox games to Windows 10 via the Windows Store, etc. Then you could buy or rent XBox, XBox 360, and XBox One games.
Apple had a Rosetta software that ran PowerMac stuff on Intel Macs until 10.7 removed it IIRC. So Apple has the patents and IP for that.
There was also an old ARDI Executor program that ran old Mac 68K stuff in Windows, Linux, BeOS, etc.
If you just need a PPC MacOSX emulator, look at PearPC. My laptop couldn't emulate a G3 at full speed in 2004 when I first tried it, but I'll bet a modern CPU would have a better chance.
This project doesn't look like it's emulating PowerPC, though. It's tracing the binary, disassembling it, and generating equivalent C++ code. It's a technique called static recompilation.
I suppose "incomplete" depends on what you're doing. It worked pretty well for playing around with all those years ago, and it at least still compiles. I don't have any PPC images around to throw at it.
Well, if you need another PPC Mac OS X emulator, look at QEMU. QEMU's Mac PPC support has improved by leaps and bounds recently. It's now a pretty competent Mac emulator for OS 9.0-10.5.
The Rosetta software (original name was QuickTransit) was developed by Transitive Corporation and licensed by Apple. They were acquired by IBM in 2009.
I'm pretty sure it relies on the fact that the xbox 360 enforces strict W^X.
Which means all the code on 360 games is clearly marked in the binary and it's impossible for the game to do anything weird like generate or decrypt new code on the fly.
That doesn't actually guarantee no funny business, someone could jump into the middle of a multibyte instruction. Generally speaking games shouldn't be doing that so it should be fine, but there's nothing stopping it. It does however show the problem that you can't always actually tell where some code starts within a block.
Note: I'm not talking about the Xbox specially. Its binary format my actually make it clear where every piece of code starts, or there may be other ways of telling.
> That doesn't actually guarantee no funny business, someone could jump into the middle of a multibyte instruction.
Does PowerPC use multibyte instruction that don't have to be aligned? It would be very atypical for RISC processors, but if you can provide evidence that these exist, I will believe you.
(sidenote: I am aware that Thumb and in partiular Thumb-2 on ARM uses instructions of variable length)
I wasn't talking specifically about the PowerPC architecture, I was just pointing out that just because a computer supports 'W^X' doesn't mean you can always know exactly what instructions a program will run. You need stronger guarantees then that (Like, for example, that you can't run unaligned instructions, and that all instructions are the same length). You could never really write a recompiler like this for x86 code even though you can mark data with the NX bit, because you can't tell where the instructions start, or whether or not a jump may end-up half-way between an instruction.
That said, when I wrote that comment I didn't know for sure if the XBox 360 CPU had that or not. Looking it up, it's still not 100% clear to me but I would wager no. There are extensions to the PowerPC architecture that have shorter instructions (It appears to be somewhat similar to Thumb), but I was unable to tell whether or not the XBox 360's Xenon supported that. Being that clearly people more familiar with the XBox 360 then I am believe compiling the code beforehand is possible, I think it's likely the XBox 360 doesn't have the optional support for variable-length instructions.
Still, there are some RISCs that have variable length instructions [0], but as you'd expect the majority of them are fixed length (Though, especially in this context RISC vs. CISC can be pretty arbitrary). Even just talking about Game Systems not all of them run RISCs. The 6502, z80, and 68k are all classified as CISC and have variable-length instructions sets, and all of those are featured in lots of Game Systems. So it is a problem you have to consider if you're going to be writing a JIT/recompiler/etc. for those architectures.
I can confirm. The PowerPC variant used in the 360 (and PS3) only supports 4 byte instructions and throws an exception if you try to force it to execute instructions not 4 byte aligned.
Both the PS3 and 360 are in a unique position where they both have strict W^X, OS enforced code signing, and an arch that doesn't let assembly programmers do weird things. This means they are both extremely viable targets for static recompilation. The programmers have zero ablity to do funky business.
Also on the PS3 (I'm not sure about the 360) the ABI requires the complier to generate and ship with a list of every single function entry point. Which basically means you are static recompiling with easy mode on.
However, you are correct about basically every other console, static recompilation is not easy. (Though it's my theory that static recompilation might be easier on the Atari 2600, because the limited ram makes it really hard for self modifying code to exist.
> Though it's my theory that static recompilation might be easier on the Atari 2600, because the limited ram makes it really hard for self modifying code to exist.
My gut feeling is you probably wouldn't get great benefits out of that since 2600 code relied on a lot of cycle counting to get anything other than pong out of the graphics hardware.
The amortization of instruction decode and dispatch represented recompilation (whether static or dynamic) probably doesn't get you much. You've got to do most of that bookkeeping either way to make the effects of the CPU cycle accurate with regards to the rest of the system. Pretty much every 2600 title relies on this.
So, in addition to lack of self modifying code, PS3/360 emulators benefit from hardware that's complex enough that it essentially has to be treated as non-deterministic by the original game developers (within reason). ie. code written against them uses explicit synchronization primitives rather than cycle counting. Sure you have to get things in the right ballpark (stuff like the bugs fixed in Dolphin by not making EXI->MemCard transfers happen instantly), but there's no code saying 'nop for 12 cycles, and expect the system to be in exactly the right state'.
> There were rumors that Microsoft would port Windows 10 to the XBox One and then convert XBox games to Windows 10 via the Windows Store, etc. Then you could buy or rent XBox, XBox 360, and XBox One games.
You can sorta do this already. Xbox One is running a Windows 10 core and major releases area aligned with windows. You can play significant percentage of Xbox, Xbox 360 and UWP games (like Astroneer for example) on Xbox One.
Yeah, the GPU is a custom part not found elsewhere (IIRC the NV2A GPU is meant to be somewhat similar to the GeForce 2 and GeForce 3).
However, there is an emulator being worked on for the original Xbox called CxBx Reloaded, which is a continuation of the CxBx emulator. To give an idea of how far along it is, here's some recent footage of JSRF:
Yeah, although not any more obscure than the GC/Wii GPU really.
I'd argue that the fact that the OG Xbox's homebrew scene used the illegally acquired official XDK for the mos part played a part. This meant that they didn't get as deep into documenting the hardware at that early stage, and instead relied on closed source libraries.
The same could be argued about the Dreamcast where there SDK was concerned but that has been emulated. Not perfectly but pretty well.
I have read it was not just interest in the console but the GPU.
I wonder if this type of project would be easier for the XBoxOne since it uses x86-64.
I realize x86 != PC as shown by the team that ported Linux to the PS4, another x86 system that has almost no standard PC hardware in it. So I'm sure trying to run XBoxOne binaries on other x86 systems still wouldn't be trivial.
Of course, you don't need to write the whole thing. Both parts (XBox 360 and PS4) are at least similar to existing Radeon GCN parts, and we have good quality open source Radeon drivers which outperform the closed ones.
If you want to run PC executables on PS4 and Xbox One (Xbox 360 has been made well before GCN, it's an AMD VLIW (R300 perhaps?) so I figure you mean Xbox One) you can probably make drivers from the Radeon drivers. However it does not help with running PS4/X1 executables on PC in any way other than knowing a bit about underlying hardware.
My brain scrambled while trying to write One. I have trouble keeping the names straight because they called the XBox Three the XBox One. I meant the new XBox.
I think I maybe misunderstood which way nejenendn was talking about. I thought nejenendn was referring to the difficulty of running Linux on a PS4/XBox One.
Sorry, i was speculating on running xbox one and ps4 executables on the desktopp. Assumably microsoft would add support for xbox executables in this scenario. I don’t see Sony ever doing that; we’d need something like wine, or an actual emulator/dynamic recompiler.
This is amazing -- seems almost like a better way to play an Xbox 360 game on PC, without an emulator. Could even be what Microsoft wanted, for better retrocompatibility support.
I wonder if it can also be done to other consoles? Let's hope it gives good fruits.
For a lot of consoles, particularly older ones, this tends to yield worse performance than a JIT. Between different memory layouts, weird threaded code, jump tables, (and especially) self modifying code, a JIT can do a much better job building efficient code since it has all of the runtime information.
For newer consoles which more closely resemble PC hardware, things are definitely better; the current generation will basically look like the Starcraft port; an API compatibility layer.
Yeah - as evidenced by the NES story the other day, the elegance of AOT blows up completely as soon as there's any form of self-modifying code or even memory arithmetic tricks or jump tables. It's all possible to work around, but JIT is cleaner and more flexible in the long run.
In other words, you create a JIT to run the executable, building a profile of the execution paths at runtime, and use that profile to guide a static decompilation process. That way it should be easier to identify the sections of self-modifying code, as well as model the behaviour of this code.
The only difficulty I can see is how much time it might take to map out all the code paths, but in principal it's possible, and there may be some efficient approaches for doing so.
I've always wanted to find time looking at recompiling the Xbox Live Arcade (XBLA) titles from the 360 to Windows. The games are just dotNET applications which run against a variation of the standard libraries.
How is it possible for one single person to figure all of these things ? I am new to the industry ( 2 years ) and have only worked on Front End and Back End ( APIs) .
But the 360's security model specifically out rules self modifying code. And this is enforced by the hypervisor; the system is W xor X, and the transition of a page from W to X has to be accompanied by signatures signed by Microsoft's private key. Nobody, including the kernel, can modify their code.
That might make this one of the first platforms where the static recompilation scheme might make sense in the long term architecture of the emulator.