Well, the first question you want to ask yourself is what is the purpose of the VM? Or what do you want to emulate?
VM doesn't necessarily means writing assembly code. In the context of games, there are 2 common emulation concepts:
- Emulating hardware - could be old game system such as NES emulator, other computer architecture such as commodore 64. For this type you really need to know the architecture you are planning to emulate - basically learn the assembly language, how it is encoded, memory architecture, video and sound systems, etc. Depending on your choice of HW to emulate, this can become really tricky.
- Emulating game engines - this type is similar to ScummVM. In this case, you write an emulator for a scripting language. There are some similarities to the first kind, but the main difference is that you don't need to understand the hardware internals, which greatly simplifies the problem. For old game systems (Scumm, SCI) - it's a simple mapping of the scripting instructions to some high-level Java/C++ calls, no need for assembly.
There are also other kind of VMs - for example, one can write a virtual machine for HLSL/GLSL shader execution. Very simple, but limited and pretty useless without proper context.
There's really no reason to learn x86 assembly (at least not at the beginning). If you are targeting some old architecture/game engine - your current PC is thousands of times more powerful, so even a decent interpreter would do.
Java will do just fine (in fact, Java runs on VM with JIT compilation).
You can take a look at the ScummVM source code and use it as an example (C++, uses OpenGL for the video system).
If you are just starting, I suggest that you:
- Work on Scumm. You have reference code for it, which means it's well documented.
- Start with basic interpreter, just to make sure you understand how the system work.
- Once it's working, identify the performance bottlenecks and optimize them (use HW accelerators for graphics/sound system, learn and apply x86 optimizations, etc.).
- That should basically do. If you still feel unsatisfied, start learning JIT compilation(JIT == Just In Time). This takes you to a whole new level of x86 low-level understanding.
Edited by N.I.B., 17 August 2014 - 12:47 PM.