Sign in to follow this  
LackOfGrace

Send code to compiler

Recommended Posts

Hi! i just got this crazy idea. for my 2d engine i was planning to have a script language to handle object updates and such, but discarded the idea because it would be to slow. now, i where thinking about creating a developer program for my engine. here is the idea. i could let the user create object types and write code for special events (like update , collision etc ) and then the app will stich together the code and send it to a compiler, wich will compile the code to machine code. it would remove slowness from having a script language, but it would grant the engine a better development solution. so, is there anyone that has created a compiler or used one without the overlaying IDE. i would need to know how to send header files and how to send libs for linking. /Victor

Share this post


Link to post
Share on other sites
Are you sure a scripting solution is too slow for your needs or are you just assuming? Think about that, and if you still believe it's too slow then you need to evaluate how you plan to use scripting to control behavior.

Scripting isn't intended for low-level implimentation of behaviors, its intended more for high-level behavior and response. The example I always use is this: A Script might say "Go to this position" or "follow this character" but you don't impliment the path-finding (A* or what-have-you) in the scripting language. 99% of the time if its too heavy for scripting, it wasn't a good match for scripting in the first place.


Now, if you truly do have a need for customizable plug-ins with native-code speed, you'll probably be best off looking into a Plug-in architecture using DLLs, rather than attempting to generate native code on the fly. The DLLs will have to be compiled seperately, so its not a run-time process like scripting, but you can provide a small "SDK" of sorts for your application. This SDK might contain something like an "IEntityController" abstract base class, which users can derive from and compile into a DLL which the application can load, along with their standard scripts.

Attempting to generate code at run-time is generally a bad idea unless you're an expert programmer. Even assuming you use a stable, existing library to generate the code there are still many difficulties. The primary difficulty is that your IDE's debugger will not be able to help you at all for run-time generated code. At best this means you are left to mentally evaluate and debug any flaws. It gets worse though because, eventually, harder-to-track bugs will probably require you to look at, and understand, the machine-code it spit out.

Share this post


Link to post
Share on other sites
Quote:
Original post by LackOfGrace
so, is there anyone that has created a compiler or used one without the overlaying IDE.


Most compilers are available without any sort of IDE. MSVC++ 6.0, 7.0, 7.1, and 8.0, along with gcc and borland (plus probably several others) are all usable just from the command line.

I'd also like to reiterate what ravyne2001 said: Scripts should be used to control high level behaviors, and have you actually written them and tested their performance before judging them as too slow?

Share this post


Link to post
Share on other sites
You don't "send" header files, they're included into the source files by the preprocessor. There's two stages to creating an exe or dll:

  1. Compiling
    The compiler compiles one module (.cpp file) at a time, and the preprocessor includes any header files needed into the file. By the time it gets to the compiler, it's just one large file with all the preprocessor definitions expanded. The compiler generates an intermediate file for this module (a .obj usually).

  2. Linking
    The linker comes next, after every source file has been compiled to object code, and links them together into one chunk of code along with any .lib files and generates the rest of the required information (header, import / export table, and so on)


You can have a look at the build log once you compile a program with Visual Studio 2005 if you want to see what command line it's generating for cl.exe (the compiler) and link.exe (the linker).

However, using machine code for scripting isn't usually a good idea as it opens up all sorts of security holes and completely removes the safety of scripting languages (If a script crashes, it doesn't usually bring down the host program).

Share this post


Link to post
Share on other sites
as i see it there are 3 possible ways of doing what i want.

1. DLL
let the user create a dll that will be run by the main application

pros:
fast

cons:
wont have extensive access to engine
(could be a good thing, though it would take some time to release features that the user would like)

2. Script

pros:
simple for the user

const:
writing my own takes time (could use others)
slow
would need an extensive database with commands

3. Compiled code

pros:
total access to engine
best speed

const:
need to write a interface for programming (will be doing this anyway since the user would need to create material files and such)

so, i really think that creating a ide for the engine would be really nice.
and event driven programming (the user gets to implemnt only update functions etc) is a proven concept (actionscript)


and i think it would be really funny to see if it was doable in a good way.


anyone have any articles or should i resort to command line references?



Share this post


Link to post
Share on other sites
Quote:
Original post by LackOfGrace
as i see it there are 3 possible ways of doing what i want.

1. DLL
...

cons:
wont have extensive access to engine

...
2. Script
...

cons[sic]:
would need an extensive database with commands

...
3. Compiled code
...

pros:
total access to engine


These are all different faces to the same problem. In all 3 cases, the DLL, Script, or compiled code will only have access to what you expose. Compiled code will not have "total access to engine" magically, you still need to expose function and data addresses in some fashion, just as you would with a DLL or scripted solution. The main difference between the 3 is that DLLs and scripts have developed proven solutions to accomplish such exposures.

How do you intend to expose your functions/data to your generated code? How do you intend to expose your generated code/data to your engine? How do you intend to accomodate the various calling conventions?

You seem to believe that generating code will magically take care of all the details and work just as easily as your compiler of choice, but that is not at all the truth. You'll have to do all the lifting yourself, make no doubt that what you would be in for is nothing less than creating a rather full-fledged compiler, linker and then some.

Share this post


Link to post
Share on other sites
wouldnt the compiled code have access to the system singleton?
if it would the whole engine would be open.

but it seems that pretty much everyone thinks that the compiled way is an unnecessary step, so ill fall back to using the dll implementation.
my idea was to create a usable IDE.
wich would give my engine an advantage vs other 2d engines.

well, ill create the graphical parts of the ide first (material creation, particle systems etc) and then ill do a little research on how hard / time consuming a compiler system would take to build.

Share this post


Link to post
Share on other sites
Just wanted to point this out:

The Unreal Engine's scripting language "Unreal Script" ist bytecode-compiled and according to Epic runs about 20 times slower than compiled C-code. Yet, a lot of the game logic in Unreal and UT is written in UnrealScript and both games are actually pretty fast.
So what's been said earlier: if a scripting language is too slow for your needs, you're probably trying to do something in scripting that you should be doing in-Engine.
Of course there are also scripting languages that do just-in-time compilation into machine code which gives you additional performance, but when done properly, the performance even of a (bytecode-) interpreted language should be sufficient for pretty much all gaming needs.

Share this post


Link to post
Share on other sites
Quote:
Original post by LackOfGrace
wouldnt the compiled code have access to the system singleton?
if it would the whole engine would be open.


No, not unless you expose it. Remember, executable code retains no symbolic information what-so-ever. Class, Variable and function names no longer exists, they are only addresses at this point. Class and struct definitions go out the window as well, only the class's constructor knows its size, and only the compiler knew how it was laid out in memory.

There is absolutely no way to hook into an executable and call "System::GetInstance()" in a straight-forward fashion. You need the address of System::GetInstance() and you need to know its calling convention. You need to know what parameters are expected and how they are arranged before jumping to the function (In this case, there are no parameters), next, you'll need to know what the function returns and how it returns it. Once you've gotten this far, you need to know how the returned object, presumably a pointer to a System object in this case, is laid out and where each of it's member functions are... oh, and don't forget to pass that hidden "this" parameter to all member functions.

This process must be repeated for every class, structure, function and variable the generated code needs access to. This is tedious and painstaking at best and is highly susceptible to mistakes.

The bottom line is that its not at all practical to impliment run-time code generation for scripting purposes. Most popular scripting solutions used in professional games today, such as Lua or UnrealScript, run on byte-code interpretters, usually implimented as either stack-based virtual machines or memory-to-memory virtual machines, even high-end games on limited devices (I know a recent FPS on the PSP uses Lua) use scripting, but they use it at an appropriately high level.

Share this post


Link to post
Share on other sites
Options 1 (DLL) and 3 ("compiled code") are the same. Unless you plan on magically reinventing the DLL mechanism, which is stupid (and more work than you're ready for). You can't take the object files produced by the compiler you "send" the code to and "embed" them into your currently running executable... or rather, you can (sortof) but the way you do that is via a DLL. Any other method would be very over engineered for the apparent scope of your game.

Scripting implementations, done well, are not as slow as you seem to believe. They can be quite efficient when used for the right things. It sounds like you're really suffering from a combination of (a) using them for the wrong things and (b) making assumptions about the performance of scripting in general.

Both are bad ideas. Use an existing scripting language, the performance will very likely be fine as long as you don't try to make the entire "engine" super-scriptable-zomg-you-can-change-anything (in other words, keep the scripting high level).

Share this post


Link to post
Share on other sites
i think i didnt explain my idea satisfyingly. (i my mind dont work well at the end of the day)

i didnt plan to write the accual compiler myself

i where planning to have a simple program stored as text inside the development app.

then, when the user wishes to compile, the program adds the user specific code into the code that resides inside the application.

the application then compiles the code using an appropriate command line compiler (probably visual studio since i belive its free).

and voila ; an executable file

i very well know that im not capable of creating a compiler from scratch by myself, therefore i would use an existing one.

personally i cant se any problems with this approatch (only that it will take some time to finish neeable features like intellisense) , on the other hand - i havent throughoutly checked how to send stuff to the commandline compiler.

Share this post


Link to post
Share on other sites
Quote:
Original post by jpetrie
Scripting implementations, done well, are not as slow as you seem to believe. They can be quite efficient when used for the right things. It sounds like you're really suffering from a combination of (a) using them for the wrong things and (b) making assumptions about the performance of scripting in general.


i would like to build an interface similiar to flash's actionscript ( every object have a set number of functions the user can implement (update,press etc)

i asked around in scripting forums for timing data.
though i couldnt recive any timing data i got told that x scripting language couldnt do what i wanted ( i want the engine to handle alot of objects ( ~2k ) (wich it can when using machine code) ).

therefore i do not consider using a script language since (for my needs) it will be to slow.

i would be really happy if im wrong with this. if you have any good speed data on a specific scripting language, please point at it. if you dont have any speed data please dont tell me to use a scripting language (already pretty many posts telling me to do so).

Share this post


Link to post
Share on other sites
You want a specific example of a sufficiently fast scripting language? C#.

You can use C# as a scripting language (e.a. the user writes scripts and the VM compiles them JIT for you to use them). I even know of games that use C# as a scripting language (in fact, it's one of my options too). Cellfactor is an example of a game that has lots of dynamic objects on the screen at any one time. Check any video you find on that game. The DirectX SDK has an example on how to implement a C# scripting system, but it still uses the preliminary syntax and doesn't use proper reflection mechanics, so it runs slower than engines doing it right. Still, it's plenty fast.

Share this post


Link to post
Share on other sites
what this *almost* sounds like is "dynamic recompilaton"... i.e. compile something at run time because interpreting it would be too slow... this technique often comes into use for emulators of newer systems (i.e. Nintnedo64, Sega Saturn, PS1, PS2, Gamecube) and becomes necessary to get the needed run time speed, but in these cases, the stuff getting compiled is not just "game logic" but a program written in some low level (machine) language... if you need for stuff of your engine to be accessed by other's to provide fast modules, your best best is .DLL's (or if you are using Unix, .so) and ldlt is your friend (if you are using GNU tools)... the only reason I can see for needing your own compiler rather than interpreter is if you want these add on moodules to be the same across all platforms you are planing on supporting (for .dll/.so solution each platform will require their own file from the 3rd party) but really, what kind of stuff are you planning on putting there? if it is jsut logic, a script is worlds better both to use and read and deal with....

Share this post


Link to post
Share on other sites
Quote:
Original post by kRogue
what this *almost* sounds like is "dynamic recompilaton"... i.e. compile something at run time because interpreting it would be too slow


maybe i got this wrong...
the development app should be a standalone app wich would compile a standalone exe file. not dynamicly interprenting or compiling , just a compilation that our standard c++ development studio does.


C# sounds interesting, i will look more into scripting with it

Share this post


Link to post
Share on other sites
Binary translation is a kludge; dll are just a heavy handed way to load binary bits, re-map them as executable and trampoline.
http://llvm.org/, JIT away, be done (that or use a fancy linker script).

Share this post


Link to post
Share on other sites
Quote:

then, when the user wishes to compile, the program adds the user specific code into the code that resides inside the application.

Uh huh, that's exactly what I'm saying is not possible. Use DLLs to do that, otherwise all you'll be doing is reimplementing the DLL mechanism in a broken way. Options 1 and 3 are the same.

Scripting can easily handle 2k+ objects, it's just a matter of what you want to do when you "handle" them. It really sounds like you're trying to use scripting for the wrong reasons, to make some kind of supergeneric game construction tool. This is ill-advised, as it sounds like its beyond your current experience level, however, the way do go about that kind of very low-level client-side behavioral modification is via plug-ins implemented as DLLs.

Share this post


Link to post
Share on other sites
Look, the long and the short of it is this: Either scripting is fast enough, or you're trying to do something that's not suited to scripting. Period. If you really need to allow custimization that's more performant than what scripting offers, then DLLs are the answer. Period.

In fact, from the description you gave of wanting to follow the OnEvent() model, DLLs are very well suited to what you want to do. Have your engine provide a means to register these EventHandler objects and manage communication between them. Your EventHandler "SDK" can consist of something as simple as a handfull of Abstract Base Classes (Interfaces) for EventHandler Objects and access to the engine's registration and communication mechanism.

Furthermore, from a practical standpoint, you really can't rely on third-party software that you cannot distribute yourself and is not commonly available. You can't simply take Microsoft's executable and ship it with your program, because you'd be violating the rights they've granted you in the EULA. You could tell your users to download the command-line compiler toolkit themselves and give them a URL, technically, but do you really want to say "To play my game you have to download this compiler"?

Having a compiler as a requirement is a pretty rediculous situation for anything other than an IDE or code generator [grin]

Share this post


Link to post
Share on other sites
TinyCC can do just that - compile and run C on-the-fly as a scripting language - but it's limited to i386 and there haven't been any significant developments in almost 2 years.

Share this post


Link to post
Share on other sites
Well, there may be options such as tinyCC that make the distribution concerns go away (presuming the liscense allows for it) but there's still a phalanx of reasons why its a bad idea:


  • You still have to jump through hoops to expose your engine. Except now you have to expose what is presumably a C++ engine to C-style scripts.

  • You have no way to sandbox the code to prevent crashes or malicious scripts.

  • C makes for a pretty terrible scripting langage

  • Many more


Share this post


Link to post
Share on other sites
Quote:
Original post by jpetrie
Quote:

then, when the user wishes to compile, the program adds the user specific code into the code that resides inside the application.

Uh huh, that's exactly what I'm saying is not possible. [...]


You're misunderstanding the point. He would write the skeleton program himself (in some high level language), cut it up, throw in some user-written code (in the same language) and then compile it as a whole. It's flawed in many ways, really.

Quote:
Original post by ravyne2001
If you really need to allow custimization that's more performant than what scripting offers, then DLLs are the answer.


They're not the only answer anymore these days. Consider my previous post about C#. DLL's also allow for awfully unsafe code injection, so for games, you might want to consider the alternatives first. ravyne2001 point remains perfectly valid however, the OP is trying to find a means to a wrong end.

Share this post


Link to post
Share on other sites
Quote:

You're misunderstanding the point. He would write the skeleton program himself (in some high level language), cut it up, throw in some user-written code (in the same language) and then compile it as a whole. It's flawed in many ways, really.

If he's augmenting the source code for the game/engine/whatever with source code written by the user, that's not so much scripting as it is modifying the code base, since the entire project needs recompilation, and avoiding such far-reaching recompilation is part of the point of scripting in the first place... so yea, if that's what he really means then the concept is both flawed, and misnamed.

Share this post


Link to post
Share on other sites
Quote:
Original post by Todo
Quote:
Original post by ravyne2001
If you really need to allow custimization that's more performant than what scripting offers, then DLLs are the answer.


They're not the only answer anymore these days. Consider my previous post about C#. DLL's also allow for awfully unsafe code injection, so for games, you might want to consider the alternatives first. ravyne2001 point remains perfectly valid however, the OP is trying to find a means to a wrong end.



The .net languages are an option, however for me at least it is only an option when the application itself is written in one of the .net languages to begin with. My reasoning is that it seems unwieldy for a C++ application that likely has an executable in the mid-hundred Ks, to be dependant on a 30-40MB run-time environment. Even though .net is pretty standard these days, not everyone has it yet. Its a bit like the VB6 runtime all over again.

.NET is a really sweet technology no doubt. Something like VB.net would make a pretty cool scripting language, and its actually not terribly hard to compile your own scripting langauge into .net bytecode (I've got two books on that topic and another on the .net assembly language). Most popular scripting languages have .net counterparts as well. There's Lua#, and Python implimentations, for instance. For a .net project, this would actually be my prefered method.


With a native-code engine however, I still see some combination of DLL + Scripting language as the optimal, prefered solution.

Share this post


Link to post
Share on other sites
Quote:
Original post by ravyne2001
Quote:
Original post by Todo
Quote:
Original post by ravyne2001
If you really need to allow custimization that's more performant than what scripting offers, then DLLs are the answer.


They're not the only answer anymore these days. Consider my previous post about C#. DLL's also allow for awfully unsafe code injection, so for games, you might want to consider the alternatives first. ravyne2001 point remains perfectly valid however, the OP is trying to find a means to a wrong end.



The .net languages are an option, however for me at least it is only an option when the application itself is written in one of the .net languages to begin with. My reasoning is that it seems unwieldy for a C++ application that likely has an executable in the mid-hundred Ks, to be dependant on a 30-40MB run-time environment. Even though .net is pretty standard these days, not everyone has it yet. Its a bit like the VB6 runtime all over again.

.NET is a really sweet technology no doubt. Something like VB.net would make a pretty cool scripting language, and its actually not terribly hard to compile your own scripting langauge into .net bytecode (I've got two books on that topic and another on the .net assembly language). Most popular scripting languages have .net counterparts as well. There's Lua#, and Python implimentations, for instance. For a .net project, this would actually be my prefered method.


With a native-code engine however, I still see some combination of DLL + Scripting language as the optimal, prefered solution.


You're absolutely right. The runtime would be the main "disadvantage", especially considering cross-platform development. Mono is getting there, but it's still like "playing with fire" at the moment.

That being said, I opt for Lua, too.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this