LLVM Library for Scripting

Started by
6 comments, last by _the_phantom_ 10 years, 1 month ago

For my engine I would like to be able to have scripting in an any arbitrary language. I have heard that LLVM is very efficient and also supports JIT compiling and can be used as a virtual machine. It can be called as a library but, from all the documentation I've read for it, it seems like the library can only be used for compilers and not for scripting features. My question is, does that the library only supports compilers?

SAMULIKO: My blog

[twitter]samurliko[/twitter]

BitBucket

GitHub

Itch.io

YouTube

Advertisement

LLVM started life as the "Low-Level Virtual Machine" but its contributors soon enough realized that the work they were doing was much more useful and feasible as the basis for a compiler. The LLVM moniker at this point is like a vestigial appendage -- its still there, but contributes in no useful way to its modern form. It is no longer a VM in any way, shape, or form.

Furthermore, I would say that your goals are misguided and your presumed implementation thus misinformed. One does not script in anything other than a scripting language -- there are languages that blur the line to be sure, but for the most part anything you would care to 'script' in the proper sense is best done using a language designed to support those use-cases, rather than by a language that is general enough to be designed first as a stand-along programming language. And likewise, anything you would care to do that requires real performance should be written in a 'real' programming language and linked into your binary somehow, perhaps through a plugin interface. Anyways, the point is that you would no sooner write your cut-scenes in C++ than you would write your low-level geometry management in Lua.

Scripts control high-level game behaviors, configurations, macro-bindings, and similar things. They do not usually do those things by implementing them directly in the scripting language, but by leveraging an API exposed by the host program and well-defined in its capabilities. To give scripts the kind of low-level, unfettered access to program execution that supporting any old language would require would be a massive security hole. If you accept that scripts must work through a well-defined API, then the low-level idea of a language melts away, and you are really talking about 'binding' that API to some chosen scripting language. This is one of the reasons that Lua is so incredibly popular -- it has powerful and easy-to-use binding libraries.

The popular choices for application scripting languages are Lua, Python, and Ruby. I wish JavaScript was more common among them, but unfortunately no one really has a good solution for JavaScript execution that's easy to integrate into a host application.

throw table_exception("(? ???)? ? ???");

Ravyne is spot on, but you can use other languages for scripting depending on target platform.

For example if you are running on windows you can actually compile C# on the fly and use it as a scripting language.

You could use LLVM, but it would be a hell of an effort. We have our own LLVM compiler based on gcc and getting that to work was a major effort. Getting the LLVM to then execute on a target platform is another major effort. Doing all that just for a scripting langauge, ..... priceless smile.png

I'm not certain I entirely agree with the idea that a script language is a specifically created language for scripting. Scripting is just a conceptual separation of responsibility, I have used the concepts of scripting through all sorts of languages, C/C++ included, the language hardly matters. The language does matter when you want to measure the benefits, as such most compiled languages are not all that useful since the turn around times negate the benefits of avoiding compiles. But, depending on your needs, even assembly can but used for scripting, not that I really suggest it. :D

Anyway, having used LLVM as the backend to a script system, it has good and bad involved. The good is that any language supported by LLVM such as C, C++, D and flavors of Ruby, Python and others can be used interchangeably somewhat like CLI (Mono). And of course, if you have a language already going to a VM, which was our case, switching to the LLVM backend sped things up considerably when the optimization passes were used. The bad though, as stainless says, the level of effort was considerable, no doubts about that. Another bad item is the turn around time, the optimization passes are considerably slower than most script language compile/interpret startup times. It is not horrible but it is notable.

So, all said and done, the choice of scripting is really more a case of balancing your needs/desires versus the amount of effort you want to put into it. LLVM is not really meant for scripting, but the advantages it offers could be used to argue it as a good solution in some cases. Again, this is not endorsing the direction, it is always going to be based on your requirements and amount of time to work on such things.

I tried to embed SpiderMonkey and V8 at first, but my attempts to build them for msvc weren't fruitifull so far, same for Mono.

Squirrel is quite handy, and has a really cool syntax.

I tried to embed SpiderMonkey and V8 at first, but my attempts to build them for msvc weren't fruitifull so far, same for Mono.

Squirrel is quite handy, and has a really cool syntax.

my thoughts:

looked at LLVM before, but it looked a bit much like using a sledgehammer for a job better suited to a clawhammer (ex: typically stuff involved in running script code). what it provides isn't exactly a great fit for what makes sense for a typical script-language backend.

there is kind of a common mindset though of people using the biggest / most powerful / most "industrial strength" tool, regardless of whether or not it is most appropriate for the task. using something overly big and heavyweight though may actually make things worse in terms of development/maintenance work and sometimes also performance (say, if the workload of the task does not match the workload the tool was developed for).

SpiderMonkey and V8: dunno. usually GCC -> MSVC porting is doable, if sometimes annoying, though it requires a bit of a willingness to "dig in" and sometimes partly rewrite things, ... wouldn't think it would be as much of an issue for SpiderMonkey as IIRC Mozilla uses MSVC for their Windows builds, so the code should be reasonably free of GCC-isms. what little I had looked at them, things seemed "fairly reasonable" at least.

Mono: ... I have looked at the code for this before. at the time I found it almost absurdly questionable/bad even for performance-oriented VM back-end code (clear abstractions were largely absent, pretty much everything was tangled up in everything, ...). now, I am less certiain, as dealing some with performance-oriented code may have increased my tolerance for "general nastiness" to some extent (though even as such, I still prefer to try to maintain things like consistent internal API abstractions and uphold naming conventions and similar...).

not really looked too much into Squirrel personally.

in my case, I am using my own VM technology, and my efforts are pretty well invested in this. major/core pieces of codebase architecture and infrastructure are not something that can be easily swapped out without a lot of pain. more so when one considers that "little things" done by one things may not be done, or may be done a fair bit differently, with something else.

also partly because it has been developed incrementally in an on-off manner over the past 15 years or so, and 15 years ago, there weren't as many good options available for scripting, and at the time it seemed like such a simple task to throw together a naive interpreter.

maybe funny, or sad, or just some sort of personal fault, that things often start out so small/easy/simple, but often over a period of time seem to mutate into something a bit far from being nearly so small or simple anymore.

For me, I don't like scripting in the traditional sense. I know, I know, I'm the weird one. I've come to accept that.

For me loading in text files in whatever format in a release build has too many issues. Hackers can alter the text files and change the game behaviour, you have the overhead of parsing them, etc. etc. etc.

I like the ability to script while developing the game though. When I was doing XNA coding I would use C# for scripting while I'm developing the game. Then compile the scripts into the game for the release. That worked well for me.

What do people do these days?

For me loading in text files in whatever format in a release build has too many issues. Hackers can alter the text files and change the game behaviour, you have the overhead of parsing them, etc. etc. etc.


Well, for starters you don't load text files in release; you compile the script to whatever byte code format the language uses and then use that - this is why Lua is a popular choice as it's a light scripting system, flexible, bindable AND the frontend can accept both text and byte code as source data.

This topic is closed to new replies.

Advertisement