Sign in to follow this  

Writing new scripting language - suggestions needed

This topic is 3584 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hi guys, I'd like to start works on some small scripting language that would be specifically well suited for game dev. The language has to: - be possibly simple (though not as simple as Lua for example) - have basic OO support (single-inheritance model, maybe interfaces; built-in rtti; garbage collected) - be very expressive (through explicit/deductive type definition or even typeless); by expressive I mean native support for structures a'la tables in Lua, tuples, ability to create objects without defining its class, etc. - have support for states (a'la UnrealScript) - support latent/resumable functions - have easy binding with C++ - have good debugger Besides that also: - stay platform independent (bytecode interpreted) - be possibly efficient Things I haven't decided yet are: - whether the language should be typeless (see Lua) or maybe strongly typed (see Ocaml) - whether it has to support multi-threading / synchronization schemes and which if so Also I still haven't decided what the best syntax and exact feature set would be, so I'm asking you to give me some ideas: What do you think are you missing in scripting languages available today (Lua, Squirrel, Game Monkey, Angelscript, UnrealScript, other)? What would you change / extend but it's difficult? What exact syntactical constructions, features are you missing in these languages? Any other ideas - feel free to post them here. This is going to be an experiment that I hope might result in something useful for anyone. My experience writing compilers is relatively small, but not none - I've implemented one crappy compiler once at the uni :) Also, since this isn't going to be a small task, I can't be sure I'm going to finish it. I'm just realistic ;) At the moment I'm willing to give it a try though, so please let me know if you have any suggestions. I'll be grateful for any. Thanks!

Share this post


Link to post
Share on other sites
Efficiency is a problem with interpreted languages.. Nothing else comes to mind immediately.
By the way, I am knowledgeable with lex and yacc, and BNF and regular expressions, and I could help if necessary.

Share this post


Link to post
Share on other sites
This is right, efficiency is always going to be an issue if the code is going to be platform independent. However, once the independent bytecode interpreter is done, it should be relatively easy to write some unoptimized, but still way faster, platform dependent compiler. I wish I'll be able to do so.

In terms of tools I've already decided to use flex + SLK combination. I've been using yacc before and and SLK is really a lot nicer to work with - particularly in that you don't need to mix your compiler's code with grammar file.

Share this post


Link to post
Share on other sites
Thanks for info. Type inference is something I really like and will probably attempt to take into account as well.

Btw, I just had a look at the Boo language and I must say it looks really cool.
Lots of nice and useful features not seen in other languages. Especially liked yieldable generators and macros.

The only thing that makes Boo somewhat problematic for game dev is it generates .NET compatible CLI bytecode, which is cool on one side, but causes problems with platform independence (hence console game dev) or easy embedding within your app. Unless implementing your own CLI bytecode interpreter, but that would be pointless in my case.

Share this post


Link to post
Share on other sites
You wish for the language to be simple and expressive, but you also want built-in support for states. In my opinion that is a bad design decision. If your language is truly expressive then you should easily be able to define states in terms of the language itself. If you can't then I'd argue your language doesn't accomplish its goal of being expressive, and if you simply choose not to then you chose to make the core language more complicated. The final decision is of course yours, but would you mind explaining the reasoning behind including states?

Quote:
- whether it has to support multi-threading / synchronization schemes and which if so

I would very much recommend getting some decent concurrency support. Since your language is biased towards games where quad-core CPUs are already being used and dual-cores are almost main-stream. You want to get some model that scales well. If you haven't read it yet I would recommend reading some of the stuff by Herb Sutter (his famous The Free Lunch is Over article and the more in-depth Effective Concurrency series of articles). He touches upon the scalability of concurrent programs and the safe use of synchronization primitives.

Alternatively you should take a look at Erlang's concurrency model (which it's pretty famous for). I actually think it would work very well for a scripting language for games since we have many pretty independent actors with well-defined communication channels. I don't know if it has been done before, but I certainly think using such an approach would be pretty interesting.

Share this post


Link to post
Share on other sites
Quote:
Original post by CTar
You wish for the language to be simple and expressive, but you also want built-in support for states. In my opinion that is a bad design decision. If your language is truly expressive then you should easily be able to define states in terms of the language itself. If you can't then I'd argue your language doesn't accomplish its goal of being expressive, and if you simply choose not to then you chose to make the core language more complicated. The final decision is of course yours, but would you mind explaining the reasoning behind including states?


First, thanks a lot for all your points here.

My decision about having native support for states was based on experience with UnrealScript (which is generally not language I want to follow in design). I found it nice feature that each class could have several states, each with its own implementation of the same method. Depending on current object's state appropriate method gets invoked automatically.

Without native state support typical approach looks like either a massive switch-case for each state or manual function table composition and few hacks around that to make the right function execute when needed. Am I missing something people are doing?

Being able to treat states just like classes in terms of polymorphism makes them even more interesting.

I would be interested to hear why exactly would you find this a bad decision to have native support for states.

In terms of concurrency, I'll need to investigate the topic a bit more (thanks for links). I'm just not quite sure whether any reasonable concurrency can be achieved at all in game dev. This is because I'm used to the fact that the game entities (actors) usually need to touch too many other entities or native engine functionality that it's nearly impossible to make several actors' "tick" (update) function run in parallel.
Maybe that could be achieved for games with big world, but probably not for a typical FPS games. Still I would like to investigate that a bit more and possibly come up with reasonable solution for concurrency in scripting.

Share this post


Link to post
Share on other sites
Quote:
Original post by MickeyMouse
Without native state support typical approach looks like either a massive switch-case for each state or manual function table composition and few hacks around that to make the right function execute when needed. Am I missing something people are doing?

Well, what would your native support look like? If your language is expressive enough, you should be able to provide the same support in a library.

Languages ideally should have a small set of well designed general purpose, orthogonal features that can be used to express whatever you want, rather than lots of ad-hoc features.

Let's look at Scheme for example. Scheme is one of the simplest practical languages there is, yet it is extremely expressive. It has first-class continuations, so it can express latent/resumable functions and coroutines and generators as libraries (as well as many more control structures); it can also express object systems as libraries, from basic systems to multiple-dispatch systems with meta-object protocols.

Quote:

- whether the language should be typeless (see Lua) or maybe strongly typed (see Ocaml)
- whether it has to support multi-threading / synchronization schemes and which if so

It's pretty hard to design a language that is actually genuinely better than an existing language. Making your language typed and concurrent would give it something over existing scripting languages like Python and Ruby.

Quote:

I'm just not quite sure whether any reasonable concurrency can be achieved at all in game dev. This is because I'm used to the fact that the game entities (actors) usually need to touch too many other entities or native engine functionality that it's nearly impossible to make several actors' "tick" (update) function run in parallel.
For the usual highly stateful approach to game entity update, some people think the only viable solution is Software Transactional Memory.

Share this post


Link to post
Share on other sites
Quote:
Original post by Rebooted
Well, what would your native support look like? If your language is expressive enough, you should be able to provide the same support in a library.


Just to answer your question. I was thinking about state syntax like e.g.:

class Monster
{

Actor target;

state Idle
{
function Update(deltaTime) { ... }
function OnSee(Actor a) { target = a; change_state(Attack) }
}

state Attack
{
function Update(deltaTime) { ... }
}

}

Quote:
Languages ideally should have a small set of well designed general purpose, orthogonal features that can be used to express whatever you want, rather than lots of ad-hoc features.

Let's look at Scheme for example. Scheme is one of the simplest practical languages there is, yet it is extremely expressive. It has first-class continuations, so it can express latent/resumable functions and coroutines and generators as libraries (as well as many more control structures); it can also express object systems as libraries, from basic systems to multiple-dispatch systems with meta-object protocols.


I fully agree on one hand, i.e. that ideal language would have very small set of features that would allow for expressing any complex concept. That would be great.

Following this, one might try to add full support for a typical OO rules to some non-OO expressive language. However from my experience (which isn't big for scripting languages) I can say there hasn't been developed any expressive enough language to express typical OO rules in an acceptable manner (easy to use, efficient). This is because typical OO rules (i.e. support for inheritance, virtual members, visibility, interfaces) is just too much for any non-OO language (I heard of) because it extends syntax and lots of other compilation rules.

I'm not saying I'm convinced having native support for states is good. Of course, I see the point that would complicate the language which is something I would like to avoid. The reasoning behind having states is the fact that the language is going to be used for game dev only, and a typical approach to implementing actors is to use states. So, if it's going to be something that widely used why not to have it supported natively?

Quote:
For the usual highly stateful approach to game entity update, some people think the only viable solution is Software Transactional Memory.


I like STM a lot, but it's a little hard to make it really useful in game dev because of lots of dependencies between script code and native game code (by that I mean the engine's language, C++ for most of the games nowadays). If script code wasn't dependent on engine code and so an update of thousands of actors would not rely on native calls then I would probably be more enthusiastic for STM.

Also, the other problem is no shared physical memory between processors on some platforms (see PS3) which makes spreading update of multiple actors among several processors problematic. To update an actor on any other than the "main" processor you'd have to copy necessary memory parts to that processor. This doesn't seem to be good thing to do of course.

I've seen Tim Sweeney's presentation on STM but he didn't touch these issues at all. The only thing he said was that using this method to update around 10 000 actors, each touching on average 5-10 other actors will result in 2-4 percent collisions (which require script code retry). Overall this means that multi-threaded update can be almost fully parallelized. This looks very cool but I wonder if he has any ideas how to solve mentioned issues.

Share this post


Link to post
Share on other sites
Quote:
Original post by MickeyMouse
Following this, one might try to add full support for a typical OO rules to non-OO language. However from my experience (which isn't big for scripting languages) I can say there hasn't been developed any expressive enough language to express typical OO rules in an acceptable manner (easy to use, efficient).

Well, Scheme is one. Google "scheme object system" and you should get a lot of results. CLOS (Common Lisp Object System) is completely defined as a Lisp library.

Quote:
This is because [..] it extends syntax and lots of other compilation rules.

If the encoding as a library is awkward to use, Scheme lets you use macros to define your own syntactic sugar. The library implementation then appears identical to a primitive feature defined in the core language.

The remaining issue is efficiency. If your language has support for "staging", you can do compile-time specialisation of your code, effectively controlling compilation of your library yourself. Scheme supports this, again, through its macro system. You can do stuff like write an interpreter for a DSL and by staging it, turn it into a compiler. This is how these object systems can be efficient.

While Scheme, Lisp, SML and OCaml can define untyped object systems as libraries, I think it would be much harder to express typed object systems (I think you could do it if you had type functions, so your library could compute the type to assign to an object). So you probably want your object system to be a primitive feature, but I'd recommend you include first-class continuations so you can define things like generators as libraries rather than primitives.

Quote:
Overall this means that multi-threaded update can be almost fully parallelized. This looks very cool but I wonder if he has any ideas how to solve mentioned issues.

I don't know a lot about STM. I think there are quite a lot of issues to still be ironed out; which makes it a good area to explore. OO languages could do with better concurrency features and better type systems.

Share this post


Link to post
Share on other sites
You guys have pointed me few important things I might have missed otherwise. I've spent quite some time recently researching various languages.

After reading a lot on native concurrency support in languages I must say it's really hard topic. The solution that I like the most is STM because it ideally allows the programmer not to think about concurrency at all. It's got difficult to solve issues though (as pointed before). Therefore I decided that for now I'm not going to support any kind of multi-threading.

So, my current set of features is going to be changed as follows:
- no multi-threading
- no built-in support for states (after thinking about that a lot I seem to have been convinced that they might overcomplicate the language)
- support for generators (via yield keyword)

...Ok, now I'm going to start works on grammar. Will let you know guys when I've got my first "Hello World" running :D

Thanks for all hints and suggestions!

Share this post


Link to post
Share on other sites

This topic is 3584 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this