I've been thinking about developing my scripting language a bit actually, after some conversations round here. I'm wondering if it would actually be of use to other people if I could supply some better features.
There are three major areas I'd like to improve upon, some of which would affect the VM and others would affect the compiler:
1) Dynamic memory - I'd like to add the concept of a heap to the VM with some assembler instructions to allocate and delete blocks of memory. This would mean that the language could support proper string operations like concatenation, as well as maybe stuff like linked lists and so on, as well as hopefully meaning you could finally pass strings into the VM from the host application.
2) Classes - this has always been a struggle for me but I would love to add some object-oriented aspects to the language. This would require a complete compiler rewrite from scratch, but would be so satisfying if the basics of classes with members and methods, as well as operator overloading, could be added. Even maybe single-inheritence and virtual functions would be cool, although I may be running away with myself a bit here [smile].
The biggest problem I've had with that in the past is that the symbol table part of the compiler needs to become namespace-based, which is fine in terms of implementing a tree structure for name storage, but the complication arises in that every single text ID that is scanned can potentially be a set of ID's seperated by ::s.
3) Integration with C and C++. Here I really struggle. I'd love to be able to call C or C++ functions directly from my language but I have no idea how to implement this. Clearly the VM would need to maintain some kind of map between function names and function pointers, but since a function pointer type is dependant on its parameters and return types, I have never been able to work out a way to do this flexibily.
A hack solution would be to only allow a callback function that took as a parameter a container of multi-purpose variables, but I've never really liked that.
Equally a system where I try to predict every possible combination of parameters is out of the question, so I'm really quite stuck.
Last problem, thinking about all of the above, is I end up writing a scripting language that becomes almost as complicated to use as C++. Given that I plan for my language to require compilation into byte code, the question arises of is there any actual benefit to using my language over using DLL plug-ins in a host app for "scripting" instead.
Thinking myself in circles here really (again).