Jump to content

  • Log In with Google      Sign In   
  • Create Account






C# for scripting - runtime compilation

Posted by et1337, 10 December 2011 · 806 views

I set out to add scripting support to Project Lemma the other day. End result: I can recompile C# scripts on the fly and cache the bytecode in DLLs. The best part: there's no special binding code, and no performance hit.

There are a lot of .NET scripting solutions out there. Here's a few I found in my research:

  • IronPython. Fully dynamic, kinda slow. Requires marshalling of some kind between the script world and .NET.
  • CSScript. Very well supported, includes Visual Studio extensions and shell extensions. Compiles C# to bytecode at runtime, with caching. Scripts cannot be changed once loaded.
  • Lua. The industry standard in scripting. From what I understand, a little challenging to get working with .NET.

I decided to try out C# runtime compilation. Really, C# is the best scripting language I could ask for. If I succeeded, I could keep writing the same code I've been writing, but I could recompile it and see it in action with a keystroke instead of restarting the game!


Setup


I started by writing a ScriptBase class that every script would inherit from.

public class ScriptBase
{
	public static Main main;

	protected static Entity get(string id)
	{
		return ScriptBase.main.GetByID(id);
	}
}
"Main" is my main game class. The "get" function is a utility function I wanted to provide for the scripting environment. Using the ScriptBase class, I can add utility functions that can speed up the scripting process.

Next, I wrote a prefix and postfix for the script files. This will help the scripts actually look like scripts, i.e. not object oriented, pretty much just a list of statements to execute in order.

private const string scriptPrefix =
@"
using System;
using Microsoft.Xna.Framework;
// ...

namespace Lemma.Scripts
{
	public class Script : ScriptBase
	{
		public static void Run()
		{
";

		private const string scriptPostfix =
@"
		}
	}
}
";
The idea is: compile the assembly, reflect it and get the Script type, then call the static Run function on it. Like so:

Type t = assembly.GetType("Lemma.Scripts.Script");

t.GetField("main", BindingFlags.Static | BindingFlags.Public | BindingFlags.FlattenHierarchy)
	.SetValue(null, this.main);

this.scriptMethod = t.GetMethod("Run", BindingFlags.Static | BindingFlags.Public);
this.scriptMethod.Invoke(null, null);

Compile some scripts already!


This code isn't particularly interesting, but I'll provide it for reference:

string scriptPath = ...;
string binaryPath = ...;

try
{
	Assembly assembly = null;

	using (Stream stream = TitleContainer.OpenStream(scriptPath))
	using (TextReader reader = new StreamReader(stream))
	{
		CodeDomProvider provider = CodeDomProvider.CreateProvider("CSharp");

		CompilerParameters cp = new CompilerParameters
		{
			GenerateExecutable = false,
			GenerateInMemory = false,
			TreatWarningsAsErrors = false,
			OutputAssembly = binaryPath
		};

		// Add references to all the assemblies we might need.
		Assembly executingAssembly = Assembly.GetExecutingAssembly();
		cp.ReferencedAssemblies.Add(executingAssembly.Location);
		foreach (AssemblyName assemblyName in executingAssembly.GetReferencedAssemblies())
			cp.ReferencedAssemblies.Add(Assembly.Load(assemblyName).Location);

		// Invoke compilation of the source file.
		CompilerResults cr = provider.CompileAssemblyFromSource(cp, Script.scriptPrefix + reader.ReadToEnd() + Script.scriptPostfix);

		if (cr.Errors.Count > 0)
		{
			// Display compilation errors.
			StringBuilder builder = new StringBuilder();
			foreach (CompilerError ce in cr.Errors)
			{
				builder.Append(ce.ToString());
				builder.Append("\n");
			}
			this.Errors.Value = builder.ToString();
		}
		else
			assembly = cr.CompiledAssembly;
	}
}
catch (Exception e)
{
	this.Errors.Value = e.ToString();
}
The goal here is to compile the script to a DLL so the next time the game runs, I could check if it already exists and just load the DLL instead of recompiling. You can do this with the CompilerParameters object, by setting GenerateInMemory to false and OutputAssembly to the path of the DLL you want to save.

Recompiling scripts on the fly


Now the problem is, once we load the DLL, the .NET runtime locks the file until the program exits. There's no way to unload the library and unlock the file, unless we put it in a different AppDomain, which means we have to marshal data back and forth between the script and game, which defeats the purpose.

Since one of our requirements is being able to recompile the script without restarting the game, we need to find a way to load the DLL without locking the file. Enter shadow copying. You can tell the .NET runtime to make a copy of every assembly it loads, and load the "shadow copy" instead of the real file. That leaves the original DLL free for us to rewrite. Great, so let's just enable shadow copying and we're done.

Unfortunately, life is never that easy. You can't enable shadow copying for an AppDomain once the domain has been created. You have to create a new AppDomain with the settings you want. Solution: create a tiny launcher executable that creates an AppDomain with shadow copying enabled, and then executes our game in that AppDomain. Here's the code:

public static void Main(string[] args)
{
	string baseDirectory = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);

	AppDomainSetup setup = new AppDomainSetup();
	setup.ShadowCopyFiles = "true";
	setup.ApplicationBase = baseDirectory;
	setup.PrivateBinPath = baseDirectory;

	AppDomain domain = AppDomain.CreateDomain("", AppDomain.CurrentDomain.Evidence, setup);
	domain.ExecuteAssembly(Path.Combine(baseDirectory, "Lemma.exe"), args);
}
Note: the AppDomainSetup.ShadowCopyDirectories property lets you provide a list of directories to limit the use of shadow copying. I tried to use it to specify that only the script DLLs be shadow copied, but it didn't work. YMMV.

Okay, so I fired up my game, compiled the script, went to delete the compiled DLL, annnnnd... still locked! What's going on?

I did some reflecting and found out the C# runtime compiler is kinda janky. It isn't, like I assumed, a .NET wrapper around some kind of compiler library. Instead, it actually invokes the C# compiler executable, then loads the resulting DLL. If you specify GenerateInMemory = true, it invokes the compiler with a temporary DLL file path and loads the temp file into memory.

Furthermore, it doesn't shadow copy the DLL when it loads it, even if the AppDomain is configured to do so. I'm guessing this is because it uses a different Assembly.Load function... don't quote me but I believe Assembly.LoadFile does not shadow copy files, while Assembly.LoadFrom does.

Anyway, to solve this, we can take advantage of the compiler's quirky nature here by not specifying an output assembly path. That will cause the compiler to generate a temporary file path for the output assembly. So it will still lock the DLL, but we don't care, because it's a temp file. Luckily, we can still read the DLL even though it's locked, so we can copy it to the path we originally wanted for later use.

Here's the modifications to our original compilation code:
CompilerParameters cp = new CompilerParameters
{
	GenerateExecutable = false,
	GenerateInMemory = false,
	TreatWarningsAsErrors = false,
};

// ...


assembly = cr.CompiledAssembly;
File.Copy(cp.OutputAssembly, binaryPath, true);
The next time we load the script, we'll see the DLL at binaryPath and load it. But it will be automatically shadow copied. Mission accomplished!

Conclusion


One disadvantage to this approach is that when you recompile a script, the old assembly remains in memory, and the temp DLL file will remain locked until the game exits. Like I said, there's no way to unload an assembly from a running AppDomain, and we don't want to put the scripts in another AppDomain. Luckily most script assemblies should be pretty small, and we'll only be recompiling scripts in the editor, not the final game. I can live with that.

I'm still in the middle of all this, but I thought I'd share what I've learned about .NET in the meantime. Might be useful if you're thinking about doing something similar. Here's two StackOverflow threads that helped me through this process:

Thanks for reading!

Mirrored on my blog




Can you go into a little more detail about what your scripts would look like, perhaps posting an example or two? The terms "prefix" and "postfix" seem a little confusing in this context.
Sure. The prefix and postfix are just strings that I tack on at the beggining and end of each script.

An example script might be something like this:

get("player1").Get<Transform>().Position.Value = new Vector3(100, 0, 50);
Obviously this isn't a complete C# source file. It needs "using" statements, and it needs to be put in a function, inside a class, inside a namespace. That's what the prefix and postfix do, so I don't have to write that stuff in every script. I can still create functions and such via closures.

tl;dr: It's just for convenience.
Lua is really easy to get working with C# if you use the library LuaInterface. It does suffer from the lack of really good documentation and I haven't seen a release in quite a while. However, I've been using it for a few years now and the only modification I had to make was the proper handling of params arguments (which the code needed to do this can be found in the bug tracker.)
Interesting idea. Does the rest of your game use C# or do you have some C++ stuff mixed in?

My approach is to build a game using a bunch of C++ libraries (graphics, sound, etc) with wrappers around them and put the game code together with C# (but not XNA). It works really well once you get the boilerplate code done. The performance is very promising too. I don't have the advantage of on the fly scripting but the build time and dev time is significantly better than C++ as you'd already know.

Anyway, I'll be following your posts. Interesting stuff.
@Programmer16: That sounds promising... another developer that used my approach here said that the memory leak issue I mentioned got to be a big problem. If that happens to me too, I'll definitely look into LuaInterface.

@zarfius: It's all C#, which is what makes it so convenient. Your approach sounds a lot like Panda3D. All their high performance code is C++, but everything is exposed to Python. Best of both worlds!

Recent Entries

August 2014 »

S M T W T F S
     12
3456789
10111213141516
1718192021 22 23
24252627282930
31      

Recent Comments

PARTNERS