Okay, after Álvaro's question I realised I was confusing the problem. for two triangles sharing an edge I want to find out whether they are facing away from each other, facing towards each other or laying flat (same normal).
So a slight change from C0lumbo's method, I dot product the normal from the first triangle to the non shared edge of the second triangle, where dotproduct=0 is both triangles laying flat, dotproduct<0 facing away (convex), dotproduct>0 facing towards (concave).
Well one thing you can do is use c code from within it.
I've just started looking at LuaJIT recently, and an idea I had apart from scripting world objects, was to use it to setup all the monotonous code, such geometry data (eg vertexarrayobjects, vertex buffers, attribute bindings, index buffers etc), shader programs, textures, texture samplers etc. Using luajit ffi to minimise the amount of cbindings necessary.
And then in your main program, load those 'resource files' into a table and just say:
GLuint program = *((GLuint*)lua_touserdata(L,-1));
Well I'm using racket scheme to make small demos, though not a game. The advantages for are that it's functional, and has all the advantages that come with that (Of which I'm never going back to an imperative language for personal use). You can mix imperative code with functional, which is necessary for doing inputs and outputs, e.g. drawing graphics, loading files, keyboard input etc.
Also it has macros which allow you do add extensions to the language, e.g. I used this to make a system for loading and setting shaders.
The disadvantages are that you have to write interfaces to c shared libraries to do anything useful, e.g. latest version of opengl, physics, sound, window etc. Though some of the other lisps might have bindings for those things already (I know common lisp does, but I prefer scheme).
I'm currently in the process of writing a library that auto generates bindings from c header files (nearly complete), it uses macros so that you just make a module and call a macro from my library which when compiled will auto generate the bindings for you.
I have a feeling writing a physics engine in it wouldn't be as fast as one written in c/c++, but for a game, if all the intensive code is imported from c bindings leaving the the rest in scheme, it should be fine.