Jump to content
  • Advertisement
Sign in to follow this  
smr

Unity Run-time compiled C in python scripts. Works.

This topic is 3171 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I've written a module that uses ctypes and Tiny C to compile C source code from within a python program and makes it callable within python. Is this something the community would be interested in having? If so I can put up a page and post the code on my gamedev.net space for all to have. The application I'm currently working on has some performance critical sections where python just wasn't cutting it. It's easy to extend python with C using ctypes, but it's not very convenient when you only need to write very small amounts of performance-critical code in C. I found a few libraries that would suit this purpose one way or another, but I felt that they'd be too cumbersome for what I am doing. I did run across a very old, seemingly abandoned project called "C in Python" that used the Tiny C compiler and ctypes to compile C code from strings at run-time. It didn't work, but I thought it was a great idea, so I've rewritten it so that it works with the latest TCC and python 2.6. For now I've only tested it on linux 32 bit. It would need some modifications to work on Windows, but it shouldn't be difficult to do. It would probably work on Mac OS X also, but I couldn't get Tiny C to build on my mac. Here's a working example:
import ctypes
import compiler

# Defining functions that will be exported to python from the C source.
# compile expects a list of two-tuples containing first the name
# of the exported function in the C source code and the ctypes prototype
exports = (('fib', ctypes.CFUNCTYPE(ctypes.c_int, ctypes.c_int)),
           ('test_callback', ctypes.CFUNCTYPE(None)))

# defining a function to call back into from the C code
def callback():
    print "Calling back into python WORKS!!!!"
    
# You can optionally make your python code available to the C program.
# compile expects a list of three-tuples containg first the name
# to use in the C source, the ctypes prototype, and the actual
# python function object to be called.
imports = (('callback', ctypes.CFUNCTYPE(None), callback),)

# The C source code
code = """
int fib(int n)
{
    if (n <= 2)
        return 1;
    else
        return fib(n-1) + fib(n-2);
}

void test_callback() {
    callback();
}
"""

# compile returns a dictionary of callables. imports is optional.
lib = compiler.compile(exports, code, imports)

print "c fib result:", lib['fib'](30)
print "should see the same text repeated twice:"
callback()
lib['test_callback']()
Tiny C also supports importing functions from a dynamic link library so I might implement that also, but I haven't needed it yet. [Edited by - smr on December 1, 2009 2:40:26 PM]

Share this post


Link to post
Share on other sites
Advertisement
I think I'll make this available once I get it working under Windows and Mac OS X. I'm also considering automatically grabbing the exported functions by analyzing the C source code. More to come later.

Share this post


Link to post
Share on other sites
Quote:
Original post by smr
I'm also considering automatically grabbing the exported functions by analyzing the C source code.
Doesn't ctypes already build a list of functions from the DLL/so/dylib symbol table?

Share this post


Link to post
Share on other sites
Quote:
Original post by swiftcoder
Quote:
Original post by smr
I'm also considering automatically grabbing the exported functions by analyzing the C source code.
Doesn't ctypes already build a list of functions from the DLL/so/dylib symbol table?


It does, but this compiles into a chunk of memory. There's no library to load, so I have to grab the symbols and define the function arguments myself.

Share this post


Link to post
Share on other sites
Quote:
Original post by smr
Quote:
Doesn't ctypes already build a list of functions from the DLL/so/dylib symbol table?
It does, but this compiles into a chunk of memory. There's no library to load, so I have to grab the symbols and define the function arguments myself.
Hmm, that seems efficient, but I am not sure how portable it is. How are you making sure the generated machine code ends up in an executable page?

My original thought on how to do this was to mmap a file, compile into it, and then load that with ctypes.

Share this post


Link to post
Share on other sites
Quote:
Original post by swiftcoder
Quote:
Original post by smr
Quote:
Doesn't ctypes already build a list of functions from the DLL/so/dylib symbol table?
It does, but this compiles into a chunk of memory. There's no library to load, so I have to grab the symbols and define the function arguments myself.
Hmm, that seems efficient, but I am not sure how portable it is. How are you making sure the generated machine code ends up in an executable page?

My original thought on how to do this was to mmap a file, compile into it, and then load that with ctypes.


Aww, shucks... I guess I was so excited about the idea of this that I hadn't thought that part through. I have to give the tiny c library a place to put the code after it's compiled, which I'm doing now by allocating an array in python of the correct size then passing tiny c the address of its buffer. I suppose I could find a way to ensurr that I get a chunk in an executable page and also properly aligned, but that would be more work than just compiling into a mmap'ed file.

Share this post


Link to post
Share on other sites
Quote:
Original post by smr
I suppose I could find a way to ensurr that I get a chunk in an executable page and also properly aligned, but that would be more work than just compiling into a mmap'ed file.
In theory at least, you can configure it so that the mmap'ed file shouldn't ever touch the physical disk, so the performance difference should be very much. Plus, you get ctypes to do the legwork of symbol table passing.

I would love to take a look when you have a working implementation - might fit right in with my planet renderer...

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!