# Unity Run-time compiled C in python scripts. Works.

This topic is 3257 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

I've written a module that uses ctypes and Tiny C to compile C source code from within a python program and makes it callable within python. Is this something the community would be interested in having? If so I can put up a page and post the code on my gamedev.net space for all to have. The application I'm currently working on has some performance critical sections where python just wasn't cutting it. It's easy to extend python with C using ctypes, but it's not very convenient when you only need to write very small amounts of performance-critical code in C. I found a few libraries that would suit this purpose one way or another, but I felt that they'd be too cumbersome for what I am doing. I did run across a very old, seemingly abandoned project called "C in Python" that used the Tiny C compiler and ctypes to compile C code from strings at run-time. It didn't work, but I thought it was a great idea, so I've rewritten it so that it works with the latest TCC and python 2.6. For now I've only tested it on linux 32 bit. It would need some modifications to work on Windows, but it shouldn't be difficult to do. It would probably work on Mac OS X also, but I couldn't get Tiny C to build on my mac. Here's a working example:
import ctypes
import compiler

# Defining functions that will be exported to python from the C source.
# compile expects a list of two-tuples containing first the name
# of the exported function in the C source code and the ctypes prototype
exports = (('fib', ctypes.CFUNCTYPE(ctypes.c_int, ctypes.c_int)),
('test_callback', ctypes.CFUNCTYPE(None)))

# defining a function to call back into from the C code
def callback():
print "Calling back into python WORKS!!!!"

# You can optionally make your python code available to the C program.
# compile expects a list of three-tuples containg first the name
# to use in the C source, the ctypes prototype, and the actual
# python function object to be called.
imports = (('callback', ctypes.CFUNCTYPE(None), callback),)

# The C source code
code = """
int fib(int n)
{
if (n <= 2)
return 1;
else
return fib(n-1) + fib(n-2);
}

void test_callback() {
callback();
}
"""

# compile returns a dictionary of callables. imports is optional.
lib = compiler.compile(exports, code, imports)

print "c fib result:", lib['fib'](30)
print "should see the same text repeated twice:"
callback()
lib['test_callback']()

Tiny C also supports importing functions from a dynamic link library so I might implement that also, but I haven't needed it yet. [Edited by - smr on December 1, 2009 2:40:26 PM]

##### Share on other sites
that looks awesome

Thanks!

##### Share on other sites
Cool stuff! I'd love to see more.

##### Share on other sites
I think I'll make this available once I get it working under Windows and Mac OS X. I'm also considering automatically grabbing the exported functions by analyzing the C source code. More to come later.

##### Share on other sites
Quote:
 Original post by smrI'm also considering automatically grabbing the exported functions by analyzing the C source code.
Doesn't ctypes already build a list of functions from the DLL/so/dylib symbol table?

##### Share on other sites
Quote:
Original post by swiftcoder
Quote:
 Original post by smrI'm also considering automatically grabbing the exported functions by analyzing the C source code.
Doesn't ctypes already build a list of functions from the DLL/so/dylib symbol table?

It does, but this compiles into a chunk of memory. There's no library to load, so I have to grab the symbols and define the function arguments myself.

##### Share on other sites
Quote:
Original post by smr
Quote:
 Doesn't ctypes already build a list of functions from the DLL/so/dylib symbol table?
It does, but this compiles into a chunk of memory. There's no library to load, so I have to grab the symbols and define the function arguments myself.
Hmm, that seems efficient, but I am not sure how portable it is. How are you making sure the generated machine code ends up in an executable page?

My original thought on how to do this was to mmap a file, compile into it, and then load that with ctypes.

##### Share on other sites
Quote:
Original post by swiftcoder
Quote:
Original post by smr
Quote:
 Doesn't ctypes already build a list of functions from the DLL/so/dylib symbol table?
It does, but this compiles into a chunk of memory. There's no library to load, so I have to grab the symbols and define the function arguments myself.
Hmm, that seems efficient, but I am not sure how portable it is. How are you making sure the generated machine code ends up in an executable page?

My original thought on how to do this was to mmap a file, compile into it, and then load that with ctypes.

Aww, shucks... I guess I was so excited about the idea of this that I hadn't thought that part through. I have to give the tiny c library a place to put the code after it's compiled, which I'm doing now by allocating an array in python of the correct size then passing tiny c the address of its buffer. I suppose I could find a way to ensurr that I get a chunk in an executable page and also properly aligned, but that would be more work than just compiling into a mmap'ed file.

##### Share on other sites
Quote:
 Original post by smrI suppose I could find a way to ensurr that I get a chunk in an executable page and also properly aligned, but that would be more work than just compiling into a mmap'ed file.
In theory at least, you can configure it so that the mmap'ed file shouldn't ever touch the physical disk, so the performance difference should be very much. Plus, you get ctypes to do the legwork of symbol table passing.

I would love to take a look when you have a working implementation - might fit right in with my planet renderer...

1. 1
2. 2
Rutin
19
3. 3
4. 4
5. 5

• 9
• 9
• 9
• 14
• 12
• ### Forum Statistics

• Total Topics
633302
• Total Posts
3011275
• ### Who's Online (See full list)

There are no registered users currently online

×