• Advertisement
Sign in to follow this  

Unity Run-time compiled C in python scripts. Works.

This topic is 2990 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I've written a module that uses ctypes and Tiny C to compile C source code from within a python program and makes it callable within python. Is this something the community would be interested in having? If so I can put up a page and post the code on my gamedev.net space for all to have. The application I'm currently working on has some performance critical sections where python just wasn't cutting it. It's easy to extend python with C using ctypes, but it's not very convenient when you only need to write very small amounts of performance-critical code in C. I found a few libraries that would suit this purpose one way or another, but I felt that they'd be too cumbersome for what I am doing. I did run across a very old, seemingly abandoned project called "C in Python" that used the Tiny C compiler and ctypes to compile C code from strings at run-time. It didn't work, but I thought it was a great idea, so I've rewritten it so that it works with the latest TCC and python 2.6. For now I've only tested it on linux 32 bit. It would need some modifications to work on Windows, but it shouldn't be difficult to do. It would probably work on Mac OS X also, but I couldn't get Tiny C to build on my mac. Here's a working example:
import ctypes
import compiler

# Defining functions that will be exported to python from the C source.
# compile expects a list of two-tuples containing first the name
# of the exported function in the C source code and the ctypes prototype
exports = (('fib', ctypes.CFUNCTYPE(ctypes.c_int, ctypes.c_int)),
           ('test_callback', ctypes.CFUNCTYPE(None)))

# defining a function to call back into from the C code
def callback():
    print "Calling back into python WORKS!!!!"
    
# You can optionally make your python code available to the C program.
# compile expects a list of three-tuples containg first the name
# to use in the C source, the ctypes prototype, and the actual
# python function object to be called.
imports = (('callback', ctypes.CFUNCTYPE(None), callback),)

# The C source code
code = """
int fib(int n)
{
    if (n <= 2)
        return 1;
    else
        return fib(n-1) + fib(n-2);
}

void test_callback() {
    callback();
}
"""

# compile returns a dictionary of callables. imports is optional.
lib = compiler.compile(exports, code, imports)

print "c fib result:", lib['fib'](30)
print "should see the same text repeated twice:"
callback()
lib['test_callback']()
Tiny C also supports importing functions from a dynamic link library so I might implement that also, but I haven't needed it yet. [Edited by - smr on December 1, 2009 2:40:26 PM]

Share this post


Link to post
Share on other sites
Advertisement
I think I'll make this available once I get it working under Windows and Mac OS X. I'm also considering automatically grabbing the exported functions by analyzing the C source code. More to come later.

Share this post


Link to post
Share on other sites
Quote:
Original post by swiftcoder
Quote:
Original post by smr
I'm also considering automatically grabbing the exported functions by analyzing the C source code.
Doesn't ctypes already build a list of functions from the DLL/so/dylib symbol table?


It does, but this compiles into a chunk of memory. There's no library to load, so I have to grab the symbols and define the function arguments myself.

Share this post


Link to post
Share on other sites
Quote:
Original post by smr
Quote:
Doesn't ctypes already build a list of functions from the DLL/so/dylib symbol table?
It does, but this compiles into a chunk of memory. There's no library to load, so I have to grab the symbols and define the function arguments myself.
Hmm, that seems efficient, but I am not sure how portable it is. How are you making sure the generated machine code ends up in an executable page?

My original thought on how to do this was to mmap a file, compile into it, and then load that with ctypes.

Share this post


Link to post
Share on other sites
Quote:
Original post by swiftcoder
Quote:
Original post by smr
Quote:
Doesn't ctypes already build a list of functions from the DLL/so/dylib symbol table?
It does, but this compiles into a chunk of memory. There's no library to load, so I have to grab the symbols and define the function arguments myself.
Hmm, that seems efficient, but I am not sure how portable it is. How are you making sure the generated machine code ends up in an executable page?

My original thought on how to do this was to mmap a file, compile into it, and then load that with ctypes.


Aww, shucks... I guess I was so excited about the idea of this that I hadn't thought that part through. I have to give the tiny c library a place to put the code after it's compiled, which I'm doing now by allocating an array in python of the correct size then passing tiny c the address of its buffer. I suppose I could find a way to ensurr that I get a chunk in an executable page and also properly aligned, but that would be more work than just compiling into a mmap'ed file.

Share this post


Link to post
Share on other sites
Quote:
Original post by smr
I suppose I could find a way to ensurr that I get a chunk in an executable page and also properly aligned, but that would be more work than just compiling into a mmap'ed file.
In theory at least, you can configure it so that the mmap'ed file shouldn't ever touch the physical disk, so the performance difference should be very much. Plus, you get ctypes to do the legwork of symbol table passing.

I would love to take a look when you have a working implementation - might fit right in with my planet renderer...

Share this post


Link to post
Share on other sites
So I've coded up a version that I *think* will work on a linux system with NX. Unfortunately I don't have one of these so I was wondering if someone with a linux machine with NX would test for me?

Share this post


Link to post
Share on other sites
Quote:
Original post by Vectorian
On Windows you should use VirtualAlloc and VirtualProtect to make sure the memory you allocate is executable (msdn)


Already working on it ;)

Share this post


Link to post
Share on other sites
Quote:
Original post by smr
Quote:
Original post by Vectorian
On Windows you should use VirtualAlloc and VirtualProtect to make sure the memory you allocate is executable (msdn)


Already working on it ;)


OK, I've got it working under windows. I need to clean it up a bit before I can release it. I think I am setting up the execute access properly, but I don't have a machine with NX, so I can't be sure.

Share this post


Link to post
Share on other sites
Quote:
Original post by smr
I need to clean it up a bit before I can release it. I think I am setting up the execute access properly, but I don't have a machine with NX, so I can't be sure.
I have access to a couple of Red Hat boxes running SELinux, etc. if that is any help. Want me to give it a shot?

Share this post


Link to post
Share on other sites
I'll release what I have as soon as I can get the setup script to work properly on linux and windows. This is my first go at creating a setup script. The package will build and everything works on Windows, but on linux I'm having some problems getting the libraries to go in the right places.

Share this post


Link to post
Share on other sites
I've uploaded the python module, dynamic libraries for Windows and Linux and example program to my gdnet page. This isn't an installable setup package. I couldn't get that to work on Linux. The setup script on linux copies the libraries to my package's dist-packages folder, but I can't seem to figure out a portable way to get the path to these libraries unless I hard code them. I'll work on this later.

For now, you can copy the tcc.py module and the lib folder into your application's directory and everything should work on Linux and Windows. This does not work on Mac OS X. It shouldn't be difficult to get working for osx, I just couldn't get Tiny C to build on my Mac.

This module should work on systems using NX protection. I can't test it myself because I don't have a system with NX. Would someone with NX mind testing this for me and letting me know whether it works or not? To test, first try running example.py without any modifications. If that works, set tcc.TCC_TEST_NX = True directly after importing the tcc module. If you receive a page fault with TCC_TEST_NX set true but no page fault when reset False, then then everything is working the way it should.

Share this post


Link to post
Share on other sites


My question would be :

What use would this be put to -- on-the-fly C lisp-like Lambda functions ???

It would have to be some kind of dynamicly defined heavy number-crunching (code reused alot to offset the overhead of the initial code-building and compilation stage, over a less efficient code interpretor (possibly a data option control or byte code system working for a limited parameterizing scheme).


If its something that changes constantly (code being rebuilt frequently) the overhead might make the non-native compilation more efficient.

Since the code is building C code it has to have finite/limited logic to do that operation (unless the code is being made from use input fragments, ie- equations) and even that could be a wizard that build bytecode or activated precanned flavors in static code. Again unless if is build once and run ALOT, so to amortize the problem->C code->native code conversion overhead.


Share this post


Link to post
Share on other sites
Quote:
Original post by wodinoneeye


My question would be :

What use would this be put to -- on-the-fly C lisp-like Lambda functions ???

It would have to be some kind of dynamicly defined heavy number-crunching (code reused alot to offset the overhead of the initial code-building and compilation stage, over a less efficient code interpretor (possibly a data option control or byte code system working for a limited parameterizing scheme).


If its something that changes constantly (code being rebuilt frequently) the overhead might make the non-native compilation more efficient.

Since the code is building C code it has to have finite/limited logic to do that operation (unless the code is being made from use input fragments, ie- equations) and even that could be a wizard that build bytecode or activated precanned flavors in static code. Again unless if is build once and run ALOT, so to amortize the problem->C code->native code conversion overhead.


The C code is intended to be only built once and the Tiny C compiler is very fast. Have a look at the TCC website to see some metrics. And you are correct, there is quite a bit of overhead when marshaling across the C / python border, so you should always profile before you decide to convert some algorithm from python to C. And when you do replace some python code with C, you should think hard about how you'll marshal the data and how often. In many circumstances it will be more efficient to wad up a big chunk of data to be processed with C code once rather than iterating over it in python, crossing the python-C barrier for each item.

Though building this module has been a fun experience, I'll admit that I'm not entirely sure how useful it will be. There's no debug support for the C code, so for anything non-trivial you'll probably have to code and debug with standard C tools anyway then copy-paste the code into python. Since you've already gone through all of that trouble, you might as well just compile your code as a dynamic library and load it with ctypes. Any other compiler will probably produce better code than TCC because TCC only performs a handful of optimizations, trading fast code for fast compiling.

Share this post


Link to post
Share on other sites
Quote:
Original post by wodinoneeye
What use would this be put to -- on-the-fly C lisp-like Lambda functions ???
Assembling composite functions at runtime: i.e. function composition as supported by lisp/python, but in C.

For example, I have a set of procedural generation algorithms, which can be composed at runtime. At the moment, these are implemented as a large set of C functions in a DLL, each of which takes a function pointer and a void argument pointer to chain with the previous function. These are then loaded using ctypes, and some clever (and blatantly unsafe) ctypes tricks are used to chain the C functions together.

With smr's on-the-fly compiler, I can just splice the actual source of the functions together (as I already do for GLSL), and compile a single large function which performs the whole generation step. For the general case, I have a feeling that the removal of all the function/void pointers and related hacks will offset the performance loss of using TinyC, and the maintenance will be much reduced.

Share this post


Link to post
Share on other sites
Quote:
Original post by swiftcoder
Quote:
Original post by wodinoneeye
What use would this be put to -- on-the-fly C lisp-like Lambda functions ???
Assembling composite functions at runtime: i.e. function composition as supported by lisp/python, but in C.

For example, I have a set of procedural generation algorithms, which can be composed at runtime. At the moment, these are implemented as a large set of C functions in a DLL, each of which takes a function pointer and a void argument pointer to chain with the previous function. These are then loaded using ctypes, and some clever (and blatantly unsafe) ctypes tricks are used to chain the C functions together.

With smr's on-the-fly compiler, I can just splice the actual source of the functions together (as I already do for GLSL), and compile a single large function which performs the whole generation step. For the general case, I have a feeling that the removal of all the function/void pointers and related hacks will offset the performance loss of using TinyC, and the maintenance will be much reduced.



My question WAS asking if it was for that purpose (which Ive done before with TinyC by building a DLL 'on the fly' but that was part of an all C/C++ project).

But again thes same problem -- procedurally generated X is either :

just a string of high level calls with parameters and shared memory assets -- which would probably better be done just by calling the C library calls from python directly when they are composed (or by a python called composer in C itself).

OR

A composition of a very intricate set of interrelated generated objects which have ALOT of validation rules -- which calls for backtracking and culling invalid options, something best composed while still in a simpler symbolic data state which is then converted into source code (or again just call C library directly without spewing C source code that needs to be compiled). Alot of processing by the composer is done up front symbolicly and the 'generator' is issued (possibly alot) of simple directives which are high level calls with parameters (like option 1 above).


In one case its probably just extra work to do the problem->C source->C native code conversions. For the other so much work would be best done on the C side for efficiency that you might as well do it all there and push the result to the python side (or rather directly to the pythons game engine components which are likely a C based render engine or game mechanics). Of course whole or part of the generation might be python behavior scripting which would be C building Python (I have done on the fly python composition and an inline 'include' mechanism before, but it was messy -- required a seperate file and another file invoking it run in a different thread)



Again, building C source to compile and run what you could run directly (immediately) is usually a waste. The overhead of repeated C calls from python are probably overshadowed by the processing the C does added to all the conversion overhead you have for a Problem->C source->C native scheme.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
  • Advertisement
  • Popular Tags

  • Advertisement
  • Popular Now

  • Similar Content

    • By GytisDev
      Hello,
      without going into any details I am looking for any articles or blogs or advice about city building and RTS games in general. I tried to search for these on my own, but would like to see your input also. I want to make a very simple version of a game like Banished or Kingdoms and Castles,  where I would be able to place like two types of buildings, make farms and cut trees for resources while controlling a single worker. I have some problem understanding how these games works in the back-end: how various data can be stored about the map and objects, how grids works, implementing work system (like a little cube (human) walks to a tree and cuts it) and so on. I am also pretty confident in my programming capabilities for such a game. Sorry if I make any mistakes, English is not my native language.
      Thank you in advance.
    • By Ovicior
      Hey,
      So I'm currently working on a rogue-like top-down game that features melee combat. Getting basic weapon stats like power, weight, and range is not a problem. I am, however, having a problem with coming up with a flexible and dynamic system to allow me to quickly create unique effects for the weapons. I want to essentially create a sort of API that is called when appropriate and gives whatever information is necessary (For example, I could opt to use methods called OnPlayerHit() or IfPlayerBleeding() to implement behavior for each weapon). The issue is, I've never actually made a system as flexible as this.
      My current idea is to make a base abstract weapon class, and then have calls to all the methods when appropriate in there (OnPlayerHit() would be called whenever the player's health is subtracted from, for example). This would involve creating a sub-class for every weapon type and overriding each method to make sure the behavior works appropriately. This does not feel very efficient or clean at all. I was thinking of using interfaces to allow for the implementation of whatever "event" is needed (such as having an interface for OnPlayerAttack(), which would force the creation of a method that is called whenever the player attacks something).
       
      Here's a couple unique weapon ideas I have:
      Explosion sword: Create explosion in attack direction.
      Cold sword: Chance to freeze enemies when they are hit.
      Electric sword: On attack, electricity chains damage to nearby enemies.
       
      I'm basically trying to create a sort of API that'll allow me to easily inherit from a base weapon class and add additional behaviors somehow. One thing to know is that I'm on Unity, and swapping the weapon object's weapon component whenever the weapon changes is not at all a good idea. I need some way to contain all this varying data in one Unity component that can contain a Weapon field to hold all this data. Any ideas?
       
      I'm currently considering having a WeaponController class that can contain a Weapon class, which calls all the methods I use to create unique effects in the weapon (Such as OnPlayerAttack()) when appropriate.
    • By Vu Chi Thien
      Hi fellow game devs,
      First, I would like to apologize for the wall of text.
      As you may notice I have been digging in vehicle simulation for some times now through my clutch question posts. And thanks to the generous help of you guys, especially @CombatWombat I have finished my clutch model (Really CombatWombat you deserve much more than a post upvote, I would buy you a drink if I could ha ha). 
      Now the final piece in my vehicle physic model is the differential. For now I have an open-differential model working quite well by just outputting torque 50-50 to left and right wheel. Now I would like to implement a Limited Slip Differential. I have very limited knowledge about LSD, and what I know about LSD is through readings on racer.nl documentation, watching Youtube videos, and playing around with games like Assetto Corsa and Project Cars. So this is what I understand so far:
      - The LSD acts like an open-diff when there is no torque from engine applied to the input shaft of the diff. However, in clutch-type LSD there is still an amount of binding between the left and right wheel due to preload spring.
      - When there is torque to the input shaft (on power and off power in 2 ways LSD), in ramp LSD, the ramp will push the clutch patch together, creating binding force. The amount of binding force depends on the amount of clutch patch and ramp angle, so the diff will not completely locked up and there is still difference in wheel speed between left and right wheel, but when the locking force is enough the diff will lock.
      - There also something I'm not sure is the amount of torque ratio based on road resistance torque (rolling resistance I guess)., but since I cannot extract rolling resistance from the tire model I'm using (Unity wheelCollider), I think I would not use this approach. Instead I'm going to use the speed difference in left and right wheel, similar to torsen diff. Below is my rough model with the clutch type LSD:
      speedDiff = leftWheelSpeed - rightWheelSpeed; //torque to differential input shaft. //first treat the diff as an open diff with equal torque to both wheels inputTorque = gearBoxTorque * 0.5f; //then modify torque to each wheel based on wheel speed difference //the difference in torque depends on speed difference, throttleInput (on/off power) //amount of locking force wanted at different amount of speed difference, //and preload force //torque to left wheel leftWheelTorque = inputTorque - (speedDiff * preLoadForce + lockingForce * throttleInput); //torque to right wheel rightWheelTorque = inputTorque + (speedDiff * preLoadForce + lockingForce * throttleInput); I'm putting throttle input in because from what I've read the amount of locking also depends on the amount of throttle input (harder throttle -> higher  torque input -> stronger locking). The model is nowhere near good, so please jump in and correct me.
      Also I have a few questions:
      - In torsen/geared LSD, is it correct that the diff actually never lock but only split torque based on bias ratio, which also based on speed difference between wheels? And does the bias only happen when the speed difference reaches the ratio (say 2:1 or 3:1) and below that it will act like an open diff, which basically like an open diff with an if statement to switch state?
      - Is it correct that the amount of locking force in clutch LSD depends on amount of input torque? If so, what is the threshold of the input torque to "activate" the diff (start splitting torque)? How can I get the amount of torque bias ratio (in wheelTorque = inputTorque * biasRatio) based on the speed difference or rolling resistance at wheel?
      - Is the speed at the input shaft of the diff always equals to the average speed of 2 wheels ie (left + right) / 2?
      Please help me out with this. I haven't found any topic about this yet on gamedev, and this is my final piece of the puzzle. Thank you guys very very much.
    • By Estra
      Memory Trees is a PC game and Life+Farming simulation game. Harvest Moon and Rune Factory , the game will be quite big. I believe that this will take a long time to finish
      Looking for
      Programmer
      1 experience using Unity/C++
      2 have a portfolio of Programmer
      3 like RPG game ( Rune rune factory / zelda series / FF series )
      4 Have responsibility + Time Management
      and friendly easy working with others Programmer willing to use Skype for communication with team please E-mail me if you're interested
      Split %: Revenue share. We can discuss. Fully Funded servers and contents
      and friendly easy working with others willing to use Skype for communication with team please E-mail me if you're interested
      we can talk more detail in Estherfanworld@gmail.com Don't comment here
      Thank you so much for reading
      More about our game
      Memory Trees : forget me not

      Thank you so much for reading
      Ps.Please make sure that you have unity skill and Have responsibility + Time Management,
      because If not it will waste time not one but both of us
       

  • Advertisement