Sign in to follow this  

Python: Extending/Embedding, libs in C/C++?

This topic is 3588 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

As some of my other posts here will attest, I'm currently learning Python. At the moment I'm still going through the basics, but I'd like to start thinking about which direction to take my learning in order to visualise where I want to go with respect to game development. In the future I'd like to incorporate Python into my 2D game system I've been working on for the last couple of years. Currently it's a mulch of useful utilities I built for simple 2D games in C++ and using SDL/OpenGL; I tend to refactor it every now and again when I have a new idea about how the whole thing should be structured. My desire for Python is for it to be used for the bulk of the game logic if possible, making it easier for me to put games together. I'm really only interested in working on 2D games; the most computationally intensive type of game I'd consider would be 2D games with a lot of physics or deeper than usual artificial intelligence. I'm not in the position to start using Python for this system yet, but I'd like to get an idea of the best way to use Python with external game code. There's two questions I'm unsure of: whether it's better to aim for embedding Python within the library or extending Python with the library; and if extending, whether to plan for using C++ or C for the code. From my understanding most commercial games out there today that use Python use it as an embedded scripting language. However I've read the article in favour of embedding here (Extending vs. Embedding: There is Only One Correct Decision), and it makes some good arguments. My gut feeling is that it's better conceptually to think of the whole program as being in Python with library calls for the computationally extensive parts, plus I can prototype the bulk of it in Python until performance becomes an issue. However there may be some downside to this that as a novice learning Python might not see. If extending, I'm also not sure whether it would be best to use C or C++ as my language of choice for the library functions. According to the Python documentation C appears to be preferred, and it seems to make more sense to me if I were writing smaller snippets of code to work with a majority Python program. But again, I'm not experienced in working with extending higher level programming languages like Python so I may be mistaken. I'm pretty comfortable with both languages (actually, I'm probably better with the C programing paradigms than C++). Are my gut feelings correct, and should I be seeing myself working on Python programs using extended 2D game library functions written in C (whether mine or from somewhere else?)

Share this post


Link to post
Share on other sites
In all my messing around I have turned to favor extending. It makes things so much more flexible. For example, I have a module coded in C(++) on windows that handles all the graphics and sound. A friend of mine wanted to play my game on his mac, so what did I do? I reimplemented the same module API in pure python using pygame. Instant mac port without needing a mac. Imagine doing that if you embedded.

On C vs C++: all my extensions are C++ classes, and I use plain functions to wrap them into python, it's pretty easy once you do it once and then you can just cut and paste. Or you can use boost which has a pretty crazy set of python things.

Share this post


Link to post
Share on other sites
I personally prefer embedding Python with C++. Basically I find when embedding Python it's easier to debug the C++ code and harder to debug the Python code compared to extending Python. However, debugging Python code is relatively painless when compared to debugging C++ code, and when debugging C++ code you really need all the help you can get.

There are downsides to this. In addition to the points in the article you mentioned, multiple threads and embedded Python do not mix well. On the other hand, I'm not sufficiently clever enough to debug multi-threaded code under the best of circumstances, so my applications are single-threaded anyways.

Also, the Python community isn't very friendly to people who embed rather than extend, so getting help from them can be an exercise in frustration if you embed.

Share this post


Link to post
Share on other sites
Quote:
Original post by wendigo23
In all my messing around I have turned to favor extending. It makes things so much more flexible. For example, I have a module coded in C(++) on windows that handles all the graphics and sound. A friend of mine wanted to play my game on his mac, so what did I do? I reimplemented the same module API in pure python using pygame. Instant mac port without needing a mac. Imagine doing that if you embedded.

Thanks for that; it's good to know that approach worked for you in practice. I might then plan on writing the whole thing in Python with pygame (or some other game library) first before considering rolling in my own functions.

Quote:
On C vs C++: all my extensions are C++ classes, and I use plain functions to wrap them into python, it's pretty easy once you do it once and then you can just cut and paste. Or you can use boost which has a pretty crazy set of python things.

I've only glanced at the documentation for Boost's python links, so I got the impression it was mainly for embedding. If it's also for extending that would be good, although I'm not sure if I need all the extra functionality C++ provides if I'm just writing extension libraries for Python.

Share this post


Link to post
Share on other sites
Quote:
Original post by Trapper Zoid
I've only glanced at the documentation for Boost's python links, so I got the impression it was mainly for embedding. If it's also for extending that would be good, although I'm not sure if I need all the extra functionality C++ provides if I'm just writing extension libraries for Python.


Actually, it's mostly for extending. The basic documentation actually is missing a few details if you want to embed.

Share this post


Link to post
Share on other sites
Quote:
Original post by SiCrane
I personally prefer embedding Python with C++. Basically I find when embedding Python it's easier to debug the C++ code and harder to debug the Python code compared to extending Python. However, debugging Python code is relatively painless when compared to debugging C++ code, and when debugging C++ code you really need all the help you can get.

Is that because it's harder to debug the embedded stuff in isolation from the rest of the program? If I were to extend rather than embed, would it be wise to consider writing native C or C++ wrapper code for the purposes of unit testing?

Quote:
There are downsides to this. In addition to the points in the article you mentioned, multiple threads and embedded Python do not mix well. On the other hand, I'm not sufficiently clever enough to debug multi-threaded code under the best of circumstances, so my applications are single-threaded anyways.

Me too; my apps haven't been advanced enough for me to worry about multi-threading. Consider everything I plan to do to be single-threaded for now.

Quote:
Also, the Python community isn't very friendly to people who embed rather than extend, so getting help from them can be an exercise in frustration if you embed.

That sucks; is there a strain of elitism in the Python community?

Share this post


Link to post
Share on other sites
I personally have only done work embedding with Boost.Python. I think really the only extra step is starting up the interpreter. Then depending on what you're doing you may need to call Python from C++.

Boost.Python's syntax is odd. When I first saw it I wondered how it even compiled properly. Its not too bad once you delve into it, but there are definitely a few gotchas.

You can also use the Python C API to link up your code, but something automagic is way better if you have a lot of code. There is also a generator for Boost.Python called Py++ that'll do all the binding code for you.

Share this post


Link to post
Share on other sites
Quote:
Original post by Trapper Zoid
Is that because it's harder to debug the embedded stuff in isolation from the rest of the program? If I were to extend rather than embed, would it be wise to consider writing native C or C++ wrapper code for the purposes of unit testing?

It's more of a combination of the fact because working with DLLs sucks in general and due to the way that I approach debugging. For example, leaving a Python instance open can keep the DLL you're working on loaded, so I'd constantly find that the linker wouldn't write to the file. People with different coding habits may find that debugging extension modules to be about the same.

For unit testing, unit tests in Python should be fine.
Quote:

That sucks; is there a strain of elitism in the Python community?

Try to find me a programming language community that doesn't have a strain of elitism.

Share this post


Link to post
Share on other sites
Quote:
Original post by Trapper Zoid
From my understanding most commercial games out there today that use Python use it as an embedded scripting language. However I've read the article in favour of embedding here (Extending vs. Embedding: There is Only One Correct Decision), and it makes some good arguments. My gut feeling is that it's better conceptually to think of the whole program as being in Python with library calls for the computationally extensive parts, plus I can prototype the bulk of it in Python until performance becomes an issue. However there may be some downside to this that as a novice learning Python might not see.


This article doesn't make any good arguments it is fluff at best and misleading at worst. Any programmer who is surprised or confused by python being embed in stead of extended is either not bright enough to call themselves a programmer or is surprised and confused by the use of python at all. I mean really can you name a programmer who can program in python but doesn't know C/C++? If you can find a few that have learned python and can't learn C then I don't think you want them working on your project.

I think and has been said previously embedding is the way to go at first, as it makes debugging easier which you are going to be doing a lot of when you first start learning the interop thing. In the end you are going to want to extend though as it leverages the rapid development abilities of python and lets you speed up the parts that need it.

Share this post


Link to post
Share on other sites
Quote:
Original post by SiCrane
That sucks; is there a strain of elitism in the Python community?

Try to find me a programming language community that doesn't have a strain of elitism.[/quote]
True; I originally wrote that question as "is there a strain of elitism in the Python community (or at least, more than usual)?" but I edited it down. I guess the Holy Language Wars will continue to be waged for as long as programmers exist, regardless of language.

Thanks for the replies; it's helping me think ahead. Since I'm still new to Python it's probably best for me to continue working up the game development chain, then consider splicing in some library code once I reach a certain level of complexity.

Share this post


Link to post
Share on other sites
Assorted comments, mostly quoting Trapper Zoid...
Quote:
There's two questions I'm unsure of: whether it's better to aim for embedding Python within the library or extending Python with the library; and if extending, whether to plan for using C++ or C for the code.


I'm going to say extending is better.

Firstly, Python simply is not designed to be embedded. There's no getting around that - the API has lots of global state, subinterpreter support is barely existent and not maintained, you typically have to recompile the Python DLL to match your current runtime (including building a debug DLL), etc.

Secondly, it goes against normal software wisdom to write the main part of your app in the lower level language and then to add a higher level language on top. <speculative>These bolt-on solutions probably originated in things like Excel where the need for user-defined formulae grew into scripts, and have crossed over to become prevalent in game development thinking because we are unable to give up the C++ legacy for various reasons.</speculative> Yet developers without such legacy code to worry about will almost always prefer to code in the most expressive, most high level language available, and then drop down to lower levels to optimise, if and when needed.

Quote:
If extending, I'm also not sure whether it would be best to use C or C++ as my language of choice for the library functions.


It's not really an issue. If you use a helper library to extend Python, it'll sort that out for you. If you don't, you'll be binding C functions through necessity. (Probably wrappers for your C++ functions.) Or just build them as .dlls and load them via ctypes.

Quote:
Quote:
There are downsides to this. In addition to the points in the article you mentioned, multiple threads and embedded Python do not mix well.


Me too; my apps haven't been advanced enough for me to worry about multi-threading. Consider everything I plan to do to be single-threaded for now.


Yet Python offers ways to make the move to multi-threading quite easy, eg. the queue module. That's one of the many benefits of working in Python.

Quote:
Quote:
Also, the Python community isn't very friendly to people who embed rather than extend, so getting help from them can be an exercise in frustration if you embed.


That sucks; is there a strain of elitism in the Python community?


It's a community of Python users, who presumably like using Python and consider it the best tool for many tasks. If you go in there asking them how to use the language in a way it wasn't designed for, and as essentially subservient to a language that (a) they are less qualified to help with, and (b) is probably worse for many of the tasks you'll put it to, then they're less likely to be able to offer you anything positive. They will probably be a bit frustrated because if you chose to use it the 'proper' way, they could provide more help.

Share this post


Link to post
Share on other sites
Trapper Zoid,

I thought PyCap might be right up your alley, or at least interesting to you.

Both the games on the website and the sample game in the download are using the PyCap library (Python bindings for the PopCap Framework).

http://www.farbs.org/pycap.html

It has a fishie game (you made that fishie game right?)

Share this post


Link to post
Share on other sites
Quote:
Original post by Boder
It has a fishie game (you made that fishie game right?)

What, another one button fish game! Must be a catching trend...

Thanks for the link; I'll look at that later. Unfortunately I'm currently using Mac OS only, so I can't use PopCap. I'm planning on starting with simple ASCII games until I've got the basic Python data structures down pat, then I'll move on to something with graphics.

Share this post


Link to post
Share on other sites
I would recommend Game Programming with Python. I originally picked it up hoping it would have some info on embedding/extending python, it does but its buried at the back of the book, and to get a grasp of what python was about. It uses PyGame throughout which may be more up your alley. Its fairly informative and an easy read.

Also saw this Beginning Game Development with Python and Pygame which is more recent. Not sure if its good but it seems to be getting decent reviews.

Share this post


Link to post
Share on other sites
Hi,

I already said this here, but I've made a little C/Python engine, with embedded python.

And my principal grief against embedding is that I miss a real Python debugger in my appli.

In the past, I had succes with SPE and HAP, but the process is quite heavy so I tend to always launch the game without any debugger, and to debug with printf.
I have the feeling that by extending, you can always launch from the IDE / debugger, and you have access to much more convivial IDEs.

If I had time, I would change my engine to be able to switch from a extended version to an embedded one. It doesn't look that difficult to support the two.

Hope it helps,

Emmanuel

Share this post


Link to post
Share on other sites

This topic is 3588 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this