• entries
155
100
• views
94072

On Python, game development and everything

Help needed

While PyEigen 0.2 is mostly ready for release, I'm stuck on weird problems with GCC on both Windows (MinGW) and Linux. PyEigen compiles fine on Visual C++, but GCC gives different errors depending on the version. On GCC 4.4 I get these mysterious errors:

source/iterator/matrix2fiterator.cpp:39: error: insufficient contextual information to determine typesource/iterator/matrix2fiterator.cpp:40: error: insufficient contextual information to determine typesource/iterator/matrix2fiterator.cpp:41: error: too many initializers for 'PyTypeObject'

On GCC 4.5, apparently every template function in PyEigen results in an undefined reference, even though the implementations are (or at least should be, as MSVC finds them) included in the object files. Frankly I don't have the faintest idea why this is happening. I even completely restructured the code to try if the (admittedly confusing) header relationships were to blame for the errors, but nothing changed.

If someone wants to help (pretty please) and try to figure out what's wrong, you can grab the code here: http://code.launchpad.net/~knarkles/pyeigen/trunk. You need the Python headers and Eigen 2 to compile; feel free to ask me for help on getting a build setup up an running.

PyEigen 0.2 feature complete

After a couple of weeks of hiatus, the 0.2 release of PyEigen is now feature complete. What's left is some testing and documentation, then release. I postponed quaternions and transformations to 0.3 because variable-size matrices were plenty of work for a single release and single programmer already; I don't want too much time between releases at such an early stage. If I'm not too busy, PyEigen 0.2 should be released in a week or two.

I did some benchmarking on 1000x1000 matrices. Insanely, enabling SSE2 instructions speeded up that test case by 10x; I was expecting a 4x increase at most, since SSE instructions handle 4 times the data per instruction. NumPy is slightly faster for 1000x1000 matrix multiplication, but there's an obvious optimization in PyEigen that will hopefully make it faster again. That will have to wait for 0.3 though, since it's a bit tedious to implement. For those interested: there's an extra matrix copy in all PyEigen functions and operators that return a value, which I can probably optimize away.

BTW, if you want more frequent progress updates, you can follow my Twitter feed. [smile]

PyEigen 0.2 teaser

I just want to assure everyone that development of PyEigen didn't stop at the 0.1 release and I've put many hours into the next release already. I have almost finished rewriting everything using C++ templates; might as well take advantage of C++ features since Eigen requires a C++ compiler anyway. Lines of code have been cut almost in half and maintenance will be much easier in the future. Here are some of the major upcoming features (subject to change, completed features will be in bold):

• Python 3 support

• Matrix iterators & slicing

• Variable-size vectors and matrices

In other news, I just found about PyBindGen, which looks very promising compared to other binding generators, and might be worth checking out for a future version of PyEigen. Can someone speak for or against it? I'm mostly worried about performance, since I want PyEigen to be as fast as possible; after all, that's the sole reason I started the whole project.

Update (2010-03-30):
Matrix iterators done, slicing dropped (you can just convert to a list first).

Update (2010-05-03):
Quaternions and transformations postponed to the next release, variable-size matrices done.

PyEigen 0.1 released

PyEigen 0.1 has finally been released! See the (still) very limited blog page, grab it at the Python Package Index and get involved at the Launchpad project page.

Here's the announcement:

I'm happy to announce PyEigen, a new linear algebra module for Python that's many times faster than existing solutions. Notably, PyEigen is about 10x faster than NumPy for 4x4 matrix multiplication and 4x faster than cgkit 1.2.0, the fastest current solution.

Python Package Index:
http://pypi.python.org/pypi/PyEigen/

Development blog:
http://www.brainfold.org/blog

PyEigen is a Python wrapper for the C++ linear algebra library Eigen.

PyEigen is currently considered PRE-ALPHA quality software and has not been widely tested outside the unit tests. The API is not stable yet and might change between releases without warning. Testing and all kinds of feedback are welcome however! Compatibility reports with different operating systems, compilers and Python versions are especially welcome.

Features

The first release of PyEigen includes basic support for fixed-size vectors (2-4-dimensional) and matrices (2x2, 3x3 and 4x4).

PyEigen progress

Development of PyEigen is going really well. All the features I want for a 0.1 release are basically done and there's just testing and documentation left now. For version 0.1 documentation I'll just write some docstrings and a readme file, but I want full unit tests for the module. Vectors and matrices are really low-level stuff and writing C code is quite prone to errors, so I want to make sure everything works and it can't be crashed.

I'd say you can expect a first release in about a week. It's going to include support for fixed-size matrices (2x2, 3x3 and 4x4) and column and row vectors (2-, 3- and 4-dimensional). Major game development -related features missing include transformations and quaternions; they are probably coming in the 0.2 release along with coefficient-wise (aka element-wise) operations. I'll probably add support for variable-size matrices and vectors later. Compatibility with ctypes and NumPy would probably also be useful. Feature requests are welcome. [smile]

More types and results

I implemented some new types for PyEigen, including a 4x4 matrix class and benchmarks for it. Same methods as last time, but quite different results.

PyEigen is still fastest by far, which is promising. This time it was about 5-10x faster than cgkit1, which was again the second fastest. You might also notice that vectypes is missing. For this test, it was so slow that I had to leave it out; other results would have been invisible if I had fitted the vectypes results in the graph. In the worst case, it was over 1000x slower than PyEigen! Euclid is missing addition and scalar multiplication scores since it doesn't support those operations. NumPy performed much better. There has to be a faster way to do vector cross product in NumPy, since matrix multiplication was much faster than cross product. But it's not a big deal anymore since PyEiglet is so much faster than anything else. [smile]

Surprising results

I did some preliminary benchmarking today and got very interesting results. I have only wrapped a 3D vector class so far, so I tested a couple of operations (add, multiply, dot & cross product) against the libs I mentioned in the previous post: NumPy, euclid, vectypes and cgkit. For cgkit, I tested both 1.2.0 and 2.0 alpha 9. All other libs were the latest version. I tested using the Python timeit module with 1,000,000 calls and 3 repeats per test and took the lowest number. Repeated test results generally differed only by milliseconds.

The immediately obvious surprise is the abysmal performance of NumPy especially in cross products. I don't think NumPy optimizes for fixed-size arrays; I would have been better off with the pure-Python euclid and vectypes modules.

The other surprises were euclid vs vectypes and cgkit1 vs cgkit2. Euclid and vectypes are both by Alex Holkner of Pyglet fame. In both cases, the newer library (vectypes and cgkit2) was also slower.

Of course the most positive surprise for me was the performance of PyEigen. It's only a trivial wrapping, but was about 2-8x faster than the best alternative, cgkit1. I'm very happy with the results and definitely going to continue development of the library / wrapper. [smile]

PyEigen

Oops, looks like my GDNet+ subscription ended without me noticing. I'm back now.

Anyway, I have a new project called PyEigen, a wrapper for the C++ linear algebra library Eigen. I just submitted the first batch of code and progress is good, at least so far. [smile] I'm hoping for an initial release within a month or so.

The whole thing started when I profiled my shooter project and found that matrix calculations take up a huge amount of time. Apparently NumPy isn't really fast enough for 3D games, and I couldn't find any replacements. While otherwise looking good, euclid and vectypes are pure Python so they aren't going to make performance any better. cgkit is C++ with a Python wrapper so looking better already, but... it's using Boost.Python, which apparently isn't very fast. Also, it's lacking SSE instrumentation and other optimizations included in Eigen. So I decided to wrap Eigen.

As I already found out, Boost.Python is slow and looked too complex for such a simple library anyway. I tried Cython next, but its C++ support is (still) very limited and the lack of support for C++ references destroyed any hope of wrapping Eigen, which relies heavily on them. My final option before resorting to manual wrapping using the Python C API was SWIG, but I had problems getting even a simple wrapper to compile. Besides, I don't really like how SWIG generates function wrappers and a separate Python module that calls those wrappers instead of generating a Python C module directly.

So I was left with only the final option: Python C API. I feared it at first because I've never worked with it and it seemed really complex. It is complex, but not nearly as bad as I though, especially as Eigen has such a simple API. For Bullet, I'm sure I'll use Cython or some other wrapper generator, but for Eigen the Python C API is just fine.

Anyway, since there doesn't seem to be anything like this out there, I decided to make it an open source project so hopefully other people in the same situation won't have to jump through the same hoops as I did. I'll post progress reports, releases and especially benchmarks against the other options as soon as I have them.

Stealth prototyping

Yes, I know, I have been quiet on this blog for over half a year now. That doesn't mean I'm dead; quite the opposite, since I have been really busy with Real Life and professional game development. [smile] However, as some of you might have noticed from my Twitter feed, I have been working on a new game. It started as a vertical-scrolling shoot'em'up but has transformed into something else entirely. At the moment I have just a load of ideas and an early prototype, so I don't want to write too much about it yet, but more on the project later. And yes, it's in Python. [smile]

As for Artillery Brawl, the code became a mess and destroyed my motivation to continue the project, which is why I started the new game to begin with. I might start Artillery Brawl again based on this new codebase, but for now I'm working on the shooter.

I hope to write more about the new project and its progress again in the near future, but in the meantime, the best way to follow the project is my Twitter feed.

Work!

My latest excuse for not getting anything done on Artillery Brawl or any other projects is that I just landed a job in the game industry! I didn't want to announce it on the blog before I got a contract, but today I got one as a programmer for Bugbear Entertainment. It's interesting and kind of refreshing to seriously program in C++ for a change.

New city, new portfolio

I have succesfully moved to Helsinki, though the apartment is of course still a mess. While organizing the apartment, I also decided to reorganize my Portfolio page. Please check it out, all kinds of comments are very welcome!

I should be hearing about the job next week.

Sampling some C++

Since I have the job interview on Thursday, I decided to brush up my rusty C++ skills and make a quick work sample while at it, as I haven't really worked with the language since about 2004. The position I'm applying for is UI programmer, so I made a simple but relatively flexible "game style" menu system. At first, I had to look up lots of things in documentation, but at least I knew what to look for, and overall it went surprisingly smoothly. I spent about two evenings on it, so don't expect anything polished.

If you want to check out the demo and some old stuff of mine, see http://www.brainfold.org/portfolio/. I'm releasing the code in public domain, but check copyright.txt for the font copyright. Apart from the demo, my time has been spent preparing for the move to Helsinki on Wednesday, so again, don't expect any real updates for at least a week or two.

Moving on

I haven't updated for a while again. Let's get gamedev news out of the way first. I haven't worked on Artillery Brawl for a couple of weeks; instead, I have tried to get the card game ready for testing. I made some prototype cards by putting pieces of paper in sleeves that are opaque on one side. This has the advantage that I don't have to go through the trouble of making the back sides of the cards look identical, so it's much easier to make changes. Unfortunately, while the rules are basically ready for testing, I ran out of card sleeves, and when I bought some more I accidentally bought a different size... Now I'll have to wait until next week to buy some more, sigh.

Anyway, about next week... I have a job interview at a game company (wish me luck!) and I'll be moving back to Helsinki where I used to live four years ago before I moved here to Tampere. I also have a couple of deadlines looming for university courses, so I'll be quite busy for the next week or two! At least I already have internet connection at the new apartment, so my connection to civilization won't be severed. [wink] See you in a couple of weeks!

Good news, bad news

The good news is, suddenly I got a whole lot more free time for game development.

The bad news is, I don't have a job anymore (yay for the economic situation).

I didn't just freeze and panic though; instead, I called a friend at a game company I was applying to for the summer, and told I might be available for work a bit earlier after all... I hope to hear more from them next week, so wish me luck!

As for other news, I have been working on Artillery Brawl a lot, though still no particle effects... I think they might be cursed, since every time I get close to implementing them, I run into problems or some other thing I just have to fix. Anyway, the good thing about this is that the entity system and a lot of the code in general is starting to look really good and reusable. This is how I should have approached building a reusable codebase to begin with; writing a game, then generalizing the solutions that work.

As part of my effort to integrate Lepton for particle effects, I have been implementing a system of triggers and payloads to make projectiles more interesting in general. The system is heavily inspired by the similar payload system for my old artillery game, Arty, which I wrote about earlier, but of course redesigned and modernized to make it more generic and cleaner. I haven't settled on an interface yet, and looks like I'll have to make yet more changes to the entity system, but every change is a step closer to perfection, right? Anyway, about triggers and payloads...

Any number of triggers and payloads can be attached to entities. For now, they will only be used for projectiles (artillery shells), but later they could be used for many other gameplay applications. Triggers can react to collisions, height from ground, proximity to units, or they could activate after a delay. Almost anything is possible, really, but I'm starting with just collision triggers. Payloads are triggered, well, by triggers; they are the "meat" of the projectile system. Again, I'm starting simple with just kinetic (contact damage based on mass, velocity and an armor piercing multiplier) and explosive (yay, particle effects and shrapnel projectiles) payload, but almost anything would be possible. For example, it would be easy to make a cluster bomb that explodes above ground (height trigger attached to directed explosive payload) into many small bomblets (individual projectiles with contact triggers attached to explosive payloads).

More on the triggers and payloads when I get the system implemented.

Lepton

Remember when I said "if I don't hit any further roadblocks", I would add particle effects to Artillery Brawl this week? Well, guess what...

The bad news is, I had to spend a whole evening porting lepton to Windows so Visual C++ would compile it.

The good news is, I'm now the maintainer and tester of the Windows port of lepton, so hopefully this doesn't happen to anyone anymore. The first Windows installer for Python 2.6 has been released; check the lepton home page. Unfortunately, I couldn't get it compiling for Python 2.5, since I don't own Visual C++ 2003 and building extension modules with the free Visual C++ Toolkit 2003 compiler is a pain in the arse. I might try it again later, but for now, I'm considering myself defeated.

Anyway, you should all check out lepton, it's an awesome library for creating fast particle effects in Python games and other applications, and works out of the box with Pyglet, Pygame and any OpenGL-based applications. It's also easily extensible so you can add your own custom effects to particles. I'll be sure to post any further experiences (and the video I promised) as soon as I get it properly integrated with Artillery Brawl.

Quickie

Just a quick update... I have been fixing all the stuff that was broken since the change to multi-part units and then some, and everything should be ready for finally implementing particle effects. I even implemented fake sweeping collisions for shells by making them face their velocity vector and making their collision rectangle longer than the shell "sprite." There are still some collision problems when the shooting unit is next to a steep cliff, but it's acceptable for now, pending a real solution.

I'm going away for the weekend, but if I don't hit any further roadblocks, next week should see Artillery Brawl with some kind of particle effects, and of course, a second video of the game! [smile]

Last of 2008

I just wrote my last lines of Python for this year. I haven't got to particle explosions (I'm going to try lepton for those) or other interesting bits yet, and the code doesn't actually even run at the moment...

Read the rest at my blog.

Holiday coding

As promised (to myself and readers), I have continued game development during the holidays. Not much, mind you, I've mostly been enjoying the chance of being lazy for a change, but it's something, and I hope I don't get any more months-long breaks in the near future. :)

I have been working on Artillery Brawl, with stuff that was left incomplete when I stopped working on it in July. I haven't started on prototyping shooting yet, because I have some related stuff to do first. I have completely changed the units are represented in the code.

Read the rest at my blog.

I'm back to GDNet! While being busy with studies, I let my GDNet+ account expire and hardly updated my blog... But both have changed now! I'm still going to keep my blog proper on Wordpress at my domain, but continue posting sneak peeks here at GDNet, since it almost feels like home. [smile]

Read the new post at my blog.

Summer

Oops, looks like Real Life ate me for July. I haven't even worked on the university project much, but I'll have to get it ready before September. Some good news regarding my studies though: I'm very likely to finish all my remaining courses during the next academic year. After that, I'm planning to move back to the capital, Helsinki, to write my Master's thesis at some company. If everything goes according to the plan, I'll finish my degree late next year, yay! Of course, things rarely go exactly like planned...

Hopefully I'll get more game development done in the Autumn, since it looks like I'll still be quite busy during August.

Comment at my new blog.

I want to

• Finish these stupid courses - I have to use Java again for the other one, eww

• Prototype a game idea I had while driving home today - but not spend over a week on it

• Learn to draw some vector art - because it should be easier for "programmer art" than pixels

• Get a new job for Autumn

• Get some sleep

Artillery Brawl: Prototyping

Lately I've read a lot about game design (Danc's blog has been especially enlightening). I'm now convinced I have to prototype a few "risky" aspects of Artillery Brawl's gameplay to make sure what I'm doing is actually fun before wasting effort on other parts of the game.

Read the rest at my new blog.

Artillery Brawl: The concept

I'm starting to write about the game design for Artillery Brawl on this blog instead of keeping it in my head and my private wiki. As opposed to Red Nebula, whose concept is wide open, a "playground" of sorts, the vision I have for Artillery Brawl is very focused, and I have already dismissed several gameplay ideas that simply don't fit that vision. To start with, here's the concept for the game.

Read the rest at my new blog.

Box of horrors

Today I adapted the nice setuptools-based plugin module from Spineless to more easily support different physics toolkits in my code, and played around a bit trying to write a Box2D plugin.

Read the rest at my new blog.