Sign in to follow this  
Markie

Cross-Compiler Compatiblity of Plugins

Recommended Posts

Hi all, I'm planning on making a game with a component object model. To be able to add objects later on, I want to make all objects plugins. Now I read a really detailed article about plugins: Building Your Own Plugin Framework, by Gigy Sayfan My question is this now: The Binary Compatibility Problem and the fact that there is no standard C++ ABI, no standard how compilers must translate C++ to machine code, creates the problem, that different compilers (such as on Linux or Windows, or even on the same platform), even different compiler versions can create different vtables (virtual function tables) for virtual functions of derived classes (which I need). And this in turn can make plugins compiled with a different compiler than the main program not work (when vtables are created differently). Gigy Sayfan suggests sticking to pure C in plugins to get around this. In fact, one really can "simulate" C++ objects with straight C (read "Pure C"). One simply creates structs with function pointers, functions which accept their structs as first arguments, inheritance with base structs as first data member, etc. Now I've figured out a way how to implement my object model in straight C, however, I would still like to use STL containers such as maps, etc. My question is this: If I create pure C "objects", but these objects contain C++ STL containers such as "map", "vector", etc. will that still create a vtable that can break when the plugin is compiled on a different compiler? Programming my own container types, such as "map", "vector", "list", etc. in straight C seems an awful lot of work. Or should I just go with pure C++ (perhaps that's just easier in the long run) and forget about cross-compiler and cross-platform compatibility? (But then, that seems to negate a big part of what's so cool about plugins, namely that EVERYBODY, no matter what compiler he has, can create and program an add-on...) Mark

Share this post


Link to post
Share on other sites
Most people doing something similar to what you are trying to do would write the base application in C++ and then use something like Boost Python to allow third party developers to add their own Python scripts (could be Ruby, Perl or whatever the principle is the same) in order to extend the application.

Share this post


Link to post
Share on other sites
Try:

In your Application:


extern "C" {
void* CreateSomeObject()
{
return new SomeObject;
}
void DeleteSomeObject(void object)
{
delete (SomeObject*)object;
}
int CallSomeFunction(void* object,int arg1)
{
return ((SomeObject*)object)->SomeFunction(arg1);
}
// etc...
}




In the Plugin:

void* object = CreateSomeObject();
CallSomeFunction(object,10);
DeleteSomeObject(object);




You'll have to keep all your containers on their end of the application but as long as the object is obscured in this way, you should be able to have anything you like in it... except having one of your extern "C" functions return anything without C linkage.

you can typedef the void pointer to something like SomeObjectHandle to make it more meaningful.

Share this post


Link to post
Share on other sites
Quote:
My question is this: If I create pure C "objects", but these objects contain C++ STL containers such as "map", "vector", etc. will that still create a vtable that can break when the plugin is compiled on a different compiler?


No, just having more fields won't touch the vtable for that object. So you could safely access the other fields in that object, but the foreign binary probably won't be able to safely access the contents of the map/vector/etc.

I think the best strategy (which the Building Your Own Plugin Framework article mentions at the end), is to pass around opaque pointers to C++ objects, and provide pure C accessors for those objects. You can even go further and write a shim C++ class that wraps around the C accessors, if you want to.

Edit: Yeah, just like Kwizatz said. Then if you wanted to put a C++ shim on top of that, it would look like this:


class MyShimClass
{
public:
MyShimClass() {
_data = CreateSomeObject();
}
~MyShimClass() {
DeleteSomeObject(_data);
}
void callSomeFunction(int i) {
CallSomeFunction(_data, i);
}
private:
void* _data;
};

Share this post


Link to post
Share on other sites
Quote:
Original post by Markie
Or should I just go with pure C++ (perhaps that's just easier in the long run) and forget about cross-compiler and cross-platform compatibility?

This.

Quote:

(But then, that seems to negate a big part of what's so cool about plugins, namely that EVERYBODY, no matter what compiler he has, can create and program an add-on...)

Who cares, be pragmatic. Don't make your life miserable ("C objects"...) just because of a few who don't use the main compiler for a given platform. Just tell them to use MSVC on Windows and gcc on Linux or OSX. Problem solved. A lot of professional software does it this way.

Quote:
Original post by Cromulent
Most people doing something similar to what you are trying to do would write the base application in C++ and then use something like Boost Python to allow third party developers to add their own Python scripts (could be Ruby, Perl or whatever the principle is the same) in order to extend the application.

That is not a plugin system then, but a scripting system. A plugin based application relies on a multi-component concept, where each component fully interfaces with the system (and possibly with each other) at a low level, while being compiled in the same language as the main system to ensure maximum performance. Think 3DSMax, for example, which is a very heavily plugin based system (the entire application is built by plugins around a basic skeleton framework). Imagine a plugin such as VRay (global illumination renderer) written in Python...

Share this post


Link to post
Share on other sites
Quote:
Original post by Yann L
Quote:
Original post by Markie
Or should I just go with pure C++ (perhaps that's just easier in the long run) and forget about cross-compiler and cross-platform compatibility?

This.

Quote:

(But then, that seems to negate a big part of what's so cool about plugins, namely that EVERYBODY, no matter what compiler he has, can create and program an add-on...)

Who cares, be pragmatic. Don't make your life miserable ("C objects"...) just because of a few who don't use the main compiler for a given platform. Just tell them to use MSVC on Windows and gcc on Linux or OSX. Problem solved. A lot of professional software does it this way.


Well, the problem can go deeper than that. For example, a standard class, such as std::string has been different between versions of the same compiler and different for debug and release builds, which causes problems. Namely, it can blow up if created in a module that "sees" it one way and then if used or deleted in the other. It's very easy to violate the one-definition rule across DLL boundaries. Of course, there's solutions for this if you're always using a version built with the same settings and libraries.

I wonder if there's a library that does kind of what SWIG does, except using a C interface to bridge C++ DLLs instead of using the C interface to bridge between languages.

Share this post


Link to post
Share on other sites
Quote:
Original post by Rydinare
Well, the problem can go deeper than that. For example, a standard class, such as std::string has been different between versions of the same compiler and different for debug and release builds, which causes problems. Namely, it can blow up if created in a module that "sees" it one way and then if used or deleted in the other. It's very easy to violate the one-definition rule across DLL boundaries. Of course, there's solutions for this if you're always using a version built with the same settings and libraries.

Well, it depends. The main problem is DLL-boundary memory and resource allocation. But this can be solved in many different ways. Also, under Windows at least, each DLL is a separately compiled unit. Internal symbols are resolved at compile time, and only symbols that are explicitly exported are runtime linked. This is different under Linux, btw, which will always runtime bind everything, even internal dependencies. This is a major pain in the ass when developing plugin systems for Linux or OSX.

Anyway, if you're careful, then the CRT versions don't have to match 100%. You can, for example, debug a plugin (compiled as debug, linked with the debug CRT) while it is running as a host on a release application (compiled as release, linked to the release CRT).

Share this post


Link to post
Share on other sites
It's always a pain having to remember which version of VS (including service pack!) I need to use to develop for each version of Max/Maya/Gamebryo/whatever.

But it is TEN TIMES as much pain to deal with certain middleware packages which have, for ABI-agnostic shits and giggles, reimplemented virtual function binding and discarded traditional RAII and standard library classes.

ABI problems are annoying, but then you get them right and they stay right. The workarounds are infinitely worse.

Share this post


Link to post
Share on other sites
Quote:
Original post by Yann L
Quote:
Original post by Rydinare
Well, the problem can go deeper than that. For example, a standard class, such as std::string has been different between versions of the same compiler and different for debug and release builds, which causes problems. Namely, it can blow up if created in a module that "sees" it one way and then if used or deleted in the other. It's very easy to violate the one-definition rule across DLL boundaries. Of course, there's solutions for this if you're always using a version built with the same settings and libraries.

Well, it depends. The main problem is DLL-boundary memory and resource allocation. But this can be solved in many different ways. Also, under Windows at least, each DLL is a separately compiled unit. Internal symbols are resolved at compile time, and only symbols that are explicitly exported are runtime linked. This is different under Linux, btw, which will always runtime bind everything, even internal dependencies. This is a major pain in the ass when developing plugin systems for Linux or OSX.

Anyway, if you're careful, then the CRT versions don't have to match 100%. You can, for example, debug a plugin (compiled as debug, linked with the debug CRT) while it is running as a host on a release application (compiled as release, linked to the release CRT).


Good points. I was referring to a specific case, in the case of STL containers, starting with VS 2005, there is extra storage used for debugging purposes, so if you're using STL, you can no longer mix debug and release builds even from the same compiler version, which is obnoxious.

Share this post


Link to post
Share on other sites
Quote:
Original post by Sneftel
It's always a pain having to remember which version of VS (including service pack!) I need to use to develop for each version of Max/Maya/Gamebryo/whatever.

But it is TEN TIMES as much pain to deal with certain middleware packages which have, for ABI-agnostic shits and giggles, reimplemented virtual function binding and discarded traditional RAII and standard library classes.

ABI problems are annoying, but then you get them right and they stay right. The workarounds are infinitely worse.


No arguments here. It's why I dislike the use of COM in C++, unless it's utterly necessary, because most C++ paradigms are broken in COM.

But, that being said, COM had a broader purpose. If you simply want to wrap custom classes with a C-API underneath and serialize the rest, I think it can be ok.

Share this post


Link to post
Share on other sites
Guys, you are soo cool! :-)
Thanks a *LOT* for all your great answers!
I must say, you all and this forum must and will get credits if my project (MRPGS) ever does come to be (despite common sense and average intelligence suggesting otherwise... ;-). If it does, that's in part to this cool forum and you cool people for helping out with just the right answers when they were needed! Fits like a glove!! :-)
Side note: At times I wonder, are programmers a different species than homo sapiens? Seems like non-programmers compete for everything and only programmers really share what they have WITHOUT the slightest own profit. Cool! :-)


ABI-agnostic shits and giggles
(This should be the title of an own sub-forum, it's so cool! :-)
Quote:
Original post by Sneftel
It's always a pain having to remember which version of VS (including service pack!) I need to use to develop for each version of Max/Maya/Gamebryo/whatever.

But it is TEN TIMES as much pain to deal with certain middleware packages which have, for ABI-agnostic shits and giggles, reimplemented virtual function binding and discarded traditional RAII and standard library classes.

ABI problems are annoying, but then you get them right and they stay right. The workarounds are infinitely worse.

Sneftel, you are so cool! :-)
I particularly love your brief, no bullshit an precise to the point communication. Awesome! That's EXACTLY the kind of direct supporting wisdom I was really hoping for! Thanks!


Quote:
Original post by Yann L
Quote:
Original post by Markie
Or should I just go with pure C++ (perhaps that's just easier in the long run) and forget about cross-compiler and cross-platform compatibility?

This.

Quote:
Original post by Markie
(But then, that seems to negate a big part of what's so cool about plugins, namely that EVERYBODY, no matter what compiler he has, can create and program an add-on...)

Who cares, be pragmatic. Don't make your life miserable ("C objects"...) just because of a few who don't use the main compiler for a given platform. Just tell them to use MSVC on Windows and gcc on Linux or OSX. Problem solved. A lot of professional software does it this way.

You're right up there with Sneftel, Yann L! Thanks a ton for your also extremely direct and to the point wisdom and communication! :-)
I think being pragmatic is exactly what I needed to hear.

Thanks to all other repliers as well and sorry if I didn't express clearly enough what I was looking for. Right now I feel Sneftel and Yann L really hit it awesomely.

Here's a little "decision tree" I just came up with:

Problem 1:
Quote:
A.) Linux and Mac OSX Compatibility:
- Plugins are a pain in the ass, use plugins OR Linux / Mac OSX, but not both.
OR:
Quote:
B.) Windows Compatibility only:
- Plugins are "easy" and well used, use plugins, but forget about Linux and Mac OSX.

Decision:
As Linux and Mac OSX are not very wide-spread among gamers and because, with my currently limited knowledge, I believe plugins to be a VERY important part of my project, MRPGS, (extensibility is key in a big project like this), I will, despite not being a particular fan of Microsoft, decide for option B.


Problem 2:
Quote:
C.) No Compiler Compatibility:
Say what compiler (MSVS), version and service pack must be used to write a plugin, just like the big uns, 3DStudioMax, Maya, etc. do. AND:
Use SIMPLE STANDARD C++ AND get the C++ community to kick in by being able to use their favorite language out-of-the-box, without any pain in the rear "ABI-agnostic shits and giggles" :-))
OR:
Quote:
D.) Cross-Compiler Compatibility:
Allow ANY compiler to be used, BUT go for a more (how much more is open and unknown at this moment) complicated C programming model witch simulates C++ objects and potentially disgruntles C++ programmers much more when having to learn how to write a plugin, than having to use a specific compiler brand, version and service pack.

Decision:
Combined by what Sneftel and Yann L said, as well as by what the BIG ones such as 3DStudioMax, etc. are doing, I'll try NOT to go the "stony path" here. The whole point of my using plugins are 1.) Allowing others to write add-ons and 2.) To facilitate my own programming as well as enforcing / promoting general encapsulation / modularization.
Point 2.) is served better by C.) and point 1.) is, I must and will take Sneftel's word here, better served by C.) as well. Therefore I will go for option C.


*Phew!*, that pretty much closed the "case" for me! - Stomped a 60-foot orc into the ground and left nothing but a little head sticking out!! :-))
Thanks all!

Mark

[Edited by - Markie on March 20, 2010 5:56:46 AM]

Share this post


Link to post
Share on other sites
Glad we could help.

Quote:
Original post by Markie
Quote:
A.) Linux and Mac OSX Compatibility:
- Plugins are a pain in the ass, use plugins OR Linux / Mac OSX, but not both.
OR:
Quote:
B.) Windows Compatibility only:
- Plugins are "easy" and well used, use plugins, but forget about Linux and Mac OSX.


Plugins are indeed more annoying to manage under Linux/OSX, due to the nature of shared object linking on these platforms. But they're still perfectly doable.

a) You can easily get symbol collision under these systems, unless you're careful about what you expose and what not. You'll need to use the correct symbol visibility control options to gcc and use namespaces wisely. All that is less critical under Windows, but it's good practice to also keep your plugin API clean and well documented (ie. don't just put a _declspec(dllexport) on every single class in your code).

b) If you have missing internal dependencies in your plugin, then Windows will complain while linking the DLL. This is easy and convenient. Linux and OSX however will complain when you try to load the plugin with your host application, which is annoying. Well, you get used to it.

On missing external dependencies (import symbols), all OS will complain when loading the plugin, including Windows.

c) The only thing I have never managed to get working under Linux/OSX is throwing exceptions over DLL boundaries, if these exceptions are only instantiated by templates. The reason is some weird symbol instantiation 'feature' (read: bug) in gcc. None of the workarounds I found on the net worked.

Nothing prevents you from starting to write your plugin system under Windows, and later transition to Linux and OSX. If you go the full C++ with implied compiler route Sneftel and I suggested, then that transition will be very smooth. You'd just have to wrap around LoadLibrary, dlopen and friends. That's just a few lines of code.

Share this post


Link to post
Share on other sites
You could also create/use a bytecode type plugin where you compile it to a common format for any platform and then your program interprets it to what is required.

im looking at implementing a cross platform plugin system and I am considering using LLVM as a backend, and write a custom soure-LLVM intermediate code compiler.

LLVM will do JIT optimizations on its bytecode, and can usually reduce the program to machine code and therefore should ( with a little extra load time) run plugins at native speed.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this