Back to General and Gameplay Programming

lib hell

General and Gameplay Programming Programming

Started by fir May 26, 2014 10:15 AM

11 comments, last by kunos 9 years, 10 months ago

fir

-460

Author

May 26, 2014 10:15 AM

Im not quite see it clearly (got not so much experience) and maybe some will help me to claryfy this picture

Im mostly talking about windows platform (though I may be courious if on other platforms things looks better)

As far as i know there is no standarisation of lib (static libraries format)

so there we got some static lib-hell (my own name for this) where given static lib may only works with some compiler say VS or say GCC and not across them all (as it should be) [I am speeking on one specyfic system

mostly win32 or win64]

1) is this true?

2) do this incompatibility occur between across verisons of compilers

or it only occurs between different compilers (I mean can lib not works if

made with older version of compiller with the newer version of the same

brand compiler or in one compiler it will work for all verions)

3) are the gcc/vs/intel/what else partially compatible - or in general mostly not compatible

4) is there a way of mending a incompatible lib in some way ?

tnx for the answer (it seem to me tah it may be hard to answer thiss fully but at least partially - i would like to build some wiev on this static lib - hell in windows, how the things look like)

BitMaster

8,652

May 26, 2014 11:41 AM

I would strongly advise compiling a static library with the exact compiler and the exact build settings you intend to use it with. You might be able to work without that provided you have a pure C library, but I don't have significant experience in that area.

As soon as C++ enters the picture it's just not worth it. Even for the same version of the same compiler setting a different flag can break ABI-compatibility (for example the safe iterator define on earlier MSVCs would do that). That can lead to extremely annoying, very subtle, hell to find problems.
On the other hand the Windows version of gcc had one or two (unintentional) ABI breaks some time between 4.8.0 to 4.8.2. Check the changelogs for details.

Also note that is not a problem limited to static libraries. Unless you can make several restrictions regarding choice of runtime and other build settings an interface between dynamic libraries is also extremely limited (regarding allowed types and memory ownership).

On Windows you can usually get around compiling your own libraries if you stick to MSVC and you do not divert from the standard 'Debug' and 'Release' configurations. I would still suggest you get into the habit of compiling your own libraries though. You will need it sooner or later anyway and it's better to get skill points in compile-foreign-code without a really nasty deadline at your back.

Jan2go

2,057

May 26, 2014 12:33 PM

1) is this true?

Yes.

2) do this incompatibility occur between across verisons of compilers
or it only occurs between different compilers (I mean can lib not works if
made with older version of compiller with the newer version of the same
brand compiler or in one compiler it will work for all verions)

Incompatibilities between different compilers are quite likely. It may work when using only different version of the same compiler (VS doesn't allow it though, don't know about other compilers).

3) are the gcc/vs/intel/what else partially compatible - or in general mostly not compatible

From what I know Intel is compatible with VS (not vice versa though). And I was able to use a MSVC dll with MinGW, however in most cases this will not work properly.

4) is there a way of mending a incompatible lib in some way ?

If you have access to the source code you may try to compile it with the compiler you need. If you don't have the source you are out of luck in the most cases.

ongamex92

3,287

May 26, 2014 04:52 PM

As far as i know there is no standarisation of lib (static libraries format)

*.lib/a file is just an archive that contains *.obj files (+ eventually some minimalistic metadata header file).

EDIT: And those obj files are passed to the linker. So YES the sample compilation settings is a *MUST(only few exceptions are possible. I personally don't allow them in my build system, It is obvious that they will make you kill yourself).

SeanMiddleditch

17,596

May 26, 2014 11:09 PM

It's not usually considered a "static lib hell" because you have almost no good reason to actually try mixing different .lib files. Linux makes it more common, but that's only because of the walled garden repository approach of most Linux distros (if they recompile anything in the repository, they usually recompile every single package in the entire distro for the repository, too). One of the many arguments Linux folks try to use to bar commercial/binary software is that it may be compiled against different versions of static libs (and then you run into problems with IPC protocol version changes, data version changes, etc.). Really, on any OS, the only static libs you should try to use at all are the ones you compiled yourself for that application, and you should keep those to an absolute minimum (especially if the libs need to interact with the wider system in any way).

Note that .lib files on Windows are a little bit more tricky due to how .DLLs work. Older .lib files that are shims for a .DLL will typically work in newer compilers. This is because the .lib contains (almost) no code.

If anything, the real hell with Windows libraries it that there's no standard for architecture names and configurations. e.g. one lib might have its 32-bit debug name be foo32d.lib while another is just bar.lib while their 64-bit release names are foo.lib and bar64r.lib. It makes it harder to have a single unified project configuration. This is true for DLLs as well. Nuget for C++ (CoApp) for Windows is an attempt to solve this that still has a lot of ground to cover.

Sean Middleditch – Game Systems Engineer – Join my team!

cr88192

1,572

May 26, 2014 11:30 PM

Im not quite see it clearly (got not so much experience) and maybe some will help me to claryfy this picture

Im mostly talking about windows platform (though I may be courious if on other platforms things looks better)

As far as i know there is no standarisation of lib (static libraries format)
so there we got some static lib-hell (my own name for this) where given static lib may only works with some compiler say VS or say GCC and not across them all (as it should be) [I am speeking on one specyfic system
mostly win32 or win64]

1) is this true?

simple answer: yes.

less simple answer: there is some informal/de-facto standardization of LIB and OBJ file-formats, at least as far as Windows goes (as AR archives and COFF objects, known as OBJ/LIB with MSVC and O/A with GCC). however different compilers have various minor format differences (in terms of structures both within LIB/AR and the COFF objects), making things problematic.

some compilers had also used different formats entirely (for example, a few older compilers had used a 32-bit version of the OMF format and its associated LIB format instead of COFF/AR). there are also differences in the contents of various header-files (ex: what exactly is a "FILE"?), there are differences in terms of parts of the C ABI, and often a very different C++ name mangling scheme and ABI (so C++ object-files are usually incompatible).

the end result is that while sometimes it is possible to force things to link between compilers, usually the result will either not work (at all or at least not correctly) and/or be prone to crashing.

2) do this incompatibility occur between across verisons of compilers
or it only occurs between different compilers (I mean can lib not works if
made with older version of compiller with the newer version of the same
brand compiler or in one compiler it will work for all verions)

it can apply between compiler versions for the same compiler as well.

for example, both MSVC and GCC have changed their ABIs in various ways between compiler versions, so code linked between different compiler versions is prone to not work and/or result in crashes.

this is generally less of an issue for DLLs, as the basic C calling conventions (cdecl and stdcall) are pretty much frozen, and also both MSVC and GCC agree in most areas WRT the basic C ABI, and MS is adverse to changing things which will break existing software.

however, the C++ ABI is compiler specific, and there are a lot of minor edge cases where the C ABI differs between compilers (namely WRT passing/returning structs, the handling of types like "long double" and "__m128", ...), so some care is still needed.

it may also be observed that many/most Windows libraries tend to use a fairly constrained C subset for their APIs (and COM and similar are built on top of the C ABI).

3) are the gcc/vs/intel/what else partially compatible - or in general mostly not compatible
4) is there a way of mending a incompatible lib in some way ?

tnx for the answer (it seem to me tah it may be hard to answer thiss fully but at least partially - i would like to build some wiev on this static lib - hell in windows, how the things look like)

when possible, build things from source all with the same version of the same compiler.

Current Status / Downloads: http://cr88192.mooo.com:8080/wiki/index.php/BGB_Current_Status

YouTube Channel: http://www.youtube.com/user/BGBTech

Main Page: http://cr88192.mooo.com:8080/wiki/index.php/Main_Page

fir

-460

Author

May 27, 2014 08:43 AM

so it looks bad..

Is there any physical reason against the making lib (static libs format) (saying on one platform) working across all compilers and verisons?

Seem no.. (at least speaking about c, I am using c interface libraries only)

so this is bad, and this is guilt of compiler society - it should be standarized and this hell wouldnt be existing - every lib for win32 would be working with every compiler - i would call it normal situation

[as to compiling libs from sources.. you seem to suggest everyone is doing that? I am not good at it.. Probably it brings its own bag of obstacles in the box , ?]

as to reasons (when speaking about c interface libs, I am only use) I understand the main source of incompatibility are just the symbol names (underscoress and so), Or there is something other above that? I assume most compilers use coff libs today, old borland omf I incidentally know a bit because back then i was using b55 heavy by a couple of years, but (Im not sure) isnt it is conversable to coff with some tool (agner for as i vaguelly remember had such a tool

on his page)

So as i said I do not see it clearly but It vaguelly seems to me that reasons of this incompatibility between static libs are not so much Heavy.. Is there something more than can be spoiled above just the linker symbol names? is someone here able to answer this?

frob

46,221

May 28, 2014 03:21 AM

In practice it isn't a big deal.

Even when you are using the same compiler, you can have the problems when you build it twice with incompatible options.

Build everything from the source yourself. If you have a system that you cannot build from source (which is often a bad idea) then do what you can to remain compatible while trying to get the source. Note that even the major compilers provide source code to their standard libraries so you can build them from source if you need.

Hodgman

52,717

May 28, 2014 03:45 AM

In practice it isn't a big deal.

Assuming you've got the source to everything

If you're using closed-source C++ libraries, you need to be sure that the vendor has compiled it using the exact same compiler (and compiler options) to what you need...

It would probably be better to just ask the vendor to give you a C version of their library instead

. 22 Racing Series .

frob

46,221

May 28, 2014 05:55 AM

In practice it isn't a big deal.
Assuming you've got the source to everything :cool:

In two decades, with teams as small as 5 people, I have not seen it as an actual problem. Maybe it is a problem for other people.

It is easily avoided, either by getting contracts that require source code or by using major reputable companies that provide builds in all shapes and sizes, and as part of the contract will provide them for you as needed.

Games have a fun feature that approximately zero major games use the standard memory allocation systems. External libraries that fail to use custom allocators for everything tend to cause no end of problems, so 3rd parties typically ship with source for that reason alone.

I only remember one time (in 21 years) not having either source or compatible builds was an issue, and it was a simple matter: We told our boss (the studio owner) that we could not use the product due to lack of source and to cancel the contract; we were going to cut the features that relied on the library. The following day their technology lead contacted our lead programmer directly begging for us to take them back (apparently they only had 2-3 total customers), and after passing it on to the company owner that we could use the product if we were allowed to view and potentially modify the source, we not only got the source but negotiated about half the cost as well.

lib hell

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

lib hell

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

Reticulating splines