Jump to content

April 2017 »

23 24 2526272829

Recent Entries

- - - - -

DerelictGLFW and a Word on Binding D to C

D Derelict
4: Adsense

Recently, in the Derelict forums, someone asked me if I wanted him to update his GLFW binding, based on the old Derelict, for the Derelict 2 branch so that I could add it to the trunk. We had a GLFW binding in Derelict before, but removed it due to issues with building the GLFW shared libraries. Derelict, you see, is designed to load shared libraries manually and cannot link with static libs. That was quite some time ago. In the intervening years, a new maintainer has taken over the GLFW project and made some improvements to it.

So I've had it in the back of my mind to give GLFW another look at some point for possible inclusion into Derelict 2. Today, I did. A new version was released late last year (2.7) and a new branch that streamlines the API (3.0) has been started. I really like the new branch. So, being the spontaneous sort of fellow I am, I decided I wanted a binding to it. I knocked one up in just over 30 minutes. It's now sitting in my local "scratch" copy of Derelict, waiting to be compiled and tested. Given that it's 1:30 am as I type this, I don't think I'm going to get to it just yet. Tomorrow for sure.

I won't be adding this new binding to the Derelict repository just yet. GLFW 3.0 is still in development. So, just as with the binding I've begun for SDL 1.3 (which will become SDL 2 on release), I'll wait until the C library is nearing a stable release before I check it in.

Making D bindings to C libraries is not a difficult thing to do. It's just tedious if you do it manually, like I do. I have a system I've grown used to now that I've done so many of them. It goes reasonably quick for me. Some people have experimented with automating the process, with mixed results. There are always gotchas that need to be manually massaged, and they might not be easily caught if the whole process is automated. One example is bitfields.

D doesn't support bitfields at the language level. There is a library solution, a template mixin, that Andrei Alexandrescu implemented in the std.bitmanip module. I don't know how compatible it is with C. I've only had to deal with the issue once, when binding to SDL 1.2, but that was before the std.bitmanip implementation. Besides, it's a D2 only solution and Derelict has to be compatible with both D1 and D2. So what I did was to declare a single integer value of the appropriate size as a place holder. The bits can be pulled out manually if you know the order they are in on the C side. I could have gone further by adding properties to pull out the appropriate bits, but I never did the research into how different C compilers order the bitfields on different platforms.

Another issue that crops up is dealing with C strings. For the most part, it's not a problem, but if you are new to D it's a big gotcha. Like C strings, D strings are arrays of chars (or wchars or dchars as the case may be). But, char strings in D are 8-bit unicode by default. Furthermore, D arrays are more than just a block of memory filled with array values. Each array is conceptually a struct with length and ptr fields. Finally, and this is the big one, D strings are not zero terminated unless they are literals. Zero-terminated string literals are a convenience for passing strings directly to C functions. Given a C function prototype that takes a char*, you can do this:

someCFunc("This D string literal will be zero-terminated and the compiler will do the right thing and pass the .ptr property");

If you aren't dealing with string literals, you need to zero-terminate the string yourself. But there's a library function that can do that for you:

import std.string;

// the normal way

// or using the Universal Function Call Syntax, which currently only works with D arrays

A lot of D users like the Universal Function Call Syntax and would like to see it work with more types instead of just arrays. Personally, I'm ambivalent. The way it works is that any free function that takes an array as the first argument can be called as if it were a member function of the array.

Going from the C side to the D side, you would use the 'to' template in std.conv:
// with the auto keyword, I don't need to declare a char* variable. The compiler will figure out the type for me.
auto cstr = someCFuncThatReturnsACharPtr();

// convert to a D string
auto dstr = to!(string)(cstr);

// templates with one type parameter can be called with no parentheses. So for to, this form is more common.
auto dstr = to!string(cstr);

Another gotcha for new users is what to do with C longs. The D equivalent of nearly all the C integral and floating point types can be used without problem. The exceptions are long and unsigned long. D's long and ulong types are always 64-bit, regardless of platform. When I initially implemented Derelict, I didn't account for this. D2 provides the aliases c_long and c_ulong in core.stdc.config to help get around this issue. They will be the right size on each platform. So if you see 'long' in a C header, the D side needs to declare 'c_long'. I still need to go through a few more Derelict packages to make sure they are used.

The issues that crop up when actually implementing the binding aren't so frequent and are easily dealt with. Sometimes, though, you run into problems when compiling or running applications that bind to C.

D applications can link directly to C libraries without problems, as long as the object format is supported by the compiler. On Linux, this is never an issue. Both DMD and GDC can link with elf objects. Problems arise on Windows, however. The linker DMD uses, OPTLINK, is ancient. It only supports OMF object files, while many libraries are compiled as COFF objects. If you have the source code and you can get it to compile with Digital Mars C++, then you're good to go. Otherwise, you have to use the DigitalMars tool coff2omf, which comes as part of the Digital Mars Extended Utilities Package. Cheap, but not free. Then you still might face the problem that the COFF format output by recent versions of Visual Studio causes the tool to choke. There are other options, but it's all nonsense to me. That's one of the reasons when I made Derelict I decided that it would only bind to libraries that come in shared form and they will be loaded manually. Problem solved. But there are other issues.

In a past update to DMD (not sure which), the flag '--export-dynamic' was added to the DMD config file (sc.ini) on Linux. So that means that every binary you build on Linux systems with DMD has that flag passed automatically to gcc, the backend DMD uses on Linux. Normally, not an issue. Until you try to build a Derelict app. The problem is that Derelict's function pointers are all named the same as the functions in the shared library being bound to. This causes conflicts when the app is built with --export-dynamic on Linux, but they don't manifest until run time in the form of a segfault. Removing the flag from sc.ini solves the problem. One of these days I need to ask on the D newsgroup what the deal with that is.

I know all of this could sound highly negative, giving the impression it's not worth the hassle. But, seriously, that's not the case. I have been maintaining Derelict for seven years now. Many bindings have come and gone. Version 2 currently supports both D1 and D2, as well as the Phobos standard library and the community-driven alternative, Tango. I can say with confidence that D works very well with C the large majority of the time. And for anyone planning to use D to make games, you will need to use C bindings at some level (Derelict is a good place to start!). As for binding with C++... well, that's another story that someone else will have to tell.

Jun 25 2011 11:00 AM
I didn't even know about c_long and c_ulong. I'll see about mentioning these in the documentation, thanks for the heads up.
Jun 26 2011 08:49 PM
I learned of them only recently. And I'm glad. Most of the instances of longs in C were declared as ints in Derelict. With more Derelict users moving to 64-bit platforms, this might have been a hard problem to track down. Luckily, it hadn't surfaced yet. It's only an issue in three or four packages that, I believe, aren't as frequently used as the others.
Jul 11 2011 03:21 AM
The "--export-dynamic" flag was added when stack traces were implemented. IIRC without that flag stack traces only show function addresses,
but no function names (Thinking about it: this seems like a hack to solve a huge bug. I guess all phobos code should actually be declared with 'export'). If I understood correctly, your issue is that the names of the derelict function pointers now clash with the original names.

This is the way the function pointers are declared in derelict:
module test;

    int function(void* test) FT_Init_FreeType;

This means: I want a function pointer to a extern© function. But it also means: I want that function pointer to be extern© --> have C name mangling.
The resulting symbol is the following: "00000000 B FT_Init_FreeType".

The alternative is this:
module test;

alias extern(C) int function(void* test) FTInitFunc;
FTInitFunc FT_Init_FreeType;

This doesn't declare the pointer itself as extern©, the resulting symbol is: "00000000 B _D4test16FT_Init_FreeTypePUPvZi" . So it uses D name mangling and there shouldn't be clashes anymore.

However, I can't find a way to declare this on one line / without an alias?
Jul 11 2011 06:05 AM
I wouldn't have believed it would work, using different calling conventions for the pointer and the alias. But, it seems that it does. I don't look forward to implementing it, though. That's basically how Derelict 1 was handled in the beginning, the difference being that the pointers themselves were wrapped in the extern© as well (that, and we started out with typedefs before switching to aliases later). I zapped all of that to reduce code, making new packages easier to implement and maintain. But, if that's the only way to solve the problem, then I suppose I have little choice.

Thanks for pointing that out.
Jul 12 2011 03:15 AM
Well, there really should be some way to do exactly this without the alias, but I can't make it work.

BTW: It's not that surprising that it works: The D part is just a data field/variable, it doesn't have any calling convention. extern© only affects it's name mangling, nothing else. Just like extern© char* test; and char* test; are exactly the same except for name mangling. The underlying type can still be everything, even a C function.
Jul 12 2011 05:44 AM
Yes, that makes perfect sense. For some reason, I've always believed that function pointers were somehow treated specially by the compiler.

Note: GameDev.net moderates comments.