Jump to content

  • Log In with Google      Sign In   
  • Create Account

D Bits

Binding D to C

Posted by , in Binding to C, D, Derelict 08 January 2012 - * * * * * · 6,671 views

This is part one of a series on creating bindings to C libraries for the D programming language.

This is a topic that has become near and dear to my heart. Derelict is the first, and only, open source project I've ever maintained. It's not a complicated thing. There's very little actual original code outside of the Utility package (with the exception of some bookkeeping for the OpenGL binding). The majority of the project is a bunch of function and type declarations. Maintaining it has, overall, been relatively painless. And it has brought me a fair amount of experience in getting D and C to cooperate.

As the D community continues to grow, so does the amount of interest in bindings to C libraries. Recently, a project called Deimos was started over at github to collate a variety of bindings to C libraries. There are several bindings there already, and I'm sure it will grow. People creating D bindings for the first time will, usually, have no trouble. It's a straightforward process. But there are certainly some pitfalls along the way. In this post, I want to highlight some of the basic issues to be aware of. For the sake of clarity, I'm going to ignore D1 (for a straight-up "static" binding, the differences are minor, but they do exist.

The first thing to consider is what sort of binding you want, static or dynamic. By static, I mean a binding that allows you to link with C libraries or object files directly. By dynamic, I mean a binding that does not allow linking, but instead loads a shared library (DLL/so/dylib/framework) at runtime. Derelict is an example of the latter; most (if not all) of the bindings in the Deimos repository the former. There are tradeoffs to consider.

D understands the C ABI, so it can link with C object files and libraries just fine, as long as the D compiler understands the object format itself. Therein lies the rub. On Posix systems, this isn't going to be an issue. DMD (and of course, GDC and, I assume, LDC) uses the GCC toolchain on Posix systems. So getting C and D to link and play nicely together isn't much of a hassle. On Windows, though, it's a different world entirely.

On Windows, we have a variety of object file formats to contend with: COFF, OMF, ELF. DMD, which uses an ancient linker called Optlink, outputs OMF objects. GDC, which uses the MingW backend on Windows, outputs ELF objects. I haven't investigated LDC yet, but it uses whichever backend LLVM is configured to use. Meanwhile, the compiler that ships with Visual Studio outputs objects in the COFF format. What a mess!

This situation will improve in the future, but for now it is what it is. And that means when you make a C binding, you have to decide up front whether you want to deal with the mess or ignore it completely. If you want to ignore it, then a dynamic binding is the way to go. Generally, when you manually load DLLs, it doesn't matter what format they were compiled in, since the only interaction between your app and the DLL happens in memory. But if you use a static binding, the object file format determines whether or not the app will link. If the linker can't read the format, you get no executable. That means you have to either compile the C library you are binding with a compiler that outputs a format your D linker understands, use a conversion tool to convert the libraries into the proper format, or use a tool to extract a link library from a DLL. Will you ship the libraries with your binding, in multiple formats for the different compilers? Or will you push it off on the users to obtain the C libraries themselves? I've seen both approaches.

Whichever way you decide to go really doesn't matter. In my mind, the only drawback to dynamic bindings is that you can't choose to have a statically linked program. I've heard people complain about "startup overhead", but if there is any it's negligble and I've never seen it (you can try it with Derelict -- make an app using DerelictGL/SDL/SDLImage/SDLMixer/SDLNet/SDLttf and see what kind of overhead you get at startup). The only drawback to static bindings is the object file mess. But with a little initial work upfront, it can be minimzed for users so that it, too, is negligible.

Once you decide between static and dynamic, you aren't quite ready to roll up your sleeves and start implementing the binding. First, you have to decide how to create the binding. Doing it manually is a lot of work. Trust me! That's what I do for all of the bindings in Derelict. Once you develop a systematic method, it goes much more quickly. But it is still drastically more time consuming than using an automated approach. To that end, I know people have used SWIG and a tool called htod. VisualD now has an integrated C++-to-D converter which could probably do it as well. I've never used any of them (which is really incredible when I think about it, given how precious little time I usually have), so I can't comment on the pros and cons one way or another. But I do know that any automated output is going to require some massaging. There are a number of corner cases that make an automated one-for-one translation extremely difficult to get right. So regardless of your approach, if you don't want your binding to blow up on you down the road, you absolutely need to understand exactly how to translate D to C. And that's where the real fun begins.

That's going to have to wait for another post, though. It's Sunday evening here and I've got things to do. In part two, I'll talk about function declarations. I think they're easier to cover than types, which I'll save for a third post. Until then, Happy New Year!

DerelictGLFW and a Word on Binding D to C

Posted by , in D, Derelict 24 June 2011 - - - - - - · 1,916 views
D, Derelict
Recently, in the Derelict forums, someone asked me if I wanted him to update his GLFW binding, based on the old Derelict, for the Derelict 2 branch so that I could add it to the trunk. We had a GLFW binding in Derelict before, but removed it due to issues with building the GLFW shared libraries. Derelict, you see, is designed to load shared libraries manually and cannot link with static libs. That was quite some time ago. In the intervening years, a new maintainer has taken over the GLFW project and made some improvements to it.

So I've had it in the back of my mind to give GLFW another look at some point for possible inclusion into Derelict 2. Today, I did. A new version was released late last year (2.7) and a new branch that streamlines the API (3.0) has been started. I really like the new branch. So, being the spontaneous sort of fellow I am, I decided I wanted a binding to it. I knocked one up in just over 30 minutes. It's now sitting in my local "scratch" copy of Derelict, waiting to be compiled and tested. Given that it's 1:30 am as I type this, I don't think I'm going to get to it just yet. Tomorrow for sure.

I won't be adding this new binding to the Derelict repository just yet. GLFW 3.0 is still in development. So, just as with the binding I've begun for SDL 1.3 (which will become SDL 2 on release), I'll wait until the C library is nearing a stable release before I check it in.

Making D bindings to C libraries is not a difficult thing to do. It's just tedious if you do it manually, like I do. I have a system I've grown used to now that I've done so many of them. It goes reasonably quick for me. Some people have experimented with automating the process, with mixed results. There are always gotchas that need to be manually massaged, and they might not be easily caught if the whole process is automated. One example is bitfields.

D doesn't support bitfields at the language level. There is a library solution, a template mixin, that Andrei Alexandrescu implemented in the std.bitmanip module. I don't know how compatible it is with C. I've only had to deal with the issue once, when binding to SDL 1.2, but that was before the std.bitmanip implementation. Besides, it's a D2 only solution and Derelict has to be compatible with both D1 and D2. So what I did was to declare a single integer value of the appropriate size as a place holder. The bits can be pulled out manually if you know the order they are in on the C side. I could have gone further by adding properties to pull out the appropriate bits, but I never did the research into how different C compilers order the bitfields on different platforms.

Another issue that crops up is dealing with C strings. For the most part, it's not a problem, but if you are new to D it's a big gotcha. Like C strings, D strings are arrays of chars (or wchars or dchars as the case may be). But, char strings in D are 8-bit unicode by default. Furthermore, D arrays are more than just a block of memory filled with array values. Each array is conceptually a struct with length and ptr fields. Finally, and this is the big one, D strings are not zero terminated unless they are literals. Zero-terminated string literals are a convenience for passing strings directly to C functions. Given a C function prototype that takes a char*, you can do this:

someCFunc("This D string literal will be zero-terminated and the compiler will do the right thing and pass the .ptr property");

If you aren't dealing with string literals, you need to zero-terminate the string yourself. But there's a library function that can do that for you:

import std.string;

// the normal way

// or using the Universal Function Call Syntax, which currently only works with D arrays

A lot of D users like the Universal Function Call Syntax and would like to see it work with more types instead of just arrays. Personally, I'm ambivalent. The way it works is that any free function that takes an array as the first argument can be called as if it were a member function of the array.

Going from the C side to the D side, you would use the 'to' template in std.conv:
// with the auto keyword, I don't need to declare a char* variable. The compiler will figure out the type for me.
auto cstr = someCFuncThatReturnsACharPtr();

// convert to a D string
auto dstr = to!(string)(cstr);

// templates with one type parameter can be called with no parentheses. So for to, this form is more common.
auto dstr = to!string(cstr);

Another gotcha for new users is what to do with C longs. The D equivalent of nearly all the C integral and floating point types can be used without problem. The exceptions are long and unsigned long. D's long and ulong types are always 64-bit, regardless of platform. When I initially implemented Derelict, I didn't account for this. D2 provides the aliases c_long and c_ulong in core.stdc.config to help get around this issue. They will be the right size on each platform. So if you see 'long' in a C header, the D side needs to declare 'c_long'. I still need to go through a few more Derelict packages to make sure they are used.

The issues that crop up when actually implementing the binding aren't so frequent and are easily dealt with. Sometimes, though, you run into problems when compiling or running applications that bind to C.

D applications can link directly to C libraries without problems, as long as the object format is supported by the compiler. On Linux, this is never an issue. Both DMD and GDC can link with elf objects. Problems arise on Windows, however. The linker DMD uses, OPTLINK, is ancient. It only supports OMF object files, while many libraries are compiled as COFF objects. If you have the source code and you can get it to compile with Digital Mars C++, then you're good to go. Otherwise, you have to use the DigitalMars tool coff2omf, which comes as part of the Digital Mars Extended Utilities Package. Cheap, but not free. Then you still might face the problem that the COFF format output by recent versions of Visual Studio causes the tool to choke. There are other options, but it's all nonsense to me. That's one of the reasons when I made Derelict I decided that it would only bind to libraries that come in shared form and they will be loaded manually. Problem solved. But there are other issues.

In a past update to DMD (not sure which), the flag '--export-dynamic' was added to the DMD config file (sc.ini) on Linux. So that means that every binary you build on Linux systems with DMD has that flag passed automatically to gcc, the backend DMD uses on Linux. Normally, not an issue. Until you try to build a Derelict app. The problem is that Derelict's function pointers are all named the same as the functions in the shared library being bound to. This causes conflicts when the app is built with --export-dynamic on Linux, but they don't manifest until run time in the form of a segfault. Removing the flag from sc.ini solves the problem. One of these days I need to ask on the D newsgroup what the deal with that is.

I know all of this could sound highly negative, giving the impression it's not worth the hassle. But, seriously, that's not the case. I have been maintaining Derelict for seven years now. Many bindings have come and gone. Version 2 currently supports both D1 and D2, as well as the Phobos standard library and the community-driven alternative, Tango. I can say with confidence that D works very well with C the large majority of the time. And for anyone planning to use D to make games, you will need to use C bindings at some level (Derelict is a good place to start!). As for binding with C++... well, that's another story that someone else will have to tell.

Dolce Progress and A Word on Mixins

Posted by , in D, Derelict, Dolce 11 June 2011 - - - - - - · 789 views
D, Derelict, Dolce
My time has been limited lately, but the core of Dolce continues to evolve. I think it's converging into something fairly decent. I'm still not completely happy with it, but I believe I'm most of the way there. As usual, I'm getting wild ideas for new projects. Like a renewed interest in building a 2D library from scratch in D. Or a more generic framework with pluggable backends for Allegro, SDL, SFML, and GLFW. You know, like I explicitly said at the beginning I had no interest in making. This time, though, I refuse to let myself get distracted. So I'm digging my heels in.

Recently, I've added event handling, mapping keycodes or mouse button ids to delegates. The first pass is usable, but it can be improved a good deal. D's delegates are great for this sort of thing. But delegates aren't what I want to talk about today.

Until now, my test app has only been using the keyboard. With the inclusion of the event handler, I added mouse support. While testing it out, I kept getting a puzzling result. Allegro has no constants for mouse buttons. They are numbered from 1 to whatever the maximum number is (which you can find out via al_get_mouse_num_buttons()). But my event handler kept getting button 0 passed to it, no matter which button I pushed. al_get_mouse_num_buttons was correctly reporting 3 (I dumped my 8-button gaming mouse some time ago). After several minutes of digging around, I started to suspect my DerelictAllegro binding. Sure enough, I had left out 4 fields from the struct declaration. I patched that up and all is well.

Looking at the binding source reminded me of a D feature that would be worth mentioning here. In my last post, I briefly mentioned D's string mixins. There is another type of useful mixin in D that I've made use of in the Allegro binding: the template mixin.

Allegro's event structs are all declared with the following macro at the top:

#define _AL_EVENT_HEADER(srctype)                	\
   ALLEGRO_EVENT_TYPE type;                      	\
   srctype *source;                              	\
   double timestamp;

So the mouse event structure looks like this:

typedef struct ALLEGRO_MOUSE_EVENT
   struct ALLEGRO_DISPLAY *display;
   int x,  y,  z, w;
   int dx, dy, dz, dw;
   unsigned int button;
   float pressure;

D has no #defines, and no preprocessor at all. But this C idiom can easily be replicated with template mixins. Here's what I did in the DerelictAllegro binding:

// First, declare the template that you intend to use as a mixin.
template _AL_EVENT_HEADER(T)
	T* source;
	double timestamp;

// Then, mix it in.
	int x, y, z, w;
	int dx, dy, dz, dw;
	uint button;
	float pressure;

Simple. String mixins could also be used here.

 // You would want to use a compile time function with a signature something like this
string allegroEventHeader(string type) { ... }

// Then call it like so:

Personally, I prefer template mixins in situations like this.

Take a look at the template mixin documentation over at the shiny, new d-programming-language.org website for more info.

Got the Bug Again

Posted by , in Derelict, Dolce 17 May 2011 - - - - - - · 1,007 views
Derelict, Dolce
  • Up and Running

The last time I made a complete game was too long ago to think about. Over the years, I've lurked at GDNet semi-religiously. I've false-started more projects than I can count. Games, game libraries, game engines, all using several languages. Java, C, C++, D. What have I got to show for it? A hard drive full of archived source code. Well, not full, but you get the idea.

Despite the fact that I haven't completed a game in a long, long time, I have learned a lot over the years. I also have one personal project that hasn't faded away yet. For seven years (seven!?!), I've been maintaining Derelict, a collection of bindings to C libraries for the D Programming Language. Game-centric of course. Derelict 2, a branch I've been working on off-and-on for a while, will soon be moved to the trunk. Derelict 2 includes bindings for more libraries than the original. One of those, recently added, is Allegro 5.

Back in the late 90s, when I was first learning C at the ripe old age of 27, Allegro was the first game development library I ever used. I hadn't looked at it in years, until I noticed someone mention Allegro 5 recently on a forum somewhere (maybe here). I looked into it, liked what they've done with it, and immediately added a binding for it to Derelict. But that wasn't all. Working with Allegro again has reawakened the gamedev bug in me.

My latest endeavor is Dolce, a simple framework that sits on top of Allegro. The goal is to handle all of the boilerplate and provide some utilities that can reduce the amount of code needed to write games with Allegro. It's Allegro-specific, with no plans or desire to make it abstract enough to use with other libraries (like SDL or SFML, which Derelict also provides bindings for). I'm writing it for myself, to help me get some games off the ground, but I'll be putting it up in a repository somewhere once I've got it polished and documented. Most likely DSource or github. Dolce is being written with version 2 of the D Programming Language.

I've been deeply involved with D off and on since before I started Derelict. It has come a long way since the early days and has turned into a language that makes programming fun for me again. If you're interested in following D news but don't want to bother with the newsgroups or mailing lists, subscribe to my other D blog, The One With D. Here at D Bits I'll be posting my game-related D adventures. That includes writing about Dolce, Derelict, and my future unfinished game projects.

Working on Dolce has already brought me one instance of good luck. Being a big fan of KOTOR and Dragon Age: Origins, six or seven months ago I bought Mass Effect on Steam. To my great disappointment, it was totally unplayable on my system thanks to slideshow graphics. After a significant effort in troubleshooting with no results, I gave up and uninstalled. Fast forward to a few days ago. I got the Dolce test app up and running for the first time in VisualD, the D plugin for Visual Studio. I noticed a good deal of D3D debug spew in the console. Then I remembered that I had turned on the debug version of D3D some time last year for another project I was working on. A couple of days later, while reading a blog post about Mass Effect 2, the old light bulb popped up. I turned off the debug runtime and reinstalled Mass Effect. No more slideshow. And I can finally experience the adventures of Commander Shepard. Right when I don't need the distraction, of course.

August 2016 »

282930 31    

Recent Entries