Jump to content
  • Advertisement

D Bits

  • entries
    32
  • comments
    46
  • views
    127435

About this blog

Game development with the D Programming Language.

Entries in this blog

 

DerelictGLFW and a Word on Binding D to C

Recently, in the Derelict forums, someone asked me if I wanted him to update his GLFW binding, based on the old Derelict, for the Derelict 2 branch so that I could add it to the trunk. We had a GLFW binding in Derelict before, but removed it due to issues with building the GLFW shared libraries. Derelict, you see, is designed to load shared libraries manually and cannot link with static libs. That was quite some time ago. In the intervening years, a new maintainer has taken over the GLFW project and made some improvements to it.

So I've had it in the back of my mind to give GLFW another look at some point for possible inclusion into Derelict 2. Today, I did. A new version was released late last year (2.7) and a new branch that streamlines the API (3.0) has been started. I really like the new branch. So, being the spontaneous sort of fellow I am, I decided I wanted a binding to it. I knocked one up in just over 30 minutes. It's now sitting in my local "scratch" copy of Derelict, waiting to be compiled and tested. Given that it's 1:30 am as I type this, I don't think I'm going to get to it just yet. Tomorrow for sure.

I won't be adding this new binding to the Derelict repository just yet. GLFW 3.0 is still in development. So, just as with the binding I've begun for SDL 1.3 (which will become SDL 2 on release), I'll wait until the C library is nearing a stable release before I check it in.

Making D bindings to C libraries is not a difficult thing to do. It's just tedious if you do it manually, like I do. I have a system I've grown used to now that I've done so many of them. It goes reasonably quick for me. Some people have experimented with automating the process, with mixed results. There are always gotchas that need to be manually massaged, and they might not be easily caught if the whole process is automated. One example is bitfields.

D doesn't support bitfields at the language level. There is a library solution, a template mixin, that Andrei Alexandrescu implemented in the std.bitmanip module. I don't know how compatible it is with C. I've only had to deal with the issue once, when binding to SDL 1.2, but that was before the std.bitmanip implementation. Besides, it's a D2 only solution and Derelict has to be compatible with both D1 and D2. So what I did was to declare a single integer value of the appropriate size as a place holder. The bits can be pulled out manually if you know the order they are in on the C side. I could have gone further by adding properties to pull out the appropriate bits, but I never did the research into how different C compilers order the bitfields on different platforms.

Another issue that crops up is dealing with C strings. For the most part, it's not a problem, but if you are new to D it's a big gotcha. Like C strings, D strings are arrays of chars (or wchars or dchars as the case may be). But, char strings in D are 8-bit unicode by default. Furthermore, D arrays are more than just a block of memory filled with array values. Each array is conceptually a struct with length and ptr fields. Finally, and this is the big one, D strings are not zero terminated unless they are literals. Zero-terminated string literals are a convenience for passing strings directly to C functions. Given a C function prototype that takes a char*, you can do this:


someCFunc("This D string literal will be zero-terminated and the compiler will do the right thing and pass the .ptr property");


If you aren't dealing with string literals, you need to zero-terminate the string yourself. But there's a library function that can do that for you:


import std.string;

// the normal way
someCFunc(toStringz(someString));

// or using the Universal Function Call Syntax, which currently only works with D arrays
someCFunc(someString.toStringz());


A lot of D users like the Universal Function Call Syntax and would like to see it work with more types instead of just arrays. Personally, I'm ambivalent. The way it works is that any free function that takes an array as the first argument can be called as if it were a member function of the array.

Going from the C side to the D side, you would use the 'to' template in std.conv:

// with the auto keyword, I don't need to declare a char* variable. The compiler will figure out the type for me.
auto cstr = someCFuncThatReturnsACharPtr();

// convert to a D string
auto dstr = to!(string)(cstr);

// templates with one type parameter can be called with no parentheses. So for to, this form is more common.
auto dstr = to!string(cstr);


Another gotcha for new users is what to do with C longs. The D equivalent of nearly all the C integral and floating point types can be used without problem. The exceptions are long and unsigned long. D's long and ulong types are always 64-bit, regardless of platform. When I initially implemented Derelict, I didn't account for this. D2 provides the aliases c_long and c_ulong in core.stdc.config to help get around this issue. They will be the right size on each platform. So if you see 'long' in a C header, the D side needs to declare 'c_long'. I still need to go through a few more Derelict packages to make sure they are used.

The issues that crop up when actually implementing the binding aren't so frequent and are easily dealt with. Sometimes, though, you run into problems when compiling or running applications that bind to C.

D applications can link directly to C libraries without problems, as long as the object format is supported by the compiler. On Linux, this is never an issue. Both DMD and GDC can link with elf objects. Problems arise on Windows, however. The linker DMD uses, OPTLINK, is ancient. It only supports OMF object files, while many libraries are compiled as COFF objects. If you have the source code and you can get it to compile with Digital Mars C++, then you're good to go. Otherwise, you have to use the DigitalMars tool coff2omf, which comes as part of the Digital Mars Extended Utilities Package. Cheap, but not free. Then you still might face the problem that the COFF format output by recent versions of Visual Studio causes the tool to choke. There are other options, but it's all nonsense to me. That's one of the reasons when I made Derelict I decided that it would only bind to libraries that come in shared form and they will be loaded manually. Problem solved. But there are other issues.

In a past update to DMD (not sure which), the flag '--export-dynamic' was added to the DMD config file (sc.ini) on Linux. So that means that every binary you build on Linux systems with DMD has that flag passed automatically to gcc, the backend DMD uses on Linux. Normally, not an issue. Until you try to build a Derelict app. The problem is that Derelict's function pointers are all named the same as the functions in the shared library being bound to. This causes conflicts when the app is built with --export-dynamic on Linux, but they don't manifest until run time in the form of a segfault. Removing the flag from sc.ini solves the problem. One of these days I need to ask on the D newsgroup what the deal with that is.

I know all of this could sound highly negative, giving the impression it's not worth the hassle. But, seriously, that's not the case. I have been maintaining Derelict for seven years now. Many bindings have come and gone. Version 2 currently supports both D1 and D2, as well as the Phobos standard library and the community-driven alternative, Tango. I can say with confidence that D works very well with C the large majority of the time. And for anyone planning to use D to make games, you will need to use C bindings at some level (Derelict is a good place to start!). As for binding with C++... well, that's another story that someone else will have to tell.

Aldacron

Aldacron

 

Dolce and Free Functions

There's one thing that's missing from D that I really wish it had: namespaces. There have been discussions about this on the D newsgroup in the past. Walter Bright, D's daddy, has taken the position that D's modules *are* namespaces. In a way, they are. But, not really.

For those new to the party, D eliminates the need for header files. Interface and implementation are all part of the same code unit, called a module. Instead of #including headers, you import modules. Modules can reside in a package hierarchy, much like Java classes. So, given this source file:


// This is the module 'bar' in the package 'foo'.
module foo.bar;

void baz()
{
// Do something.
}

You would use it like this:

module myapp.mymod;

// Import the foo.bar module so its symbols are accessible by this module
import foo.bar;

void myfunc()
{
// baz is visible, so it can be called like this
baz();

// Or, with the fully-qualified name
foo.bar.baz();
}

So yes, a module namespace does exist, but it isn't enforced by default. That means that there can be conflicts if multiple imported modules export functions/types with the same signature. Now, from the user side, it can be enforced two ways. First, is by using static import.

static import foo.bar;

Now, everything you access in the foo.bar module must be prefixed with "foo.bar". You can go one better and add an alias that let's you call foo.bar by another name:

alias foo.bar Foobar;

This allows you to call Foobar.baz(), but it's still foo.bar.baz() that is being enforced under the hood. Remove the static keyword from the import, and baz() will still be accessible by itself. The alias is just an alternative, not an enforced namespace.

The second approach is to use a feature called "named imports", which effectively combines the two steps above, but with enforcement of the given name instead of the module name.

import Foobar = foo.bar;

Now, everything you access in the foo.bar module must be prefixed with "Foobar".

These approaches are all fine, but they put the onus on the user to create his own namespace. As a solution for solving namespace conflicts in client code, I think they're just peachy. In fact, I really like them. But from the perspective of a library designer, I have one major issue with it. I want to wrap my functions in a namespace, so that the client can't have conflicts to begin with and doesn't have to remember to use a named or static import.

Currently, in Dolce, I'm presented with the dilemma of how to handle certain functions that really ought to be free functions. I really don't want to take the C approach with function naming conventions, because it just doesn't mesh with the parts of the library that aren't free functions. On the other hand, I don't want to pollute the global namespace with function names that could easily clash with other libraries or client code. One option is to use static methods in a struct, or a non-instantiable class. But, to me, that just really feels dirty. Plus, every new type you define in a D program adds a TypeInfo instance to the final executable. It's not a lot, but it's just useless bloat for a class that is serving as a namespace hack. It offends my sensibilities, I suppose.

So one more option that I'm considering is having a sort of wrapper module that publicly imports the modules with free functions, giving them a name.

module foo.bar;

public import Bar = foo.realbar;

Imported symbols, by default, are accessible only in the module that imports them. So if module A imports module B, and you then import module A , you can't access B's symbols. However, if module A publicly imports B, then you can access B's symbols just by importing module A. So this enforces the namespace on the client if they import this module. Furthermore, clients will have a choice. They can import foo.bar and get access to the Bar namespace, or they can import foo.realbar and get the free functions, or create their own namespace if they need it. I only see a couple of drawbacks. The only problem I have is deciding if this really improves the situation or not.

Of course, the other option is just not to worry about it. Phobos is full of free functions, after all. Ultimately, my point of view is that I'm creating this library for myself, so whatever makes me happy is the way I'm going. I just have to decide which approach makes me happy.

Aldacron

Aldacron

 

Binding D to C Part Two

This is part two of a series on creating bindings to C libraries for the D programming language.

In part one, I discussed the difference between dynamic and static bindings and some of the considerations to take into account when deciding which way to go. Here in part two, I'm going to talk about an important aspect of function declarations: linkage attributes.

When binding to C, it is critical to know which calling convention is used by the C library you are binding. In my experience, the large majority of C libraries use the cdecl calling convention across each platform. Modern Windows system libraries use the stdcall calling convention (older libraries used the pascal convention). See this page on x86 calling conventions if you want to know the differences.

D provides a storage class, extern, that does two things when used with a function. It tells the compiler that the given function is not stored in the current module and it specifies a calling convention via a linkage attribute. The D documentation lists all of the supported linkage attributes, but for C bindings the three you will be working with most are C, Windows and System.

Although I'm not going to specifically talk about static bindings in this post, the following examples use function declarations as you would in a static binding. For dynamic bindings, you'll use function pointers instead.

The C attribute is used on functions that have the cdecl calling convention. If no calling convention is specified in the C headers, it's safe to assume that the default convention is cdecl. There's a minor caveat in that some compilers allow the default calling convention to be changed via the command line. This isn't an issue in practice, but it's a possibility you should be aware of if you don't have control over how the C library is compiled.


// In C
extern void someCFunction(void);

// In D
extern(C) void someCFunction();


The Windows attribute is used on functions that have the stdcall calling convention. In the C headers, this means the function is prefixed with something like __stdcall, or a variation thereof depending on the compiler. Often, this is hidden behind a define. For example, the Windows headers use WINAPI, APIENTRY, and PASCAL. Some third party libraries will use these same defines or create their own.


// In C
#define WINAPI __stdcall
extern WINAPI someWin32Function(void);

// In D
extern(Windows) someWin32Function();


The System attribute (extern(System)) is useful when binding to libraries, like OpenGL, that use the stdcall convention on Windows, but cdecl on other systems. On Windows, the compiler sees it as extern(Windows), but on other systems as extern( C ). The difference is always hidden behind a define on the C side.


// In C
#ifdef _WIN32
#include
#define MYAPI WINAPI
#else
#define MYAPI
#endif

extern MYAPI void someFunc(void);

// In D
extern(System) void someFunc();


The examples above are just examples. In practice, there are a variety of techniques used to decorate function declarations with a calling convention. It's important to examine the headers thoroughly and make no assumptions about what a particular define actually translates to.

One more useful detail to note is that when implementing function declarations on the D side, you do not need to prefix each one with an extern attribute. You can use an attribute block like so:


extern(C)
{
void functionOne();
double functionTwo();
}

// Or, if you prefer
extern(C):
void functionOne();
void functionTwo();


In part three, I'll talk a bit about builtin types and how they translate between C and D. After that, we'll be ready to look at complete function declarations and how they differ between static and dynamic bindings.

Aldacron

Aldacron

 

Binding D to C Part Three

This is the (long overdue) third part of a series on creating bindings to C libraries for the D programming language.

In part one, I introduced the difference between dynamic and static bindings and some of the things to consider when choosing which kind to implement. In part two, I talked about the different linkage attributes to be aware of when declaring external C functions in D. Here in part three, I'm going to begin discussing how to translate C type declarations into D. I'll continue the discussion in part four.

First off, I want to mention a particular page over at dlang.org called Converting C .h Files to D Modules. This is required reading for anyone planning to work on a D binding to a C library. This series should be considered a companion to that page.

Dealing With Types

When translating C types to D, the large majority take only a handful of forms:

* typedefs
* #defines
* struct declarations
* function parameters

All of the translation guidelines discussed in this post cover all four situations, but there are special cases regarding function parameters that will be covered in part five when I talk about function declarations. It's also possible to find global variable declarations in a C header. But, in my experience, they aren't generally encountered when creating library bindings.

Typedefs, Aliases, and Native Types

D used to have typedefs. And they were strict in that they actually created a new type. Given an int typdefed to a Foo, a type Foo would actually be created rather than it being just another name for int. But D also has alias, which doesn't create a new type but just makes a new name for an existing type. In D2, typedef was deprecated. Now we are left with alias.

alias is what should be used in D when a typedef is encountered in C, excluding struct declarations (more on that in part four). Most C headers have a number of typedefs that create alternative names for native types. For example, you might see something like this in a C header.


typedef int foo_t;
typedef float bar_t;


In a D binding, it's typically a very good idea to preserve the original typenames. The D interface should match the C interface as closely as possible. That way, existing C code from examples or other projects can be easily ported to D. So the first thing to consider is how to translate the native types int and long into D.

Fortunately, on the dlang page I mentioned above, there is a table that lists how all the C native types translate to D. If you look it up, you'll see that an int is an int, a float is a float, and so on. So to port the two declarations above, simply replace typedef with alias and all is well.


alias int foo_t;
alias float bar_t;


One thing I'd like to point out about that table, though. It lists the D int as equivalent to the C long. In most cases, this is true. But there is a possibility that the C long type could actually be 64-bits on some platforms, whereas D's int type is always 32-bits and D's long type is always 64-bits. As a measure of protection against this possible snafu, it's prudent to use a couple of handy aliases on the D side that are declared in core.stdc.config: c_long and c_ulong.


// In the C header
typedef long mylong_t;
typedef unsigned long myulong_t;

// In the D module
import core.stdc.config;

// Although the import above is private to the module, the aliases are public
// and visible outside of the module.
alias c_long mylong_t;
alias c_ulong myulong_t;


One more thing. If you are translating typedefs that use types from C's stdint.h, you have two options for the aliases. You can use native D types, since the sizes are fixed, or you can include core.stdc.stdint, which mirrors the C header, and just replace typedef with alias. For example, here are some types from SDL2 translated into D.


// From SDL_stdinc.h
typedef int8_t Sint8;
typedef uint8_t Uint8;
typedef int16_t Sint16;
typedef uint16_t Uint16;
...

// In D, without core.stdc.stdint
alias byte Sint8;
alias ubyte Uint8;
alias short Sint16;
alias ushort Uint16;
...

// And with the import
import core.stdc.stdint;

alias int8_t Sint8;
alias uint8_t Uint8;
alias int16_t Sint16;
alias uint16_t Uint16;
...


Enums

Translating anonymous enums from C to D requires nothing more than a copy/paste.


// In C
enum
{
ME_FOO,
ME_BAR,
ME_BAZ
};

// In D
enum
{
ME_FOO,
ME_BAR,
ME_BAZ,
}


Note that enums in D do not require a final semicolon. Also, the last member may be followed by a comma.

For named enums, you may want to do just a bit more than a direct copy/paste. Named enums in D require the name be prefixed when accessing members. Example:


// In C
typedef enum
{
ME_FOO,
ME_BAR,
ME_BAZ
} MyEnum;

// In D
enum MyEnum
{
ME_FOO,
ME_BAR,
ME_BAZ
}

// In some function...
MyEnum me = MyEnum.ME_FOO;


There's nothing wrong with this in and of itself. In fact, there is a benefit in that it gives you some type safety. For example, if a function takes a parameter of type MyEnum, you can't just pass any old int in its place. The compiler will complain that int is not implicitly convertible to MyEnum. That may be acceptable for an internal project, but for a publicly available binding it is bound to cause confusion because it breaks compatibility with existing code samples. One work around that maintains type safety is the following.


alias MyEnum.ME_FOO ME_FOO;
alias MyEnum.ME_BAR ME_BAR;
alias MyEnum.ME_BAZ ME_BAZ;

// Now this works
MyEnum me = ME_FOO;


It's obvious how tedious this could become for large enums. If type safety is not important, there's one more workaround.


alias int MyEnum;
enum
{
ME_FOO,
ME_BAR,
ME_BAZ
}


This will behave exactly as the C version.

#defines

Often in C, #define is used to declare constant values. OpenGL uses this approach to declare values that are intended to be interpreted as the type GLenum. Though these values could be translated to D using the immutable type modifier, there is a better way.

D's enum keyword is used to denote traditional enums and also manifest constants. In D, a manifest constant is an enum that has only one member, in which case you can omit the braces in the declaration. Here's an example:


// This is a manifest constant of type float
enum float Foo = 1.003f;

// We can declare the same thing using auto inference
enum Foo = 1.003f; // float
enum Bar = 1.003; // double
enum Baz = "Baz!" // string


For single #defined values in C, these manifest constants work like a charm. But often, such values are logically grouped according to function. Given that a manifest constant is essentially the same as a one-member enum, it follows that we can group several #defined C values into a single, anonymous D enum.


// On the C side.
#define FOO_SOME_NUMBER 100
#define FOO_A_RELATED_NUMBER 200
#define FOO_ANOTHER_RELATED_NUMBER 201

// On the D side
enum FOO_SOME_NUMBER = 100
enum FOO_A_RELATED_NUMBER = 200
enum FOO_ANOTHER_NUMBER = 201

// Or, alternatively
enum
{
FOO_SOME_NUMBER = 100,
FOO_A_RELATED_NUMBER = 200,
FOO_ANOTHER_NUMBER = 201,
}


Personally, I tend to use the latter approach if there are more than two or three related #defines, and the former if it's only one or two values.

But let's get back to the manifest constants I used in the example up above. I had a float, a double and a string. What if there are multiple #defined strings? Do you have to declare a seperate manifest constant for each one? No, not if you don't want to. D's enums can be typed to any existing type. Even structs.


// In C
#define LIBNAME "Some Awesome C Library"
#define AUTHOR "John Foo"
#define COMPANY "FooBar Studios"

// In D, collect all the values into one enum declaration of type string
enum : string
{
LIBNAME = "Some Awesome C Library",
AUTHOR = "John Foo",
COMPANY = "FooBar Studios",
}


Neat, eh? Again, note the trailing comma on the last enum field. I tend to always include these in case a later version of the C library adds a new value that I need to tack on at the end. A minor convenience.

More to Come

I think that's about enough for this session. In part four, we'll take a look at structs. There are a couple of pitfalls to be aware of when porting them over to D and I'll give you some advice to get around them.

Aldacron

Aldacron

 

Binding D to C Part Four

This is the fourth part of a series on creating bindings to C libraries for the D Programming Language.

In part one, I introduced the difference between dynamic and static bindings and some of the things to consider when choosing which kind to implement. In part two, I talked about the different linkage attributes to be aware of when declaring external C functions in D. In part three, I showed how to translate C types to D. Here in part four, I'll wrap up the dscussion of type translation with a look at structs.

A D Struct is a C Struct
For the large majority of cases, a C struct can be directly translated to D with little or no modification. The only major difference in the declarations is when C's typedef keyword is involved. The following example shows two cases, with and without typedef. Notice that there is no trailing semi-colon at the end of the D structs.


// In C
struct foo_s
{
int x, y;
};

typedef struct
{
float x;
float y;
} bar_t;

// In D
struct foo_s
{
int x, y;
}

struct bar_t
{
float x;
float y;
}


Most cases of struct declarations are covered by those two examples. Sometimes, a slight deviation may be encountered. Such as a struct with two names, one in the struct namespace and one outside of it (the typedef). In that case, the typedefed name should always be used.


// In C
typedef struct foo_s
{
int x;
struct foo_s *next;
} foo_t;

// In D
struct foo_t
{
int x;
foo_t *next;
}


Another common case is that of what is often called an opaque struct (in C++, more commonly referred to as a forward reference). The translation from C to D is similar to that above.


// In C
typedef struct foo_s foo_t;

// In D
struct foo_t;


Member Gotchas
When translating the types of struct members, the same rules as outlined in Part 3 should be followed. But there are a few gotchas to be aware of.

The first gotcha is relatively minor, but annoying. I've previously mentioned in this series that I believe it's best to follow the C library interface as closely as possible when naming types and functions in a binding. This makes translating code using the library much simpler. Unfortunately, there are cases where a struct might have a field which happens to use a D keyword for its name. The solution, of course, is to rename it. I've encountered this a few times with Derelict. My solution is to prepend an underscore to the field name. For publicly available bindings, this should be prominantly documented.


// In C
typedef struct
{
// oops! module is a D keyword.
int module;
} foo_t;

// In D
struct foo_t
{
int _module;
}


The next struct gotcha is that of versioned struct members. Though rare in my experience, some C libraries wrap the members of some structs in #define blocks. I find this practice rather annoying (libpng, I'm looking at you), because it can cause problems not only with language bindings but also with binary compatibility issues when using C as well. Thankfully, translating this idiom to D is simple. Using it, on the other hand, can get a bit hairy.

Here's an example.


// In C
typedef struct
{
float x;
float y;
#ifdef MYLIB_GO_3D
float z;
#endif
} foo_t;

// In D
struct foo_t
{
float x;
float y;
// Using any version identifier you want -- this is one case where I advocate breaking
// from the C library. I prefer to use an identifier that makes sense in the context of the binding.
version(Go3D) float z;
}


Then, to make use of the versioned member, the '-version=Go3D' is passed on the command line when compiling. And this is where the headache begins.

If the binding is compiled as a library, then any D application linking to that library will also need to be compiled with any version identifiers the library was compiled with, else the versioned members won't be visible. Furthermore, the C library needs to be compiled with the equivalent defines. So to use foo_t.z from the example above, the C library must be compiled with -DMYLIB_GO_3D, the D binding with -version=Go3D, and the D app with -version=Go3D. And when making a binding like Derelict that loads shared libraries dynamically, there's no way to ensure that end users will have a properly compiled copy of the C shared library on their system unless it is shipped with the app. Not a big deal on Windows, but rather uncommon on Linux. Also, if the binding is intended for public consumption, the versioned sections need to be documented.

Read more about D's version conditions in the D Programming Language documentation.

The final struct member gotcha, and a potentially serious one, is bitfields. The first issue here is that D does not have bitfields. In D2, we have a library solution in std.bitmanip, but for a C binding it's not a silver-bullet solution because of the second issue. And the second issue is that the C standard leaves the ordering of bitfields undefined.

Consider the following example from C.


typedef struct
{
int x : 2;
int y : 4;
int z: 8;
} foo_t;


There are no guarantees here about the ordering of the fields or where or even if the compiler inserts padding. It can vary from compiler to compiler and platform to platform. This means that any potential solution in D needs to be handcrafted to be compatibile with a specific C compiler version in order to guarantee that it works as expected.

Using std.bitmanip.bitfields might be the first approach considered.


// D translation using std.bitmanip.bitfields
struct foo_t
{
mixin(bitfields!(
int, "x", 2,
int, "y", 4,
int, "z", 8,
int, "", 2)); // padding
}


Bitfields implemented this way must total to a multiple of 8 bits. In the example above, the last field, with an empty name, is 2 bits of padding. The fields will be allocated starting from the least significant bit. As long as the C compiler compiles the C version of foo_t starting from the least significant bit and with no padding in between the fields, then this approach might possibly work. I've never tested it.

The only other alternative that I'm aware of is to use a single member, then implement properties that use bit shift operations to pull out the appropriate value.


struct foo_t
{
int flags;
int x() @property { ... }
int y() @property { ... }
int z() @property { ... }
}


The question is, what to put in place of the ... in each property? That depends upon whether the C compiler started from the least-significant or most-significant bit and whether or not there is any padding in between the fields. In otherwords, the same difficulty faced with the std.bitmanip.bitfields approach.

In Derelict, I've only encountered bitfields in a C library one time, in SDL 1.2. My solution was to take a pass. I use a single 'flags' field, but provide no properties to access it. Given that Derelict is intended to be used on multiple platforms with C libraries compiled by multiple compilers, no single solution was going to work in all cases. I decided to leave it up to the user. Anyone needing to access those flags could figure out how to do it themselves. I think that's the best policy for any binding that isn't going to be proprietary. Proprietary bindings, on the other hand, can be targeted at specific C compilers on specific platforms.

Conclusion
I believe that's all I wanted to say about structs. In Part 5, which I'm quite certain will be the final installment, I'll talk about how to declare functions for both dynamic and static bindings and some of the issues that need to be considered when doing so. I'll also tie off any loose ends I think of.

Aldacron

Aldacron

 

Extended Package Protection in D

Since the D1 days, D has had a package protection attribute akin to Java's (though unlike Java, it is not the default). Applying package protection to any symbol in a module makes it accessible only to modules in the same package.[code=nocode:0]module mylib.mypackage.mymodule;package struct MyStruct {...}class MyClass { package void doSomething() {...}}
Given the above, MyStruct is only usable and the member function doSomething in MyClass only callable directly inside the mylib.mypackage package. However, neither is accessible in any subpackages of mypackage (such as in a module mylib.mypackage.subpack.somemodule).
There was some demand for expanding the scope of package protection, so it was eventually added to the language. The above code still works as it always had, restricting access exclusively to the package in which a symbol is declared. Now, it's possible to do something like this:[code=nocode:0]module mylib.mypackage.mymodule;package(mylib) MyStruct { ... }class MyClass { package(mylib.mypackage) void doSomething() {...}}
In this snippet, MyStruct is accessible in the mylib package and all of its subpackages. The doSomething member function of MyClass is callable in mylib.mypackage and all of its subpackages. In effect, specifying a package name with the attribute tells the compiler the topmost package in which a symbol should be accessible, making it accessible also to all subpackages of that package.
This isn't an earth shattering feature, but it allows much more freedom for code organization. Consider a renderer package for a 2D or 3D game engine. One possible approach to supporting multiple renderers is to have a base engine.gfx package which contains the interfaces and an engine.gfx.impl package for all of the implementations (e.g. engine.gfx.impl.ogl). The extended package protection comes in handy here to make common internal declarations visible to the implementations while hiding them from the outside world:[code=nocode:0]module engine.gfx.common;package(engine.gfx):enum internalConstant = 123;struct InternalStruct { ... }void internalFunction() { ... }
Previously, you might put this module in the engine.gfx.impl package, make everything public, and tell users not to import anything in the impl package as a form of voluntary protection. Now, you don't even need a 'common' module. You can put each declaration where it makes sense to put it and the compiler will enforce your protection scheme for you.

Aldacron

Aldacron

 

Binding D to C Part Five

This is the fifth and final part of a series on creating bindings to C libraries for the D Programming Language.

In part one, I introduced the difference between dynamic and static bindings and some of the things to consider when choosing which kind to implement. In part two, I talked about the different linkage attributes to be aware of when declaring external C functions in D. In part three, I showed how to translate C types to D. In part four, I wrapped up the discussion of type translation with a look at structs. Here in part five, I'm going to revisit the static vs. dynamic binding issue, this time looking at implementation differences.

Terminology

Back in part one, I gave this definition of static vs. dynamic bindings.[quote]By static, I mean a binding that allows you to link with C libraries or object files directly. By dynamic, I mean a binding that does not allow linking, but instead loads a shared library (DLL/so/dylib/framework) at runtime.[/quote]
Those two unfortunate sentences did not clearly express my meaning. Only one person demonstrated a misunderstanding of the above in a comment on the post, but in the intervening 12 months I've found a few more people using the two terms in the wrong way. Before explaining more clearly what it is I actually mean, let me make emphatically clear what I don't.

When I talk of a static binding, I am not referring to static linking. While the two terms are loosely related, they are not the same at all. A static binding can certainly be used to link with static libraries, but, and this is what I failed to express clearly in part one, they can also be linked with dynamic libraries at compile time. In the C or C++ world, it is quite common when using shared libraries to link them at compile time. On Windows, this is done via an import library. Given an application "Foo" that makes use of the DLL "Bar", when "Foo" is compiled it will be linked with an import library named Bar.lib. This will cause the DLL to be loaded automatically by the operating system when the application is executed. The same thing can be accomplished on Posix systems by linking directly with the shared object file (extending the example, that would be libBar.so in this case). So with a static binding in D, a program can be linked at compile time with the static library Bar.lib (Windows) or libBar.a (Posix) for static linkage, or the import library Bar.lib (Windows) or libBar.so (Posix) for dynamic linkage.

A dynamic binding can not be linked to anything at compile time. No static libraries, no import libraries, no shared objects. It is designed explicitly for loading a shared library manually at run time. In the C and C++ world, this technique is often used to implement plugin systems, or to implement hot swapping of different application subsystems (for example, switching between an OpenGL and D3D renderer) among other things. The approach used here is to declare exported shared library symbols as pointers, call into the OS API for loading shared libraries, then manually extract the exported symbols and assign them to the pointers. This is exactly what a dynamic binding does. It sacrifices the convenience of letting the OS load the shared library for more control over when and what is loaded.

So to reiterate, a static binding can be used with either static libraries or shared libraries that are linked at compile time. Both cases are subject to the issue with object file formats that I outlined in part one (though the very-soon-to-be-released DMD 2.061 alleviates this a good deal, as the 64-bit version knows how to work with the Visual Studio linker). A dynamic binding cannot be linked to the bound library at compile time, but must provide a mechanism to manually load the library at run time.

Now that I've hopefully gotten that point across, it's time to examine the only difference in implementing the two types of bindings. In part two, I foreshadowed this discussion with the following.[quote]Although I'm not going to specifically talk about static bindings in this post, the following examples use function declarations as you would in a static binding. For dynamic bindings, you'll use function pointers instead.[/quote]
Static Bindings

In D, we generally do not have to declare a function before using it. The implementation is the declaration. And it doesn't matter if it's declared before or after the point at which its called. As long as it is in the currently visible namespace, it's callable. However, when linking with a C library, we don't have access to any function implementations (nor, actually, to the declarations--hence the binding). They are external to the application. In order to call into that library, the D compiler needs to be made aware of the existence of the functions that need to be called so that, at link time, it can match up the proper address offsets to make the call. This is the only case I can think of in D where a function declaration isn't just useful, but required.

I explained linkage attributes in part two. The examples I gave there, coupled with the details in part three regarding type translation, are all you need to know to implement a function declaration for a static D binding to a C library. But I'll give an example anyway.
// In C, foo.hextern int foo(float f);extern void bar(void);// In Dextern( C ){ int foo(float); void bar();}
Please be sure to read parts 2 - 4 completely. With all of that, and the info in this post up to this point, you have almost everything you need to know to implement a static binding in D, barring any corner cases that I've failed to consider or am yet to encounter myself. Oh, and function pointers. But that's coming in the next section.

Interlude -- Function Pointers

I could have covered function pointers in part three. After all, it isn't uncommon to encounter C libraries that use function pointers as typedefed callbacks (or declared inline in a function parameter list), or as struct members. And there's no difference in declaring function pointers for callbacks or for loading function symbols from a shared library. The syntax is identical. But, there are some issues that need to be considered in the different situations. And it's important to understand this before we get into dynamic bindings.

Let's look first at the syntax for declaring a function pointer in D.

int function() MyFuncPtr;
Very simple: return type->function keyword->parameter list->function pointer name. Though it's possible to use MyFuncPtr directly, it's often convenient to declare an alias:


alias int function() da_MyFuncPtr;da_MyFuncPtr MyFuncPtr;
What's the difference? Let's see.


int foo(int i){ return i;}void main(){ int function(int) fooPtr; fooPtr = &foo; alias int function(int) da_fooPtr; da_fooPtr fooPtr2 = &foo; import std.stdio; writeln(fooPtr(1)); writeln(fooPtr2(2));}
There is none! At least, not on the surface. I'll get into that later. Let's look at another example. Translating a C callback into D.

// In C, foo.htypedef int (*MyCallback)(void); // In Dextern( C ) alias int function() MyCallback;
Notice that I used the alias form here. Anytime you declare a typedefed C function pointer in D, it should be aliased so that it can be used the same way. Finally, the case of function pointers declared inline in a paramter list.

// In C, foo.hextern void foo(int (*BarPtr)(int));// In D.// Option 1extern( C ) void foo(int function(int) BarPtr);// Option 2extern( C ) alias int function(int) BarPtr;extern( C ) void foo(BarPtr);
Personally, I prefer option 2. Also, I generally prefer to use extern blocks to include multiple declarations so that I don't have to type extern( C ) or extern(System) all the time (as I did in the previous example).

Now that the function pointer intro is out of the way, it's time to look at dynamic bindings.

Dynamic Bindings

At this point, I would very much like to say that you know everything you need to know about dynamic bindings. But that would be untrue. As it turns out, simply declaring function pointers is not enough. There are two issues to take into consideration. The first is function pointer initialization.

In one of the examples above (fooPtr), I showed how a function pointer can be declared and initialized. But in that example, it is obvious to the compiler that the function foo and the pointer fooPtr have the same basic signature (return type and parameter list). Now consider this example.

// This is all D.int foo() { return 1; }void* getPtr() { return cast(void*)&foo; }void main(){ int function() fooPtr; fooPtr = getPtr();}
Try to compile this and you'll see something like:

[quote]fptr.d(10): Error: cannot implicitly convert expression (getPtr()) of type void* to int function()[/quote]
Now, obviously this is a contrived example. But I'm mimicking what a dynamic binding has to go through. OS API calls (like GetProcAddress or dlsym) return function pointers of void* type. So this is exactly the sort of error you will encounter if you try to directly assign the return value to a function pointer declared in this manner.

The first solution that might come to mind is to go ahead and insert an explicit cast. So, let's see what that gets us.

fooPtr = cast(fooPtr)getPtr();
The error here might be obvious to an experienced coder, but certainly not to most. I'll let the compiler explain.

[quote]fptr.d(10): Error: fooPtr is used as a type[/quote]
Exactly. fooPtr is not a type, it's a variable. This is akin to declaring int i = cast(i)x; You can't do that. So the next obvious solution might be to use an aliased function pointer declaration. Then it can be used as a type. And that is, indeed, one possible solution (and, for reasons I'll explain below, the best one).

alias int function() da_fooPtr;da_fooPtr fooPtr = cast(da_fooPtr)getPtr();
And this compiles. For the record, the 'da_' prefix is something I always use with function pointer aliases. It means 'D alias'. You can do as you please.

I implied above that there was more than one possible solution. Here's the second one.

int foo() { return 1; }void* getPtr() { return cast(void*)&foo; }void bindFunc(void** func) { *func = getPtr(); }void main(){ int function() fooPtr; bindFunc(cast(void**)&fooPtr);}
Here, the address of fooPtr is being taken (giving us, essentially, a foo**) and cast to void**. Then bind func is able to dereference the pointer and assign it the void* value without a cast. When I first implemented Derelict, I used the alias approach. In Derelict 2, Tomasz Stachowiak implemented a new loader using the void** technique. That worked well. And, as a bonus, it eliminated a great many alias declarations from the codebase. Until something happened that, while a good thing for many users of D on Linux, turned out to be a big headache for me.

For several years, DMD did not provide a stack trace when exceptions were thrown. Then, some time ago, a release was made that implemented stack traces on Linux. The downside was that it was done in a way that broke Derelict 2 completely on that platform. To make a long story short, the DMD configuration files were preconfigured to export all symbols when compiling any binaries, be they shared objects or executables. Without this, the stack trace implementation wouldn't work. This caused every function pointer in Derelict to clash with every function exported by the bound libraries. In other words. the function pointer glClear in Derelict 2 suddenly started to conflict with the actual glClear function in the shared library, even though the library was loaded manually (which, given my Windows background, makes absolutely no sense to me whatsoever). So, I had to go back to the aliased function pointers. Aliased function pointers and variables declared of their type aren't exported. If you are going to make a publicly available dynamic binding, this is something you definitely need to keep in mind.

I still use the void** style to load function pointers, despite having switched back to aliases. It was less work than converting everything to a direct load. And when I implemented Derelict 3, I kept it that way. So if you look at the Derelict loaders...

// Instead of seeing thisfoo = cast(da_Foo)getSymbol("foo");// You'll see thisfoo = bindFunc(cast(void**)&foo, "foo");
I don't particularly advocate one over the other when implementing a binding with the aid of a script. But if you're doing it by hand, the latter is much more amenable to quick copy-pasting.

There's one more important issue to discuss. Given that a dynamic binding uses function pointers, the pointers are subject to D's rules for variable storage. And by default, all variables in D are stashed in Thread-Local Storage. What that means is that, by default, each thread gets its own copy of the variable. So if a binding just blindly declares function pointers, then they are loaded in one thread and called in another... boom! Thankfully, D's function pointers are default initialized to null, so all you get is an access violation and not a call into random memory somewhere. The solution here is to let D know that the function pointers need to be shared across all threads. We can do that using one of two keywords: shared or __gshared.

One of the goals of D is to make concurrency easier than it traditionally has been in C-like languages. The shared type qualifier is intended to work toward that goal. When using it, you are telling the compiler that a particular variable is intended to be used across threads. The compiler can then complain if you try to access it in a way that isn't thread-safe. But like D's immutable and const , shared is transitive. That means if you follow any references from a shared object, they must also be shared. There are a number of issues that have yet to be worked out, so it hasn't seen a lot of practical usage that I'm aware of. And that's where __gshared comes in.

When you tell the compiler that a piece of data is __gshared, you are saying, "Hey, Mr. Compiler, I want to share this data across threads, but I don't want you to pay any attention to how I use it, mmkay?" Essentially, it's no different from a normal variable in C or C++. If you want to share a __gshared variable across threads, it's your responsibility to make sure it's properly synchronized. The compiler isn't going to help you.

So when implementing a dynamic binding, a decision has to be made: thread-local (default), shared, or __gshared. My answer is __gshared. If we pretend that our function pointers are actual functions, which are accessible across threads anyway, then there isn't too much to worry about. Care still need be taken to ensure that the functions are loaded before any other threads try to access them and that no threads try to access them after the bound library is unloaded. In Derelict, I do this with static module constructors and destructors (which can still lead to some issues during program shutdown, but I'll cover that in a separate post). Here's an example.

extern( C ){ alias void function(int) da_foo; alias int function() da_bar;}__gshared{ da_foo foo; da_bar bar;}
Finally, there's the question of how to load the library. That, I'm afraid, is an exercise for the reader. In Derelict, I implemented a utility package (DerelictUtil) that abstracts the platform APIs for loading shared libraries and fetching their symbols. The abstraction is behind a set of free functions that can be used directly or via a convenient object interface. In Derelict itself, I use the latter since it makes managing loading an entire library easier. But in external projects, I often use the free-function interface for loading one or two functions at a time (such as certain Win32 functions that aren't available in the ancient libs shipped with DMD). It also supports selective loading, which is a term I use for being able to load a library if specific functions are missing (the default behavior is to throw an exception when an expected symbol fails to load).

Conclusion

Overall, there's a good deal of work involved in implementing any sort of binding in D. But I think it's obvious that dynamic bindings require quite some extra effort. This is especially true given that the automated tools I've seen so far are all geared toward generating static bindings. I've only recently begun to use custom scripts myself, but they still require a bit of manual preparation because I don't want to deal with a full-on C parser. That said, I prefer dynamic bindings myself. I like having the ability to load and unload at will and to have the opportunity to present my own error message to the user when a library is missing. Others disagree with me and prefer to use static bindings. That's perfectly fine.

At this point, static and dynamic bindings exist for several popular libraries already. Deimos is a collection of the former and Derelict 3 the latter. You'll find some bindings for the same library in both and several that are in one project but not the other. Use what you need and are comfortable with. And I hope that, if the need arises, you can use the advice I've laid out in this series of posts to help fill in the holes and develop static or dynamic bindings yourself.

Given that I'm just under 4 hours from 2013 as I write this in Seoul, Korea, I want to wish you a Happy New Year. May you start and finish multiple projects in the coming year!

Aldacron

Aldacron

 

Binding D to C

This is part one of a series on creating bindings to C libraries for the D programming language.

This is a topic that has become near and dear to my heart. Derelict is the first, and only, open source project I've ever maintained. It's not a complicated thing. There's very little actual original code outside of the Utility package (with the exception of some bookkeeping for the OpenGL binding). The majority of the project is a bunch of function and type declarations. Maintaining it has, overall, been relatively painless. And it has brought me a fair amount of experience in getting D and C to cooperate.

As the D community continues to grow, so does the amount of interest in bindings to C libraries. Recently, a project called Deimos was started over at github to collate a variety of bindings to C libraries. There are several bindings there already, and I'm sure it will grow. People creating D bindings for the first time will, usually, have no trouble. It's a straightforward process. But there are certainly some pitfalls along the way. In this post, I want to highlight some of the basic issues to be aware of. For the sake of clarity, I'm going to ignore D1 (for a straight-up "static" binding, the differences are minor, but they do exist.

The first thing to consider is what sort of binding you want, static or dynamic. By static, I mean a binding that allows you to link with C libraries or object files directly. By dynamic, I mean a binding that does not allow linking, but instead loads a shared library (DLL/so/dylib/framework) at runtime. Derelict is an example of the latter; most (if not all) of the bindings in the Deimos repository the former. There are tradeoffs to consider.

D understands the C ABI, so it can link with C object files and libraries just fine, as long as the D compiler understands the object format itself. Therein lies the rub. On Posix systems, this isn't going to be an issue. DMD (and of course, GDC and, I assume, LDC) uses the GCC toolchain on Posix systems. So getting C and D to link and play nicely together isn't much of a hassle. On Windows, though, it's a different world entirely.

On Windows, we have a variety of object file formats to contend with: COFF, OMF, ELF. DMD, which uses an ancient linker called Optlink, outputs OMF objects. GDC, which uses the MingW backend on Windows, outputs ELF objects. I haven't investigated LDC yet, but it uses whichever backend LLVM is configured to use. Meanwhile, the compiler that ships with Visual Studio outputs objects in the COFF format. What a mess!

This situation will improve in the future, but for now it is what it is. And that means when you make a C binding, you have to decide up front whether you want to deal with the mess or ignore it completely. If you want to ignore it, then a dynamic binding is the way to go. Generally, when you manually load DLLs, it doesn't matter what format they were compiled in, since the only interaction between your app and the DLL happens in memory. But if you use a static binding, the object file format determines whether or not the app will link. If the linker can't read the format, you get no executable. That means you have to either compile the C library you are binding with a compiler that outputs a format your D linker understands, use a conversion tool to convert the libraries into the proper format, or use a tool to extract a link library from a DLL. Will you ship the libraries with your binding, in multiple formats for the different compilers? Or will you push it off on the users to obtain the C libraries themselves? I've seen both approaches.

Whichever way you decide to go really doesn't matter. In my mind, the only drawback to dynamic bindings is that you can't choose to have a statically linked program. I've heard people complain about "startup overhead", but if there is any it's negligble and I've never seen it (you can try it with Derelict -- make an app using DerelictGL/SDL/SDLImage/SDLMixer/SDLNet/SDLttf and see what kind of overhead you get at startup). The only drawback to static bindings is the object file mess. But with a little initial work upfront, it can be minimzed for users so that it, too, is negligible.

Once you decide between static and dynamic, you aren't quite ready to roll up your sleeves and start implementing the binding. First, you have to decide how to create the binding. Doing it manually is a lot of work. Trust me! That's what I do for all of the bindings in Derelict. Once you develop a systematic method, it goes much more quickly. But it is still drastically more time consuming than using an automated approach. To that end, I know people have used SWIG and a tool called htod. VisualD now has an integrated C++-to-D converter which could probably do it as well. I've never used any of them (which is really incredible when I think about it, given how precious little time I usually have), so I can't comment on the pros and cons one way or another. But I do know that any automated output is going to require some massaging. There are a number of corner cases that make an automated one-for-one translation extremely difficult to get right. So regardless of your approach, if you don't want your binding to blow up on you down the road, you absolutely need to understand exactly how to translate D to C. And that's where the real fun begins.

That's going to have to wait for another post, though. It's Sunday evening here and I've got things to do. In part two, I'll talk about function declarations. I think they're easier to cover than types, which I'll save for a third post. Until then, Happy New Year!

Aldacron

Aldacron

 

D's Default Initializers

I've been away from Dolce for a couple of weeks now. Just came back to it last night and realized I don't like it. I've horribly over engineered some of the modules. So from last night I started stripping stuff out and refactoring. In the process, I realized a silly mistake in my resource management code. I'm throwing it out and rewriting it anyway, but it inspired a topic for this blog.

My resource system was, overall, designed to work with any imaginable resource. So it's template-based and has a flexible interface. But I started from the perspective of Allegro resources, which means working with struct pointers. And I imagined that other potential resources that I might want to use would be class based. That led to an implementation detail that could cause compilation to fail in certain cases.

The problem boils down to something like this. Given a member called _resource of type T, I want to clear it out when I no longer need it in certain circumstances. Since I'm dealing with struct pointers and classes, which are always references, I can just do this:


_resource = null;


That works and does what I want. Then to determine whether a resource is loaded I can just test _resource for null. Until, of course, I decide one day to do something like use a struct resource by value, rather than as a pointer. Not something too farfetched. In that case, I'd get a compiler error. While DMD's template error messages are a good deal better than what most C++ compilers give us, it's annoying to get them. And there's no reason why I shouldn't support non-nullable types.

So here's a contrived example of what happens in this situation.


module nullify;

void nullify(T)(T t)
{
t = null;
}

void main()
{
int i = 10;
nullify(i);
}


So nullify is a templated function with no constraints, meaning it can accept any type at all (I'll talk a bit about D's template syntax another day -- this is a specific case where you can declare the template without the template keyword and call it as you would a normal function). Try to compile this code and you'll get the following output:


nullify.d(8): Error: cannot implicitly convert expression (null) of type void* to int
nullify.d(14): Error: template instance nullify.nullify!(int) error instantiating


Right. To solve this problem, there are two obvious choices. One is to use template constraints to restrict the template only to pointers and classes. There would still be compiler errors of a different nature, but it would be a signal that this template is not intended to work with value types. In some cases, that might be preferable. In this particular case, a better option is to make use of default initializers.

Every type in D has a default value to which instances are automatically initialized on declaration. For example, ints are initialized to 0, floats to nan, classes and pointers to null. This value is readable as a property, .init, both on the type and on the instance. So we can modify the nullify template above like so:


void nullify(T)(T t)
{
// You could use t.init or T.init here.
t = T.init;
}


Now the code will compile. Pointers and classes will be set to null, floats and doubles to nan, characters to 0xff, and so on. What about value structs? Try this:


module nullify;

import std.stdio : writeln;
import std.string : format;

void nullify(T)(T t)
{
t = T.init;
writeln(t);
}

void main()
{
struct Foo
{
int x = 10;
int y;

// Without a toString method, the name of the type
// would be output by default in the call to writeln
// above. In this case, "Foo".
string toString()
{
return format("(%d, %d)", x, y);
}
}
Foo f = Foo(23, 38);
nullify(f);
}


The output from this is "(10, 0)". Foo.y, as an int, has the default initializer 0. I've changed the default initializer of Foo.x, however, in the definition of Foo. So all instances of Foo will have the value 10 for x on instantiation. This is not the same as assignment. For example, this will not print 10, but 0:


int i = 10;
nullify(i);


Here, we are declaring an instance and assigning a value to this instance. We are not defining a type. Big difference.

So using default initializers is a convenient way to clear out a templated class/struct member for any given type. Then, tests like a hypothetical isLoaded method become


bool isLoaded()
{
return _resource == T.init;
}


You can read more about the .init property in the D documentation at d-programming-language.org.

Aldacron

Aldacron

 

D Arrays

Every D programmer loves D's arrays. All of them. I challenge you to find one who doesn't. Particularly, it's the ability to slice and manipulate arrays without worrying about memory that really pulls people in.

In my previous post, I talked about Dolce's game screens and the stack-based screen manager. Internally, the stack that the screen manager uses is not a custom-built stack class, nor any sort of container from an existing library. It's an array. You can implement a stack quite easily with a D array.


// Declare an empty dynamic array to use as your stack.
Item[] stack;

// To push an item onto the stack, use the array append/concat operator.
stack ~= something;

// To pop, simply decrease the length of the array.
stack.length -= 1;

// What if we have a queue instead of a stack and want to take the first item?
// First, save the item at the head of the queue.
auto item = queue[0];

// Then reassign the array to a slice of itself, starting from index #1 to the last index.
// The $ is a substitute for the length property of the array. The first number in a slice
// is inclusive, the second exclusive. So the resulting slice below will contain all elements
// from queue[1] to queue[queue.length - 1].
queue = queue[1 .. $];


D's arrays are quite powerful. In addition to being arrays, they are also ranges, which is something that would take more than a simple post to explain. I don't quite fully grok them myself yet. But, when used in conjunction with the std.algorithm module, there's a great deal of power behind them.

You can read more about D's arrays in this article on slicing in D by Steven Schveighoffer. Also, see the documentation for the std.range module to get an idea of what ranges are.

But wait, there's more! The registry the screen manager uses is a built-in associative array. While there are some more sophisticated hashmap implementations out there for when you really need them, the built-in AAs will cover a lot of your use cases.


// Declare an AA that uses strings as keys.
Item[string] registry;

// Add an item.
registry["foo"] = myItem;

// Remove an item.
registry.remove("foo");

// Test if an item exists in the AA .
// The in operator returns a pointer.
auto item = key in registry;
if(item !is null)
// Do something

// Efficiently iterate all the values via a delegate
foreach(val; registry.byValue())
writeln(val);

// Or the keys
foreach(key; registry.byKey())
writeln(key);

// But don't try to remove anything while iterating the delegates as above. Instead,
// iterate over an array of keys or values using the .keys or .values properties.
foreach(key; registry.keys)
{
auto item = registry[key];
if(item.removeMe)
registry.remove(key);
}



D's arrays and AAs will get you farther than those of C++ or Java. But, when you need more, there is also a range-based container module, std.container, which will eventually hold a number of container types. Also, as an alternative, Steven Schveighoffer's DCollections library is quite awesome.

Aldacron

Aldacron

 

Functional Me

I recall very clearly the first time I ever saw a video game. It must have been in the summer of either '78 or '79, just before my 7th or 8th birthday. I walked into a local 7-11, just a short distance from my house, and was puzzled to see this big box surrounded by a bunch of older kids. Space Invaders. The first time I saw the screen, it blew my mind. I totally forgot the reason I had come to the store in the first place and ran home to beg my mother for money to play. In Christmas of that year, I awoke to find an Atari 2600 under the Christmas tree. That cemented it. I knew I wanted to make games when I grew up.

As it would turn out, my parents would never buy a computer. I did manage to get a little exposure to some programming manuals and a chance to try some things out now and again. Eventually, I gave it up and moved on to baseball. That was something my parents could afford. The programming bug was still there, just stashed away. I pulled it out again when I finally got my first computer at the age of 26. I slogged my way through books, online tutorials (including resources from a handful of sites that would eventually merge to form GameDev.net), and anything I could get my hands on. As a result, I never had any formal education or training in computer science. For years, I thought it didn't matter. But lately, it's been bugging me. And I blame D.

I got into the D community pretty much near the ground floor. There have always been some lively discussions in the newsgroups about which features to add or change. Over the years, especially after D2 came along, I've realized just how many gaps there are in my knowledge base. There are a number of conversations I've tried to follow, but in which I became completely lost. And forget about contributing! Then, there's the functional bits that have made their way into the standard library, particularly with regards to the range interfaces. That stuff is just completely alien to me. Recently, I decided to rectify that.

In the past few weeks, I've signed up for three free online CS courses. Two at edX (one starts next month, the other in March), and one at Coursera (which started last week). The latter is a Programming Languages course in which, for starters, we're learning functional programming with SML. Something I never thought I'd do, but, thanks to D, now have the motivation for. I'm actually quite enjoying it.

Once these courses are finished, I plan to look for more that can be useful to me. Hopefully, I'll be a better programmer as a result.

Aldacron

Aldacron

 

D Tip: Use Scope Statements in Proper Scope

One D construct I often use is the scope guard statement. This allows you to write code that executes when a scope exits under one of three circumstances: an exception is thrown, no exception is thrown, or always. Example:[code=nocode:0]void main() { import std.stdio; scope( failure ) { writeln( "I only execute when the scope exits due to an exception being thrown."); } scope( success ) { writeln( "I only execute when the scope exits normally." ); } scope( exit ) { writeln( "I always execute when the scope exits, exceptions or no." ); }}
In this example, as written, two lines will be printed to the console when the function exits, the ones for the exit and success conditions, since no exception is thrown. The exit condition will be written first, since anything scope statements are executed in the reverse order they are declared. If you add a return statement before the success condition or throw an exception before the failure condition, nothing will be printed -- the scope blocks won't be executed because the code never reached the point where they are declared. Also, you can have more than one statement in each scope block. Here's another example.[code=nocode:0]void main() { import std.stdio; scope( exit ) { writeln( "I'm always going to execute."); writeln( "Because I'm the first scope guard in this scope." ); writeln( "And the scope doesn't exit before I am declared."); writeln( "But I will be the last scope guard to run." ); } int x = 5; int y = 6; scope( success ) writefln( "I won't execute because of the Error thrown below. BTW, x+y = %s", x + y ); scope( exit ) { writeln( "Yes, multiple scope guards of the same type can be declared." ); writeln( "And this scope( exit ) will execute before the one above." ); } if( y - x != 0 ) { throw new Error( "No scopes declared after me will execute." ); } scope( failure ) writeln( "I won't run, since an exception was thrown before this point." ); }
You can run this code online, where you will see this ouput:[quote]
Yes, multiple scope guards of the same type can be declared.
And this scope( exit ) will execute before the one above.
I'm always going to execute.
Because I'm the first scope guard in this scope.
And the scope doesn't exit before I am declared.
But I will be the last scope guard to run.
object.Error: No scopes declared after me will execute.[/quote]

D also has the try...catch...finally construct, which is what scope guard statements are lowered to by the compiler. While this is extremely useful and eliminates the need for try..catch..finally in many cases, it is not a complete replacement. For one thing, if there is an exception thrown, scope( failure ) does not give you access to the exception object. In practice, I've found that I use the exit condition the most, with success a distant second. I find that I rarely need the failure condition, but it's handy when I do.

This feature may not seem like much on the surface, but as soon as you start to use it you'll fall in love with it. I actually feel disappointed when I find I need to replace a scope guard with a manual try...catch block, as it clutters things up. But what you can't do is add them blindly to your code. The above examples only use the scope guards at function level, but you can put them in any scope. Conditional blocks, loops, anonymous scopes, wherever you want. The statements in the scope guard will execute if they meet the conditions when the scope in which they are declared exits. Like this:void main() { import std.stdio; if( 1 > 0 ) { scope( exit ) writeln( "I'll execute as soon as the if exits."); } for( size_t i=0; i
Output:[quote]
I'll execute as soon as the if exits.
How many times will I execute?
How many times will I execute?
How many times will I execute?
Will the above scopes be run before I am? Yes, they will.[/quote]

The scope guards in the if block and the for loop are declared before the last scope, but in this case all three are in separate scopes, so they aren't executed in reverse order. The first one is executed as soon as the if block exits and the second one is executed each time the for loop scope exits (three times in all). Those scopes exit before the final scope guard is every declared, so it executes last, only when the main function exits.

That brings me to the point of this post. You should always be aware of the scope in which your scope guard is declared, what is visible in that scope, and what happens in any scopes that follow it. I had never had any issues with scope guards until today. Can you spot the error?Config loadConfig( string fileName ) { auto path = Paths.findSettingsFilePath( fileName ); scope( exit ) path.destroy(); if( path !is null ) { auto config = al_load_config_file( path.toCString() ); return new Config( config, fileName ); } else { return createConfig( fileName ); }}
I'm so used to putting a scope guard right after allocation of a resource that, in this case, I didn't pay attention to how that particular resource is used in the function. In this particular case, it is possible for the path instance to be null. Because I declared the scope guard at function scope, it will always execute, regardless of whether the path instance is null or not. If the path object is null, the result is an access violation.

One way to fix this is to add a null check inside the scope guard:scope( exit ) if( path !is null ) path.destroy();
But that's rather silly, since there's already an if block that checks for null. Furthermore, since the existing if block also creates a new scope, and only in the case where the path object is non-null, then the better solution is to move the scope guard to the if block like so:auto path = Paths.findSettingsFilePath( fileName );if( path !is null ) { scope( exit ) path.destroy(); ...}
D's scope guards are quite useful, but always be aware of how you're using them. Like anything else in programming, if you aren't paying attention, you can create bugs for yourself that may not manifest at all during development and could be hard to track down if they show up on a user machine. As I used to hear all the time in the Army, "More attention to detail, soldier!"

Aldacron

Aldacron

 

Dolce Progress, Importing D Modules in a Function, and WYSIWYG Strings

I've made a bit of progress on Dolce, but I realized something while I was doing it. My purpose for starting the project was to work on a game idea I've had for a long, long time. I knew from the get go that graphics were going to be a problem The problem is the open-endedness and complexity of the game experience. To pull it off, I either need very detailed graphics, or simplistic graphics with a good deal of descriptive text.

I'm no artist and I can't be bothered to pay anyone for the level of detail I would need for the first option. This iteration is just for fun, not profit. So I have to go for the second option -- minimal graphics plus text. My intention was to go very, very, simplistic. Not quite Dwarf Fortress simplistic, but very nearly. I had it in my head to use a simple tile set, with symbols of some sort for game entities.

My problem, as always, is time. Between work, family, and other projects that have higher priority, I'm wasting too much time on the graphical side of this game idea. I want to get busy with the game itself. Since I'll be using a good deal of text anyway, why not just ditch Allegro and go for a pure text-based game? So that's exactly what I've decided to do. Dolce, however, is a neat little idea that I hope to come back to now and again when I have a lull. And at some point I will very much want to make a graphical version of this game if it turns out to be as fun as I hope. But for now, text it is.

One of my first problems to consider is colored text in the console. I want the game to run on Windows, Mac and Linux, so I want something that's portable and simple. That means ANSI escape codes. But the Windows command console doesn't support ANSI escape codes right out of the box. After a bit of digging around, I found a solution.

ansicon is a little console for Windows that can understand ANSI escape codes. And it's freely distributable so I can bundle it with my app and use it in place of the standard command prompt. Of course, I wouldn't want the user to have to open it and type a command to run my app. Luckily, any arguments passed to ansicon that it doesn't recognize are considered to be a program to execute and its arguments. So a one line batch file would do the trick:


ansicon MyApp


Another option would be to make a simple executable to launch ansicon. In D, it might look something like this:


int main(string[] args)
{
import std.process;
return system("bin\\ansicon.exe MyApp");
}


Notice how I've got the import statement inside the main function. That's a new feature that was added in the latest DMD release. It doesn't seem like a big deal, but I can't count how many times I've knocked up a quick D script to test out some idea only to realize that I forgot to import std.stdio at the top. If I'm only using a symbol in one function, then instead of scrolling back up and adding the import at the top, I can just add it in place and drive on.

There are other functions in std.process that can be used to execute processes in a cross-platform manner. Of course, before someone points it out, I do realize that the path I passed is not cross-platform because of the backslash. Calls to system on Windows will fail if the path contains a forward slash, so you'd have to do the right thing based on platform. But for this launcher, I don't need to be cross platform.

For those who don't like escaping special characters, there's one more D feature that can help: WYSIWYG strings.


// Using r""
string s = r"bin\ansicon MyApp";

// Using backticks (not single quotes)
s = `bin\ansicon MyApp`;


D also supports heredoc strings, which also do not require escapes.

So now I'm on the road to making a complex text-based game to prototype my idea and you know a little more about D's strings.

Aldacron

Aldacron

 

On My Job, and Some Big D News

Dolce is still very much alive, just getting little attention while I wrap up the second semester at the university. For anyone interested, I'm teaching in a special program where a local university here in Seoul is partnered with a university in America. The students do two semesters here in Korea before moving off to the States for three years. We started the second semester two months early this year in order to give the students more time between the end of the semester and their departure to America next January. Last year's students, the first to participate in the program, only had two weeks.

In addition to my Debate classes, I'm teaching a special English Fluency course this semester for the handful of students who have achieved their target TOEFL scores. Coupled with all of the private classes I teach six days a week, I don't have enough time to do everything I want to do. When my schedule gets full like this (which happens a couple of times a year), my D projects tend to suffer for it. I find it mentally tough to get any coding done if I can't work for long stretches at a time. I feel so unmotivated when it comes to short bursts at the keyboard.

Still, I've taken the time to do some nipping around the Dolce source here and there. Nothing major to report on that front yet. The semester ends for me in two more weeks, and I'm not giving my students a final exam this time around. So I'll find some nice gaps in my schedule soon that I'll use to give some attention to both Dolce and Derelict.

In the meantime, I'll leave you with this: a D compiler is now very close to being included in the GCC project. You might also be interested in checking out the related Reddit thread. That's great news all around. And I really need to get around to adding it to my D news blog.

Aldacron

Aldacron

 

Dolce and Overriding Methods in D

I'm making very slow progress on Dolce, but progress nonetheless. I managed to grab a couple of hours today to refactor event handling and to simplify the framework interface.

In my initial implementation, it was necessary when making a game with Dolce to make all of the initialization and termination calls yourself. All of these calls are found in the different modules of the core package. They wrap a lot of Derelict and Allegro setup/cleanup in a way that is configurable, providing sensible defaults where required. But, while it's a good deal less boilerplate than you would otherwise need to type, it could be improved.

So, now, to start implementing a game with Dolce, the minimal amount of code you need is this:


module mygame.main;

import dolce.game.game;

class MyGame : Game
{
protected override
{
string name()
{
return "MyGame";
}

void frame()
{

}
}
}

int main(string[] args)
{
Game.start(new MyGame);
return 0;
}


Add your per-frame code in the frame method, and away you go. Both name and frame are abstract methods in the Game class, so they are the only two methods that must be overridden. I may yet rework it to eliminate the frame method and have a pair of render/update methods instead (haven't decided where or how to handle timing yet). But the point is, there's very little to do just to display something on screen. Of course, there are several other methods that can be overridden for game-specific init/cleanup, event handling, and more. But as is, this will call the frame method as fast as possible and has default handling for the escape key and window close button. And now this is starting to look like what I had envisioned in the beginning.

It's also possible to avoid the Game.start method and manage the life cycle yourself:


int main(string[] args)
{
auto game = new MyGame;
scope(exit) game.term();
game.init();
game.run();
return 0;
}


Or, you can ignore the game module entirely and use the core package modules directly. Your choice.

The real point of this post, though, was a chance to talk about how D handles method overriding. Notice the override keyword in the first code block above. This is not mandatory when overriding, but it's a good idea to use it. DMD will spit out a warning if you don't. What it's good for is to protect against the case when the superclass changes. Imagine that you override a method from the Game class, but in the future that method is removed or renamed. By using override, the compiler will let you know the next time you compile that your method isn't actually overriding anything. Otherwise, it would be creating a new method in your derived class that doesn't override anything and you might never be the wiser until you get a funky bug in your app. So when writing D code, using override is a good habit to pick up.

Aldacron

Aldacron

 

Got the Bug Again

The last time I made a complete game was too long ago to think about. Over the years, I've lurked at GDNet semi-religiously. I've false-started more projects than I can count. Games, game libraries, game engines, all using several languages. Java, C, C++, D. What have I got to show for it? A hard drive full of archived source code. Well, not full, but you get the idea.

Despite the fact that I haven't completed a game in a long, long time, I have learned a lot over the years. I also have one personal project that hasn't faded away yet. For seven years (seven!?!), I've been maintaining Derelict, a collection of bindings to C libraries for the D Programming Language. Game-centric of course. Derelict 2, a branch I've been working on off-and-on for a while, will soon be moved to the trunk. Derelict 2 includes bindings for more libraries than the original. One of those, recently added, is Allegro 5.

Back in the late 90s, when I was first learning C at the ripe old age of 27, Allegro was the first game development library I ever used. I hadn't looked at it in years, until I noticed someone mention Allegro 5 recently on a forum somewhere (maybe here). I looked into it, liked what they've done with it, and immediately added a binding for it to Derelict. But that wasn't all. Working with Allegro again has reawakened the gamedev bug in me.

My latest endeavor is Dolce, a simple framework that sits on top of Allegro. The goal is to handle all of the boilerplate and provide some utilities that can reduce the amount of code needed to write games with Allegro. It's Allegro-specific, with no plans or desire to make it abstract enough to use with other libraries (like SDL or SFML, which Derelict also provides bindings for). I'm writing it for myself, to help me get some games off the ground, but I'll be putting it up in a repository somewhere once I've got it polished and documented. Most likely DSource or github. Dolce is being written with version 2 of the D Programming Language.

I've been deeply involved with D off and on since before I started Derelict. It has come a long way since the early days and has turned into a language that makes programming fun for me again. If you're interested in following D news but don't want to bother with the newsgroups or mailing lists, subscribe to my other D blog, The One With D. Here at D Bits I'll be posting my game-related D adventures. That includes writing about Dolce, Derelict, and my future unfinished game projects.

Working on Dolce has already brought me one instance of good luck. Being a big fan of KOTOR and Dragon Age: Origins, six or seven months ago I bought Mass Effect on Steam. To my great disappointment, it was totally unplayable on my system thanks to slideshow graphics. After a significant effort in troubleshooting with no results, I gave up and uninstalled. Fast forward to a few days ago. I got the Dolce test app up and running for the first time in VisualD, the D plugin for Visual Studio. I noticed a good deal of D3D debug spew in the console. Then I remembered that I had turned on the debug version of D3D some time last year for another project I was working on. A couple of days later, while reading a blog post about Mass Effect 2, the old light bulb popped up. I turned off the debug runtime and reinstalled Mass Effect. No more slideshow. And I can finally experience the adventures of Commander Shepard. Right when I don't need the distraction, of course.

Aldacron

Aldacron

 

To Wrap or Not To Wrap

One issue I briefly considered with Dolce is whether or not I should wrap Allegro in a nice, D-style API. I played around with it a little bit, wrapping an audio stream, first as a class and then as a struct. For those of you unfamiliar with D, yes, there is a difference.

Structs in D are, by default, value types. Classes are reference types. You generally allocate structs on the stack and classes on the heap (though the reverse is possible, but uncommon). Furthermore, structs have no vtables -- they aren't inheritable. This is a difference that C++ programmers sometimes have difficulty with. Typically, you'd use structs for POD types and classes for everything else. Of course, there are exceptions to the rule.

So I tried both types to see how they would play with my resource manager. It was immediately clear that classes make more sense, as I'd not want to be passing structs around by value everywhere and would likely allocate them on the heap anyway. In both cases, it was nice applying D's property syntax to some of the wrapped Allegro functions. 'al_set_audio_stream_playing(stream, true)' is a lot harder on the eyes, even for a C programmer like me, than 'stream.playing = true'.

But no. Let's not go there. It looks nice, but it really isn't my goal with this library. All I really want is boilerplate and utilities. Right now, you can load and initialize the Allegro shared library, along with any addons you want to use, set up default configuration, and create the display, all with one function call that takes three parameters. Or you can initialize each addon and create the display in your own time. That's what I mean by boilerplate. And by utilities, I don't mean wrappers. I mean a resource manager to help you with loading and managing resources, helper functions that reduce a multi-function-call process to one, and game tools like a simple math package, a 2D scene graph, entities and the like.

The goal is not to give Allegro an OOP interface, but to enable the client to get straight to the game code without mucking about with the other stuff first. Allegro gets you a long way toward that goal already, but because it's mostly just an OS abstraction, there's still another layer that can sit on top of it and give you a bit more utility. That's what Dolce is about.

Aldacron

Aldacron

 

Setting Thread Affinity on Windows in D

When working with D's standard library, it is sometimes necessary to work around missing declarations in the core.sys.windows.windows module. It's a fairly big module as is, but it isn't all-inclusive. If you are doing any heavy-duty Windows development, you'll likely want a third-party Win32 API binding. But if you just need to call a function or two, that's overkill.

A good example of this is the need to lock the timing thread to one core when dealing with the Windows timer on a multi-core machine. This is a well-known solution to the problem of erratic timing results returned by QueryPerformanceCounter. If you are using a library like SDL, which uses QPC under the hood for its timing calls, this is something you ought to consider. Unfortunately, Phobos is currently missing declarations for a couple of function calls and one type that you need to lock the timing thread down.

For many cases, this can be overcome by adding the appropriate declarations where you need them. In other cases, you'll find (particularly when using DMD) that the Win32 import libraries that ship with the compiler are outdated. In that case, you'll either need to generate them yourself or load the function symbols manually via LoadLibrary and friends. Luckily, for setting the thread affinity the solution is easy. Here's a complete example, ripped right out of the module where I use it.

[source]
version(Windows)
{
private
{
import core.sys.windows.windows;
import std.windows.syserror;

// Declarations missing from the windows module.
alias size_t DWORD_PTR;
extern(Windows)
{
DWORD SetThreadAffinityMask(HANDLE,DWORD);
BOOL GetProcessAffinityMask(HANDLE,DWORD_PTR*,DWORD_PTR*);
}

void setThreadAffinity()
{
void doThrow(string msg)
{
auto err = GetLastError();
throw new Exception(msg ~ sysErrorString(err));
}

DWORD_PTR procMask, sysMask;
if(!GetProcessAffinityMask(GetCurrentProcess(), &procMask, &sysMask))
doThrow("GetProcessAffinityMask failure: ");

DWORD_PTR mask = 1;
if(mask & procMask)
{
if(!SetThreadAffinityMask(GetCurrentThread(), mask))
doThrow("SetThreadAffinityMask failure: ");
}
else
{
throw new Exception("Unexpected affinity mask mismatch.");
}
}
}
}
[/source]

Drop this into module or class scope and add the following to an init method somewhere in the same module (or another module if you move setThreadAffinity out of the private block).

[source]version(Windows) setThreadAffinity();[/source]

Notice also that I imported std.windows.syserror, which exposes the sysErrorString function. I'm sure I wasn't the only one who overlooked that module. After years of using D, I only noticed it recently. If you're going to be making Win32 API calls, it will come in handy.

Aldacron

Aldacron

 

Game Screens

The concept of a game screen is a fairly common one in game engines and frameworks these days. Usually it's an interface or base class that defines a lifecycle for what once upon a time was commonly known as a game state. It eliminates the need for those ugly if-else trees, or long switch statements, that used to take up a large portion of the game loop in the good old days. Each game state, or screen, can be encapsulated in its own class and activated as needed. And often there is a manager that handles initialization, shutdown, and calling the tick method (or update/render methods in some implementations) of the screen object at regular intervals. A simple game screen manager might operate on one screen at a time. But you can get a good deal more flexibility if you let it run through a list of screens instead. Or, more conceptually appropriate, a stack of them.

So I've implemented a screen manager in Dolce that does just that -- maintains a stack of game screens, any of which may be active or suspended, and loops through the stack each frame, calling a tick method on the ones that are active. The implementation is based on one by James Boer from Game Programming Gems 5, one that I have implemented so many times I can do it blindfolded now.

First, you register a screen under a unique name of your choosing. When you are ready to activate it, you issue a push command to the screen manager. Internally, when a screen is pushed onto the stack, its init method is called. Then it is eligible to be ticked each time the screen manager's tick runs. It can push other screens on top of itself without interrupting any current tick processing, as all stack commands are delayed until the next time the screen manager ticks. When another screen is pushed onto the stack, the pause method is called on the screen at the top. When a screen is popped, its term method is called. And when a screen becomes the new top as the result of a pop, its resume method is called.

One aspect of the system that gives it flexibility is that whenever a stack command is issued, the name of a particular screen is passed to the lifecycle method being called. A screen's init method is given the name of the screen which it is being pushed on top of, its pause method the name of the screen which is pushed on top of it, its resume method the name of the screen popped off of it, and its term method the name of the screen beneath it. This allows you to do things like use the same instance in multiple places on the stack.

As an example, consider a game's main menu. You will show it once at game start up. Let's say the player selects 'PLAY NOW'. Now, the main menu screen pushes the game play screen, and the screen manager calls the pause method on the main menu, giving it the name of the game play screen as an argument. Since the main menu knows it pushed the game play screen on top of itself, it can check the name to recognize that the game play screen is active. Later, the player presses the escape key to see the main menu again. So, the game play screen issues a push command to the screen manager, giving it the registered name of the main menu. The main menu instance, the same one already on the bottom of the stack, is fetched again from the registry and its init command called with the name of the game play screen. It realizes it is already initialized and was paused. So, instead of pausing, it simply resumes. You obviously don't want to display 'PLAY NOW' when the player is already playing, so you verify that the name given to the init method matches the game play screen's name and change the text on the button, and its corresponding action, to 'RESUME GAME'. Everything else stays the same. When that button is clicked, the main menu pops itself off the stack. When the pop command is executed, its term method is called with the name of the game play screen. By checking the name, the main menu realizes it's not really time to terminate yet, so pauses instead. Game play resumes.

Anyway, it's a really basic system that gives several options for handling game states. You can instantiate as many screen managers as you want. No, it isn't a singleton. I figure there are use-cases for multiple screen managers. Consider an in-game editor. I think it'd be more appropriate to have separate managers going -- one for the game and one for the editor -- than trying to manage everything on one stack. You wouldn't always want to terminate the editor when switching into game mode. Instead, you just switch between the screen managers.

Aldacron

Aldacron

 

Dolce Refinements

In my last post, I showed the minimal amount of code needed to get something up and running with Dolce. And while it's a really small amount of code, something kept bugging me about the implementation. It just wasn't "D" enough.

Over the years, the lion's share of my programming experience has been with C and Java. When working with either language, I naturally, and effortlessly, use appropriate design patterns. In other words, I design "to the language." Maybe "design pattern" is the wrong noun here, but the point is that my code structure differs based on the capabilities and features of each language. And I can switch between them seamlessly, thanks to the years of doodling around I've got under my belt. My C code is "C", and my Java code is "Java". Unfortunately, I haven't quite gotten that line of clarity yet in D.

In D, it's possible to structure things like I do in C. It's also possible to structure things like I do in Java. That's why I've refactored Dolce so many times already. It's mentally distracting having both styles in the same code base. Unfortunately, I've found that sticking to one of the two styles is a source of annoyance, as I can't get over the feeling that if I'm writing Java or C in D, then why am I using D?

The approach demonstrated in my last post, where you subclass a Game class and override certain methods as needed, was obviously very Java-ish. And I didn't like the fact that I was requiring any methods to be implemented at all, even if there were only two of them. Finally, it seemed silly to me that if you want to avoid using the Game class, your only option was to handle all the boilerplate yourself. I knew I could do better than that.

So, I went back to the drawing board and came up with this.


import dolce.game.framework;

class MyGame
{
}

int main(string[] args)
{
Framework.start(new MyGame, "MyGame");
return 0;
}


This will create a window that closes if the escape key or close button are pressed. Pretty useless in and of itself, but the point is that there are no longer any specific methods required and you don't need to subclass anything. In fact, you don't need a class at all. Change MyGame to a struct instead, and it will work fine (and you wouldn't need to 'new' it either, though you could if you wanted to). Even better, you can pick and choose the methods you want to implement.

Framework is a static class with a templated start method that looks like this:


static void start(T)(T game, string appname, string orgname = null)
{
init(appname, orgname);

static if(hasMember!(T, "init")) game.init();

_running = true;
al_start_timer(_frameTimer);

while(_running)
{
// Pump all events.
pumpEvents();

// If it's time to update, do it.
if(_update)
{
static if(hasMember!(T, "update")) game.update();
_update = false;
}

static if(hasMember!(T, "render")) game.render();
}

static if(hasMember!(T, "term")) game.term();
term();
}


The thing I want to focus on is the hasMember template. It's implemented in the standard library module std.traits. It is not a function template. It takes a type list and no parameter list. The template itself contains one field, a bool value called hasMember. This calls for a brief detour to talk about templates in D.

Let's say I want to define a template that acts as a boolean value. Let's call it "isTrue".


template isTrue()
{
enum bool val = true;
}


Here, the template has an empty type list and no parameter list. Internally, there is no function declared, only a single member, val (so it's not a function template). Notice that val is declared as an enum. In D2, single-member, anonymous enums are known as manifest constants. They are not actual variables, so cannot be used as lvalues or have their addresses taken. Essentially, every time the template is instantiated, that call to the template is replaced by the value of the enum.

Notice also that the template and the member have different names. This means that you have to explicitly type the member name when you instantiate the template. Also remember that D's template instantiation syntax is templateName!(). So, given that, this is how you would use this template:


void main()
{
assert(isTrue!().val);
}


Sometimes you may want a template to have multiple members, but the large majority of templates you write are going to have only one. In that case, D let's you take some shortcuts if the member name is the same as the template name. In this case, we can eliminate the .val part when instantiating the template.


template isTrue()
{
enum bool isTrue = true;
}

void main()
{
assert(isTrue!());
}


std.traits.hasMember uses this approach. But it also takes two args to its type list -- one is the Type that it is to operate on, and the other is a string that will be used to look up a member method or variable of that type at compile time (this is accomplished via D's compile time reflection capabilities).

You may be wondering how a string value could be part of the type list, rather than the parameter list. The type list isn't just for types. You can also pass any sort of symbol that can be known at compile time. Let's change our isTrue example now to demonstrate this. The new job of the template is to evaluate to true if and only if a given string is equal to the string "true".


template isTrue(string s)
{
enum bool isTrue = "true" == s;
}

void main()
{
assert(isTrue!("true"));
assert(isTrue!("bugaboo"));
}


Run this and the first assert will pass. The second will blow up. The string 's' is never used at runtime here. It's a compile-time-only symbol. When the template is instantiated, the compiler will run the test ("true" == s) and set the value of isTrue to the result. The asserts are basically being rewritten as:


assert(true);
assert(false);


By using hasMember the way I do in the Framework.start method shown above, no calls will be made to any methods not present in the given type. They simply won't be compiled in to the executable. If the method is implemented, a call to it will be compiled in. This means anyone using Framework.start can implement any combination of init, term, update and render methods.

One more thing to point out. Let's look at the minimal example again.


import dolce.game.framework;

class MyGame
{
}

int main(string[] args)
{
Framework.start(new MyGame, "MyGame");
return 0;
}


According to the rules for instantiating templates, the call to Framework.start should look like this:


Framework.start!(MyGame)(new MyGame, "MyGame");


Again, the compiler is letting me take a shortcut. Because I only have one type in the typelist, I can omit the typelist and the ! when I instantiate the template. The compiler has all the information it needs to infer the type, so there's no need for me to be verbose about it.

With the new implementation, I managed to cut out several lines of code from the framework (previously 'game') module. And though I haven't implemented it yet, the door is now open to use Framework for handling the boilerplate if you want to use free functions instead of a class or struct. This is more what I would call "D style" than the first pass was.

Head on over to d-programming-language.org for more info on templates and the std.traits module.

Aldacron

Aldacron

 

Dolce and Da5teroids

So I've finally gotten started on some game code. In order to see if Dolce is actually useful or a waste of time, I decided to start by porting the A5teroids demo that ships with the Allegro 5 package. Here's what I've learned so far.

First, the whole idea of Dolce as a framework on top of Allegro to allow an absolute minimum amount of startup code for a game was a good idea on paper. All of the initialization details were tucked away behind a single method call. In practice, it's rather silly. I was happy with the little example I gave some time ago. But, the number of options that need configuring in order to make it useful actually make it more complex than it would be if I didn't hide all of the initialization details. So, now I don't. Now, you initialize the modules you need as you need them. It's cleaner, still takes just a few lines to get to the good stuff, and makes more sense.

Second, I've forgotten how fun it is to make a game. Once I dug into the A5teroids source, I decided it's not something I really want to port. It appears to have been cobbled together with little thought or consistency. So I decided to mostly start from scratch. I'm using the same resources and related data, but that's it. I've not spent a whole lot of time on it so far, but I'm having a blast. Along the way, I've been tweaking Dolce to make it more useful. I've also begun to expand it a bit and start implementing some utility modules.

Finally, as a result of moving forward on all of this I've hit on some aspects of D that would make useful posts here. I've been rather quiet here for a while, so it will be nice to have something to say again.

I anticipate that over the next three or four weeks I'll be able to get a lot more work done in D-Land than I have up until now. Not only do I need to refine Dolce and prepare it for public consumption, I also need to get busy putting the finishing touches on Derelict 2 (which I've also finally decided to move to github, thanks to a bit of encouragement). Fun times ahead.

Aldacron

Aldacron

 

Operator Overloading in D

I didn't add much of anything new to Dolce in the past week. The time I had to work on it was mostly spent refactoring. I'm patching this thing together haphazardly as I go along. I have a good idea of what I want it to do, but how to organize it all wasn't clear from the beginning. Package hierarchy, structs vs classes, module content, and so on. My little test app has served two functions: making sure it all works and understanding how it will be used. It's the latter that led to the refactoring.

So now that the plumbing is done and in working order, it's time to start looking at game-related utility code. For that, my little test app is woefully inadequate. So I'll be porting the a5teroids demo that ships with the Allegro package. This will get me started. But before I begin the port, I'm looking at two additions: entities and a math package/module. Since entities will likely need the math, I'm starting with that.

I'm not yet sure if I want a whole new package for the math stuff or just a single module, but for now I'm going with the former. The first module in the math package provides an implementation of a 2D vector. It's been a while since I've done any operator overloading with D, and I had completely forgotten how different it is between D1 and D2.

In D1, operator overloading is similar to the way it works in C++. The biggest difference is that instead of overloading by symbol, you overload by purpose. The following overloads the '+' operator.


struct Foo
{
int x;

// Foo + int
int opAdd(int i)
{
return x + i;
}

// int + Foo
int opAdd_r(int i)
{
return i + x;
}
}


The rationale behind naming overloads based on purpose, rather than the symbol, is that it encourages programmers not to change the meaning of the symbol. This way you can expect Foo + Bar to actually do some sort of addition, rather than an unrelated operation. I thought it was a pretty good idea when I first saw it. Whether or not it actually meets that goal in practice I couldn't say. Regardless, I'm not using D1, and D2 has changed this dramatically. For more on operator overloading in D1, see the D1 documentation.

In D2, most (not all) operator overloads are templated. You have a few basic templates that can be constrained based on the type of operator you want to overload. opUnary is for unary operations, such as negation. opBinary is for binary operators, such as addition. You can implement the templates once for each operator, or combine several into one instance, using template constraints in both cases. The following example demonstrates..


bool isMathOp(string op)
{
return op == "+" || op == "-" || op == "*" || op == "/";
}

struct Foo
{
int x;

// Overload the negation operator '-'
int opUnary(string op)() if(op == "-")
{
return -x;
}

// Overload addition, subtraction, multiplication and division for the case of Foo op int.
int opBinary(string op)(int i) if(isMathOp(op))
{
mixin("return x" ~ op ~ "i;");
}

// Overload the same for the case of int op Foo.
int opBinaryRight(string op)(int i) if(isMathOp(op))
{
mixin("return i" ~ op ~ "x;");
}
}


This code example uses two other nifty features of D. isMathOp is a function that qualifies for execution at compile time (the reasons for which I may talk about in a future post). Compile-time function execution (CTFE) is an awesome tool for generating code. In this case, it's not particularly needed, but it does make the template constraints more concise. Plus, I wanted to show off.

The second feature is string mixins. A string mixin basically inserts the given string into the code during compilation. In this case, it is generating strings of code like "x + i" or "x - i", for each template instance, using the append/concat operator. It's possible to call CTFE functions inside a mixin if you need to. That's a technique I use in Derelict to ensure compatibility between D1 and D2 (Dolce is D2 only, but Derelict has to support both).

So you can use CTFE and mixins, together with template constraints, to cut down the amount of code you need to implement in order to overload operators in D2. There are several more options for operator overloading, including some that aren't template-based. You can read more about in the D2 documentation.

Now, back to work on Dolce.

Aldacron

Aldacron

 

TicTacToe and Modules in D

Given that my BorderWatch project has languished on github for months without any updates beyond the first few days of random hacking, it's going to be a while before it can serve as an example of game programming in D. So I had some free time recently and decided to do something different. I put together a simple TicTacToe game, that I call T3, and put the source up on github.

T3 is not a complete game. When it starts, you can play two players, one with the mouse and one on the keypad. You can press space after each game to clear the board, and Esc to exit the game at any time. That's all it is or, in the master branch at least, will ever be. It shows some very basic features of D and I hope can be used as a starting point for people new to D to play around with. Maybe by adding a menu screen and a UI, some AI, networking... whatever.

I've never programmed a TicTacToe game before and didn't consult any source for this one, so don't be surprised if you see something weird. However, there are a couple of architectural points that I want to highlight.

One of the features of D whose usefulness can be overlooked is that of the module. In C++, it is quite common for each class to have its own source file, sometimes even multiple files. And the declaration and implementation are often separated between header and source. C++ classes that are "friends" can have access to each other's internals. It is even more common in Java to see one class per file (and without the declaration/implementation divide), but without the help of the friend keyword. In D, it's much more useful to think in terms not of files or classes, but of modules.

Modules are designed to be used as something analogous to a C++ namespace. You group common functionality into a common module, the difference being that one module equates to one file. It was quite natural for me to start thinking in terms of modules (to an extent... see below), as that's what I have always done when programming with C -- individual C source files are often referred to as modules, and a common C idiom is to group common functionality in a single file. Others may not have such an easy time with it.

So as a result of this focus on modules, you find features that might be surprising. For example, you can have protection attributes (such as public, package and private) at the module level. Modules can have their own (static) constructors. But one module feature that really trips people up is that private class or struct members are visible throughout the entire module in which they are declared.

Take, for example, the tt.game module in T3. The abstract Player class declares a private member, _mark (on line 173 as of this writing). Scroll down a bit to the Game class and you'll see that it accesses this member directly. On first blush, that appears to be a severe violation of the encapsulation principle of OOP. But if you think about it, it really isn't. In this case, the game class needs to perform gets and sets for a player's mark. I could have used D's @property attribute to provide a convenient getter/setter pair in the player class, but that would be rather pointless given that both classes are in the same module. One of the oft-cited reasons (and a good one) for encapsulation is that it can hide changes to an implementation. But here, anyone changing the implementation, maybe by adding some sort of calculation every time _mark is accessed, will always have access to the game class because they are in the same module. Make the changes, get compiler errors, search/replace in the same module to fix them, done. If the outside world needed to have access to Player's _mark, then property accessors would be the right thing (especially if it were in a library).

The converse is true, too. In C++, I rarely, if ever, had variables in a source file that were declared outside of a class, static to the module, but accessed by that class. In D, I do it all the time. An example of that is seen in tt.main, where the module-private _game instance is used by the HumanPlayer class and also by the module-level functions below it. These days, on the rare occasions when I toy around with C++, I often find myself declaring certain variables in an anonymous namespace to be shared by two or more classes in the same file.

Another consequence of thinking in modules is that free functions become less of an issue. In earlier posts on this blog, I had a dilemma over whether or not to use free functions for a library I was wanting to develop. This is the one issue I had with fully embracing the module concept. In C, this problem just doesn't manifest because you're always dealing with free functions. But mixing free functions with objects just feels... dirty. Plus there's the potential for name clashes, and no one likes the this_is_a_function syntax common in C. In C++, there is an easy solution: wrap the free functions in a namespace. There are different ways to handle this in D. The most straightforward is just to not worry about namespace conflicts. When they do arise, the fully-qualified package name can be prefixed to the function name at the call site. I mostly accept this now, but as you can see in tt.gfx and tt.audio, I still add a prefix to free functions whose names I am certain will have a conflict (gfxInit and audioInit, for example). I have no practical reason for avoiding typing tt.gfx.init() at the call site. It's just a personal quirk.

If you do decide to do something with the code, please make sure to give the README a once-over. I would like to highlight the following paragraph.

[quote]Pull requests with new game features will likely not be accepted. I would like to keep this simple and useful as a toy for new D users. I might very well create a branch with multiplayer and AI for myself to play around with, but I really want to keep the master as-is. However, I'll happily accept pull requests for bug fixes and improvements to the build script (gdc & ldc support, for example).[/quote]

This code is just a playground for anyone looking to experiment with D, so while I would love to see people do cool stuff with it, I'd rather not put any of that cool stuff into the master branch.

As an aside, I just realized that I didn't add a license to the repository. I'll take care of that pronto. For the record, I'm releasing this as public domain, with the exception of the Derelict bindings in the import directory which are released under the Boost Software License.
I'm overdue for part 5 of my Binding D to C series. I'll be sure to get that at some point over the next couple of weeks. Thanks for reading.

Aldacron

Aldacron

 

Uniform Function Call Syntax in D

Uniform Function Call Syntax (UFCS) is a feature of the D Programming Language that was finally implemented in all its glory in a recent compiler release. It has been available for use with arrays for quite some time, since the early days of D1. But now it is available for every type imaginable.

On the one hand, UFCS is nice syntactic sugar for those who hate free function interfaces (a group to which I do not belong). But it's more than that. It's also an easy way to extend the functionality of existing types, while maintaining the appearance that the new functionality actually belongs to the type.

Here's how it works. Given a free function that accepts at least one parameter, the function can be called on the first argument using dot notation as if it were a method of that type. Some code will make it clear.



import std.stdio;

void print(int i)
{
writeln(i);
}

void main()
{
int i = 10;
i.print();
8.print();
}


Notice that it works on both variables and literals (see the output over on DPaste, where you can compile and run D code on line).

For a long time, I was rather ambivalent about UFCS. I didn't see the need. After all, I have no problem with free functions. Then I found a situation where it's a perfect fit.

I'm using SDL2 in one of the many projects I've managed to overload myself with. The SDL rendering interface has several methods accepting SDL_Rect objects as parameters. While implementing a simple GUI, I wanted to maintain bounds information using a rect object. But I also need functionality SDL_Rect doesn't provide out of the box, like routines to determine the intersection of two rects, or if a rect contains a point. And despite not having a beef with free functions, it really does make a difference in the appearance of code when you have a bunch of free function calls mixed in with object method calls. So I started implementing my own Rect type, giving it an opCast method to easily pass it anywhere an SDL_Rect is expected. Then I realized how silly that is when I've got UFCS.

So I scrapped my Rect struct and reimplemented the methods as free functions taking an SDL_Rect as the first parameter. And now I can do things like this.


SDL_Rect rect = SDL_Rect(0, 0, 100, 100);
if(rect.contains(10, 10)) ...

auto irect = rect.intersect(rect2);


And so on. I also had need of a Point type, which SDL doesn't have. But it was ugly mixing 'Poin't and 'SDL_Rect', so I aliased the SDL_ bit away and it's now just 'Rect'. With the combination of aliasing and UFCS, it's possible to hide implementation details without using a full-on wrapper to do so. Of course, it's not entirely hidden as the SDL_Rect is still directly accessible and you can still use the type by name. But it certainly can come in handy.

Aldacron

Aldacron

 

My New Book: Learning D

From late February up until about two weeks ago, I've had my head down over my keyboard working on a book about the D programming language for PACKT called 'Learning D'. The electronic version is currently available from the publisher's site for roughly half-price. Both kindle and print versions are available at Amazon, though not at the sale price. If you have experience with a C family language (you don't have to be an expert) and are interested in learning D, this book will guide you through the language at a reasonable pace. Of course I cover all of the language fundamentals, but I've done so in a way that puts the most focus on where D differs from its cousins. There are a lot of similarities with C, C++, Java and C#, but sometimes the similarities can be deceptive. My goal was to try and point out many of the common issues people have when thinking in C++ or Java when programming in D. Once past the fundamentals, the book goes into D's compile-time features, including templates, followed by two chapters on ranges. The pace slows down in these chapters and goes into more detail, as D's approach here is quite different from C++ (though newer versions of C++ are gaining similar features). This is especially true for templates. The last few chapters go through the D ecosystem, using D and C together in the same program, a peek at web development with D, and a final chapter that gives pointers on where to go for more info. If you have little experience with C-family languages, you might be better served by Ali Cehreli's 'Programming in D', which is freely available online as HTML, but is also available for purchase in both electronic and print forms. PACKT has another D book coming in January called 'D Web Development' by Kai Nacke.

Aldacron

Aldacron

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!