Jump to content

  • Log In with Google      Sign In   
  • Create Account

D Bits

Extended Package Protection in D

Posted by , in D 02 March 2016 - - - - - - · 2,134 views
dang

Since the D1 days, D has had a package protection attribute akin to Java's (though unlike Java, it is not the default). Applying package protection to any symbol in a module makes it accessible only to modules in the same package.

module mylib.mypackage.mymodule;

package struct MyStruct {...}

class MyClass {
   package void doSomething() {...}
}
Given the above, MyStruct is only usable and the member function doSomething in MyClass only callable directly inside the mylib.mypackage package. However, neither is accessible in any subpackages of mypackage (such as in a module mylib.mypackage.subpack.somemodule).

 

There was some demand for expanding the scope of package protection, so it was eventually added to the language. The above code still works as it always had, restricting access exclusively to the package in which a symbol is declared. Now, it's possible to do something like this:

module mylib.mypackage.mymodule;

package(mylib) MyStruct { ... }

class MyClass {
   package(mylib.mypackage) void doSomething() {...}
}
In this snippet, MyStruct is accessible in the mylib package and all of its subpackages. The doSomething member function of MyClass is callable in mylib.mypackage and all of its subpackages. In effect, specifying a package name with the attribute tells the compiler the topmost package in which a symbol should be accessible, making it accessible also to all subpackages of that package.

 

This isn't an earth shattering feature, but it allows much more freedom for code organization. Consider a renderer package for a 2D or 3D game engine. One possible approach to supporting multiple renderers is to have a base engine.gfx package which contains the interfaces and an engine.gfx.impl package for all of the implementations (e.g. engine.gfx.impl.ogl). The extended package protection comes in handy here to make common internal declarations visible to the implementations while hiding them from the outside world:

module engine.gfx.common;

package(engine.gfx):

enum internalConstant = 123;

struct InternalStruct { ... }

void internalFunction() { ... }
Previously, you might put this module in the engine.gfx.impl package, make everything public, and tell users not to import anything in the impl package as a form of voluntary protection. Now, you don't even need a 'common' module. You can put each declaration where it makes sense to put it and the compiler will enforce your protection scheme for you.




My New Book: Learning D

Posted by , in D, Projects 29 November 2015 - - - - - - · 1,324 views

From late February up until about two weeks ago, I've had my head down over my keyboard working on a book about the D programming language for PACKT called 'Learning D'. The electronic version is currently available from the publisher's site for roughly half-price. Both kindle and print versions are available at Amazon, though not at the sale price.

 

If you have experience with a C family language (you don't have to be an expert) and are interested in learning D, this book will guide you through the language at a reasonable pace. Of course I cover all of the language fundamentals, but I've done so in a way that puts the most focus on where D differs from its cousins. There are a lot of similarities with C, C++, Java and C#, but sometimes the similarities can be deceptive. My goal was to try and point out many of the common issues people have when thinking in C++ or Java when programming in D.

 

Once past the fundamentals, the book goes into D's compile-time features, including templates, followed by two chapters on ranges. The pace slows down in these chapters and goes into more detail, as D's approach here is quite different from C++ (though newer versions of C++ are gaining similar features). This is especially true for templates.

 

The last few chapters go through the D ecosystem, using D and C together in the same program, a peek at web development with D, and a final chapter that gives pointers on where to go for more info.

 

If you have little experience with C-family languages, you might be better served by Ali Çehreli's 'Programming in D', which is freely available online as HTML, but is also available for purchase in both electronic and print forms. PACKT has another D book coming in January called 'D Web Development' by Kai Nacke.




D Tip: Use Scope Statements in Proper Scope

Posted by , in D 03 November 2013 - - - - - - · 1,995 views
d language
One D construct I often use is the scope guard statement. This allows you to write code that executes when a scope exits under one of three circumstances: an exception is thrown, no exception is thrown, or always. Example:
void main() {
    import std.stdio;

    scope( failure ) {
        writeln( "I only execute when the scope exits due to an exception being thrown.");
    }

    scope( success ) {
        writeln( "I only execute when the scope exits normally." );
    }

    scope( exit ) {
        writeln( "I always execute when the scope exits, exceptions or no." );
    }
}
In this example, as written, two lines will be printed to the console when the function exits, the ones for the exit and success conditions, since no exception is thrown. The exit condition will be written first, since anything scope statements are executed in the reverse order they are declared. If you add a return statement before the success condition or throw an exception before the failure condition, nothing will be printed -- the scope blocks won't be executed because the code never reached the point where they are declared. Also, you can have more than one statement in each scope block. Here's another example.
void main() {
    import std.stdio;

    scope( exit ) {
        writeln( "I'm always going to execute.");
        writeln( "Because I'm the first scope guard in this scope." );
        writeln( "And the scope doesn't exit before I am declared.");
        writeln( "But I will be the last scope guard to run." );
    }

    int x = 5;
    int y = 6;

    scope( success ) writefln( "I won't execute because of the Error thrown below. BTW, x+y = %s",  x + y );

    scope( exit ) {
        writeln( "Yes, multiple scope guards of the same type can be declared." );
        writeln( "And this scope( exit ) will execute before the one above." );
    }

    if( y - x != 0 ) {
        throw new Error( "No scopes declared after me will execute." );
    }

    scope( failure ) 
        writeln( "I won't run, since an exception was thrown before this point." );
    
}
You can run this code online, where you will see this ouput:

Yes, multiple scope guards of the same type can be declared.
And this scope( exit ) will execute before the one above.
I'm always going to execute.
Because I'm the first scope guard in this scope.
And the scope doesn't exit before I am declared.
But I will be the last scope guard to run.
object.Error: No scopes declared after me will execute.


D also has the try...catch...finally construct, which is what scope guard statements are lowered to by the compiler. While this is extremely useful and eliminates the need for try..catch..finally in many cases, it is not a complete replacement. For one thing, if there is an exception thrown, scope( failure ) does not give you access to the exception object. In practice, I've found that I use the exit condition the most, with success a distant second. I find that I rarely need the failure condition, but it's handy when I do.

This feature may not seem like much on the surface, but as soon as you start to use it you'll fall in love with it. I actually feel disappointed when I find I need to replace a scope guard with a manual try...catch block, as it clutters things up. But what you can't do is add them blindly to your code. The above examples only use the scope guards at function level, but you can put them in any scope. Conditional blocks, loops, anonymous scopes, wherever you want. The statements in the scope guard will execute if they meet the conditions when the scope in which they are declared exits. Like this:
void main() {
    import std.stdio;

    if( 1 > 0 ) {
        scope( exit ) writeln( "I'll execute as soon as the if exits.");
    }

    for( size_t i=0; i<3; ++i ) {
        scope( exit ) writeln( "How many times will I execute?" );
    }
    
    scope( exit ) writeln( "Will the above scopes be run before I am? Yes, they will." );    
}
Output:

I'll execute as soon as the if exits.
How many times will I execute?
How many times will I execute?
How many times will I execute?
Will the above scopes be run before I am? Yes, they will.


The scope guards in the if block and the for loop are declared before the last scope, but in this case all three are in separate scopes, so they aren't executed in reverse order. The first one is executed as soon as the if block exits and the second one is executed each time the for loop scope exits (three times in all). Those scopes exit before the final scope guard is every declared, so it executes last, only when the main function exits.

That brings me to the point of this post. You should always be aware of the scope in which your scope guard is declared, what is visible in that scope, and what happens in any scopes that follow it. I had never had any issues with scope guards until today. Can you spot the error?
Config loadConfig( string fileName ) {
    auto path = Paths.findSettingsFilePath( fileName );            
    scope( exit ) path.destroy();

    if( path !is null ) {
        auto config = al_load_config_file( path.toCString() );
        return new Config( config, fileName );
    } else {
        return createConfig( fileName );
    }
}
I'm so used to putting a scope guard right after allocation of a resource that, in this case, I didn't pay attention to how that particular resource is used in the function. In this particular case, it is possible for the path instance to be null. Because I declared the scope guard at function scope, it will always execute, regardless of whether the path instance is null or not. If the path object is null, the result is an access violation.

One way to fix this is to add a null check inside the scope guard:
scope( exit ) if( path !is null ) path.destroy();
But that's rather silly, since there's already an if block that checks for null. Furthermore, since the existing if block also creates a new scope, and only in the case where the path object is non-null, then the better solution is to move the scope guard to the if block like so:
auto path = Paths.findSettingsFilePath( fileName );
if( path !is null ) {
    scope( exit ) path.destroy();
    ...
}
D's scope guards are quite useful, but always be aware of how you're using them. Like anything else in programming, if you aren't paying attention, you can create bugs for yourself that may not manifest at all during development and could be hard to track down if they show up on a user machine. As I used to hear all the time in the Army, "More attention to detail, soldier!"


Functional Me

Posted by , in D, Me 23 January 2013 - - - - - - · 2,070 views

I recall very clearly the first time I ever saw a video game. It must have been in the summer of either '78 or '79, just before my 7th or 8th birthday. I walked into a local 7-11, just a short distance from my house, and was puzzled to see this big box surrounded by a bunch of older kids. Space Invaders. The first time I saw the screen, it blew my mind. I totally forgot the reason I had come to the store in the first place and ran home to beg my mother for money to play. In Christmas of that year, I awoke to find an Atari 2600 under the Christmas tree. That cemented it. I knew I wanted to make games when I grew up.

As it would turn out, my parents would never buy a computer. I did manage to get a little exposure to some programming manuals and a chance to try some things out now and again. Eventually, I gave it up and moved on to baseball. That was something my parents could afford. The programming bug was still there, just stashed away. I pulled it out again when I finally got my first computer at the age of 26. I slogged my way through books, online tutorials (including resources from a handful of sites that would eventually merge to form GameDev.net), and anything I could get my hands on. As a result, I never had any formal education or training in computer science. For years, I thought it didn't matter. But lately, it's been bugging me. And I blame D.

I got into the D community pretty much near the ground floor. There have always been some lively discussions in the newsgroups about which features to add or change. Over the years, especially after D2 came along, I've realized just how many gaps there are in my knowledge base. There are a number of conversations I've tried to follow, but in which I became completely lost. And forget about contributing! Then, there's the functional bits that have made their way into the standard library, particularly with regards to the range interfaces. That stuff is just completely alien to me. Recently, I decided to rectify that.

In the past few weeks, I've signed up for three free online CS courses. Two at edX (one starts next month, the other in March), and one at Coursera (which started last week). The latter is a Programming Languages course in which, for starters, we're learning functional programming with SML. Something I never thought I'd do, but, thanks to D, now have the motivation for. I'm actually quite enjoying it.

Once these courses are finished, I plan to look for more that can be useful to me. Hopefully, I'll be a better programmer as a result.


Binding D to C Part Five

Posted by , in D, Binding to C 31 December 2012 - - - - - - · 2,855 views

This is the fifth and final part of a series on creating bindings to C libraries for the D Programming Language.

In part one, I introduced the difference between dynamic and static bindings and some of the things to consider when choosing which kind to implement. In part two, I talked about the different linkage attributes to be aware of when declaring external C functions in D. In part three, I showed how to translate C types to D. In part four, I wrapped up the discussion of type translation with a look at structs. Here in part five, I'm going to revisit the static vs. dynamic binding issue, this time looking at implementation differences.

Terminology

Back in part one, I gave this definition of static vs. dynamic bindings.

By static, I mean a binding that allows you to link with C libraries or object files directly. By dynamic, I mean a binding that does not allow linking, but instead loads a shared library (DLL/so/dylib/framework) at runtime.

Those two unfortunate sentences did not clearly express my meaning. Only one person demonstrated a misunderstanding of the above in a comment on the post, but in the intervening 12 months I've found a few more people using the two terms in the wrong way. Before explaining more clearly what it is I actually mean, let me make emphatically clear what I don't.

When I talk of a static binding, I am not referring to static linking. While the two terms are loosely related, they are not the same at all. A static binding can certainly be used to link with static libraries, but, and this is what I failed to express clearly in part one, they can also be linked with dynamic libraries at compile time. In the C or C++ world, it is quite common when using shared libraries to link them at compile time. On Windows, this is done via an import library. Given an application "Foo" that makes use of the DLL "Bar", when "Foo" is compiled it will be linked with an import library named Bar.lib. This will cause the DLL to be loaded automatically by the operating system when the application is executed. The same thing can be accomplished on Posix systems by linking directly with the shared object file (extending the example, that would be libBar.so in this case). So with a static binding in D, a program can be linked at compile time with the static library Bar.lib (Windows) or libBar.a (Posix) for static linkage, or the import library Bar.lib (Windows) or libBar.so (Posix) for dynamic linkage.

A dynamic binding can not be linked to anything at compile time. No static libraries, no import libraries, no shared objects. It is designed explicitly for loading a shared library manually at run time. In the C and C++ world, this technique is often used to implement plugin systems, or to implement hot swapping of different application subsystems (for example, switching between an OpenGL and D3D renderer) among other things. The approach used here is to declare exported shared library symbols as pointers, call into the OS API for loading shared libraries, then manually extract the exported symbols and assign them to the pointers. This is exactly what a dynamic binding does. It sacrifices the convenience of letting the OS load the shared library for more control over when and what is loaded.

So to reiterate, a static binding can be used with either static libraries or shared libraries that are linked at compile time. Both cases are subject to the issue with object file formats that I outlined in part one (though the very-soon-to-be-released DMD 2.061 alleviates this a good deal, as the 64-bit version knows how to work with the Visual Studio linker). A dynamic binding cannot be linked to the bound library at compile time, but must provide a mechanism to manually load the library at run time.

Now that I've hopefully gotten that point across, it's time to examine the only difference in implementing the two types of bindings. In part two, I foreshadowed this discussion with the following.

Although I'm not going to specifically talk about static bindings in this post, the following examples use function declarations as you would in a static binding. For dynamic bindings, you'll use function pointers instead.

Static Bindings

In D, we generally do not have to declare a function before using it. The implementation is the declaration. And it doesn't matter if it's declared before or after the point at which its called. As long as it is in the currently visible namespace, it's callable. However, when linking with a C library, we don't have access to any function implementations (nor, actually, to the declarations--hence the binding). They are external to the application. In order to call into that library, the D compiler needs to be made aware of the existence of the functions that need to be called so that, at link time, it can match up the proper address offsets to make the call. This is the only case I can think of in D where a function declaration isn't just useful, but required.

I explained linkage attributes in part two. The examples I gave there, coupled with the details in part three regarding type translation, are all you need to know to implement a function declaration for a static D binding to a C library. But I'll give an example anyway.
// In C, foo.h
extern int foo(float f);
extern void bar(void);

// In D
extern( C )
{

    int foo(float);

    void bar();
}
Please be sure to read parts 2 - 4 completely. With all of that, and the info in this post up to this point, you have almost everything you need to know to implement a static binding in D, barring any corner cases that I've failed to consider or am yet to encounter myself. Oh, and function pointers. But that's coming in the next section.

Interlude -- Function Pointers

I could have covered function pointers in part three. After all, it isn't uncommon to encounter C libraries that use function pointers as typedefed callbacks (or declared inline in a function parameter list), or as struct members. And there's no difference in declaring function pointers for callbacks or for loading function symbols from a shared library. The syntax is identical. But, there are some issues that need to be considered in the different situations. And it's important to understand this before we get into dynamic bindings.

Let's look first at the syntax for declaring a function pointer in D.

int function() MyFuncPtr;
Very simple: return type->function keyword->parameter list->function pointer name. Though it's possible to use MyFuncPtr directly, it's often convenient to declare an alias:


alias int function() da_MyFuncPtr;
da_MyFuncPtr MyFuncPtr;
What's the difference? Let's see.


int foo(int i)
{
  return i;
}

void main()
{
    int function(int) fooPtr;
    fooPtr = &foo;


    alias int function(int) da_fooPtr;
    da_fooPtr fooPtr2 = &foo;


    import std.stdio;
    writeln(fooPtr(1));
    writeln(fooPtr2(2));
}
There is none! At least, not on the surface. I'll get into that later. Let's look at another example. Translating a C callback into D.

// In C, foo.h
typedef int (*MyCallback)(void);
 
// In D
extern( C ) alias int function() MyCallback;
Notice that I used the alias form here. Anytime you declare a typedefed C function pointer in D, it should be aliased so that it can be used the same way. Finally, the case of function pointers declared inline in a paramter list.

// In C, foo.h
extern void foo(int (*BarPtr)(int));

// In D.
// Option 1
extern( C ) void foo(int function(int) BarPtr);

// Option 2
extern( C ) alias int function(int) BarPtr;
extern( C ) void foo(BarPtr);
Personally, I prefer option 2. Also, I generally prefer to use extern blocks to include multiple declarations so that I don't have to type extern( C ) or extern(System) all the time (as I did in the previous example).

Now that the function pointer intro is out of the way, it's time to look at dynamic bindings.

Dynamic Bindings

At this point, I would very much like to say that you know everything you need to know about dynamic bindings. But that would be untrue. As it turns out, simply declaring function pointers is not enough. There are two issues to take into consideration. The first is function pointer initialization.

In one of the examples above (fooPtr), I showed how a function pointer can be declared and initialized. But in that example, it is obvious to the compiler that the function foo and the pointer fooPtr have the same basic signature (return type and parameter list). Now consider this example.

// This is all D.
int foo() { return 1; }

void* getPtr() { return cast(void*)&foo; }

void main()
{

    int function() fooPtr;

    fooPtr = getPtr();
}
Try to compile this and you'll see something like:

fptr.d(10): Error: cannot implicitly convert expression (getPtr()) of type void* to int function()

Now, obviously this is a contrived example. But I'm mimicking what a dynamic binding has to go through. OS API calls (like GetProcAddress or dlsym) return function pointers of void* type. So this is exactly the sort of error you will encounter if you try to directly assign the return value to a function pointer declared in this manner.

The first solution that might come to mind is to go ahead and insert an explicit cast. So, let's see what that gets us.

fooPtr = cast(fooPtr)getPtr();
The error here might be obvious to an experienced coder, but certainly not to most. I'll let the compiler explain.

fptr.d(10): Error: fooPtr is used as a type

Exactly. fooPtr is not a type, it's a variable. This is akin to declaring int i = cast(i)x; You can't do that. So the next obvious solution might be to use an aliased function pointer declaration. Then it can be used as a type. And that is, indeed, one possible solution (and, for reasons I'll explain below, the best one).

alias int function() da_fooPtr;
da_fooPtr fooPtr = cast(da_fooPtr)getPtr();
And this compiles. For the record, the 'da_' prefix is something I always use with function pointer aliases. It means 'D alias'. You can do as you please.

I implied above that there was more than one possible solution. Here's the second one.

int foo() { return 1; }void* getPtr() { return cast(void*)&foo; }void bindFunc(void** func) { *func = getPtr(); }void main()
{
    int function() fooPtr;
    bindFunc(cast(void**)&fooPtr);
}
Here, the address of fooPtr is being taken (giving us, essentially, a foo**) and cast to void**. Then bind func is able to dereference the pointer and assign it the void* value without a cast. When I first implemented Derelict, I used the alias approach. In Derelict 2, Tomasz Stachowiak implemented a new loader using the void** technique. That worked well. And, as a bonus, it eliminated a great many alias declarations from the codebase. Until something happened that, while a good thing for many users of D on Linux, turned out to be a big headache for me.

For several years, DMD did not provide a stack trace when exceptions were thrown. Then, some time ago, a release was made that implemented stack traces on Linux. The downside was that it was done in a way that broke Derelict 2 completely on that platform. To make a long story short, the DMD configuration files were preconfigured to export all symbols when compiling any binaries, be they shared objects or executables. Without this, the stack trace implementation wouldn't work. This caused every function pointer in Derelict to clash with every function exported by the bound libraries. In other words. the function pointer glClear in Derelict 2 suddenly started to conflict with the actual glClear function in the shared library, even though the library was loaded manually (which, given my Windows background, makes absolutely no sense to me whatsoever). So, I had to go back to the aliased function pointers. Aliased function pointers and variables declared of their type aren't exported. If you are going to make a publicly available dynamic binding, this is something you definitely need to keep in mind.

I still use the void** style to load function pointers, despite having switched back to aliases. It was less work than converting everything to a direct load. And when I implemented Derelict 3, I kept it that way. So if you look at the Derelict loaders...

// Instead of seeing this
foo = cast(da_Foo)getSymbol("foo");

// You'll see this
foo = bindFunc(cast(void**)&foo, "foo");
I don't particularly advocate one over the other when implementing a binding with the aid of a script. But if you're doing it by hand, the latter is much more amenable to quick copy-pasting.

There's one more important issue to discuss. Given that a dynamic binding uses function pointers, the pointers are subject to D's rules for variable storage. And by default, all variables in D are stashed in Thread-Local Storage. What that means is that, by default, each thread gets its own copy of the variable. So if a binding just blindly declares function pointers, then they are loaded in one thread and called in another... boom! Thankfully, D's function pointers are default initialized to null, so all you get is an access violation and not a call into random memory somewhere. The solution here is to let D know that the function pointers need to be shared across all threads. We can do that using one of two keywords: shared or __gshared.

One of the goals of D is to make concurrency easier than it traditionally has been in C-like languages. The shared type qualifier is intended to work toward that goal. When using it, you are telling the compiler that a particular variable is intended to be used across threads. The compiler can then complain if you try to access it in a way that isn't thread-safe. But like D's immutable and const , shared is transitive. That means if you follow any references from a shared object, they must also be shared. There are a number of issues that have yet to be worked out, so it hasn't seen a lot of practical usage that I'm aware of. And that's where __gshared comes in.

When you tell the compiler that a piece of data is __gshared, you are saying, "Hey, Mr. Compiler, I want to share this data across threads, but I don't want you to pay any attention to how I use it, mmkay?" Essentially, it's no different from a normal variable in C or C++. If you want to share a __gshared variable across threads, it's your responsibility to make sure it's properly synchronized. The compiler isn't going to help you.

So when implementing a dynamic binding, a decision has to be made: thread-local (default), shared, or __gshared. My answer is __gshared. If we pretend that our function pointers are actual functions, which are accessible across threads anyway, then there isn't too much to worry about. Care still need be taken to ensure that the functions are loaded before any other threads try to access them and that no threads try to access them after the bound library is unloaded. In Derelict, I do this with static module constructors and destructors (which can still lead to some issues during program shutdown, but I'll cover that in a separate post). Here's an example.

extern( C )
{
    alias void function(int) da_foo;
    alias int function() da_bar;
}

__gshared
{
    da_foo foo;
    da_bar bar;
}
Finally, there's the question of how to load the library. That, I'm afraid, is an exercise for the reader. In Derelict, I implemented a utility package (DerelictUtil) that abstracts the platform APIs for loading shared libraries and fetching their symbols. The abstraction is behind a set of free functions that can be used directly or via a convenient object interface. In Derelict itself, I use the latter since it makes managing loading an entire library easier. But in external projects, I often use the free-function interface for loading one or two functions at a time (such as certain Win32 functions that aren't available in the ancient libs shipped with DMD). It also supports selective loading, which is a term I use for being able to load a library if specific functions are missing (the default behavior is to throw an exception when an expected symbol fails to load).

Conclusion

Overall, there's a good deal of work involved in implementing any sort of binding in D. But I think it's obvious that dynamic bindings require quite some extra effort. This is especially true given that the automated tools I've seen so far are all geared toward generating static bindings. I've only recently begun to use custom scripts myself, but they still require a bit of manual preparation because I don't want to deal with a full-on C parser. That said, I prefer dynamic bindings myself. I like having the ability to load and unload at will and to have the opportunity to present my own error message to the user when a library is missing. Others disagree with me and prefer to use static bindings. That's perfectly fine.

At this point, static and dynamic bindings exist for several popular libraries already. Deimos is a collection of the former and Derelict 3 the latter. You'll find some bindings for the same library in both and several that are in one project but not the other. Use what you need and are comfortable with. And I hope that, if the need arises, you can use the advice I've laid out in this series of posts to help fill in the holes and develop static or dynamic bindings yourself.

Given that I'm just under 4 hours from 2013 as I write this in Seoul, Korea, I want to wish you a Happy New Year. May you start and finish multiple projects in the coming year!


TicTacToe and Modules in D

Posted by , in D, Projects 02 December 2012 - - - - - - · 2,061 views

Given that my BorderWatch project has languished on github for months without any updates beyond the first few days of random hacking, it's going to be a while before it can serve as an example of game programming in D. So I had some free time recently and decided to do something different. I put together a simple TicTacToe game, that I call T3, and put the source up on github.

T3 is not a complete game. When it starts, you can play two players, one with the mouse and one on the keypad. You can press space after each game to clear the board, and Esc to exit the game at any time. That's all it is or, in the master branch at least, will ever be. It shows some very basic features of D and I hope can be used as a starting point for people new to D to play around with. Maybe by adding a menu screen and a UI, some AI, networking... whatever.

I've never programmed a TicTacToe game before and didn't consult any source for this one, so don't be surprised if you see something weird. However, there are a couple of architectural points that I want to highlight.

One of the features of D whose usefulness can be overlooked is that of the module. In C++, it is quite common for each class to have its own source file, sometimes even multiple files. And the declaration and implementation are often separated between header and source. C++ classes that are "friends" can have access to each other's internals. It is even more common in Java to see one class per file (and without the declaration/implementation divide), but without the help of the friend keyword. In D, it's much more useful to think in terms not of files or classes, but of modules.

Modules are designed to be used as something analogous to a C++ namespace. You group common functionality into a common module, the difference being that one module equates to one file. It was quite natural for me to start thinking in terms of modules (to an extent... see below), as that's what I have always done when programming with C -- individual C source files are often referred to as modules, and a common C idiom is to group common functionality in a single file. Others may not have such an easy time with it.

So as a result of this focus on modules, you find features that might be surprising. For example, you can have protection attributes (such as public, package and private) at the module level. Modules can have their own (static) constructors. But one module feature that really trips people up is that private class or struct members are visible throughout the entire module in which they are declared.

Take, for example, the tt.game module in T3. The abstract Player class declares a private member, _mark (on line 173 as of this writing). Scroll down a bit to the Game class and you'll see that it accesses this member directly. On first blush, that appears to be a severe violation of the encapsulation principle of OOP. But if you think about it, it really isn't. In this case, the game class needs to perform gets and sets for a player's mark. I could have used D's @property attribute to provide a convenient getter/setter pair in the player class, but that would be rather pointless given that both classes are in the same module. One of the oft-cited reasons (and a good one) for encapsulation is that it can hide changes to an implementation. But here, anyone changing the implementation, maybe by adding some sort of calculation every time _mark is accessed, will always have access to the game class because they are in the same module. Make the changes, get compiler errors, search/replace in the same module to fix them, done. If the outside world needed to have access to Player's _mark, then property accessors would be the right thing (especially if it were in a library).

The converse is true, too. In C++, I rarely, if ever, had variables in a source file that were declared outside of a class, static to the module, but accessed by that class. In D, I do it all the time. An example of that is seen in tt.main, where the module-private _game instance is used by the HumanPlayer class and also by the module-level functions below it. These days, on the rare occasions when I toy around with C++, I often find myself declaring certain variables in an anonymous namespace to be shared by two or more classes in the same file.

Another consequence of thinking in modules is that free functions become less of an issue. In earlier posts on this blog, I had a dilemma over whether or not to use free functions for a library I was wanting to develop. This is the one issue I had with fully embracing the module concept. In C, this problem just doesn't manifest because you're always dealing with free functions. But mixing free functions with objects just feels... dirty. Plus there's the potential for name clashes, and no one likes the this_is_a_function syntax common in C. In C++, there is an easy solution: wrap the free functions in a namespace. There are different ways to handle this in D. The most straightforward is just to not worry about namespace conflicts. When they do arise, the fully-qualified package name can be prefixed to the function name at the call site. I mostly accept this now, but as you can see in tt.gfx and tt.audio, I still add a prefix to free functions whose names I am certain will have a conflict (gfxInit and audioInit, for example). I have no practical reason for avoiding typing tt.gfx.init() at the call site. It's just a personal quirk.

If you do decide to do something with the code, please make sure to give the README a once-over. I would like to highlight the following paragraph.

Pull requests with new game features will likely not be accepted. I would like to keep this simple and useful as a toy for new D users. I might very well create a branch with multiplayer and AI for myself to play around with, but I really want to keep the master as-is. However, I'll happily accept pull requests for bug fixes and improvements to the build script (gdc & ldc support, for example).


This code is just a playground for anyone looking to experiment with D, so while I would love to see people do cool stuff with it, I'd rather not put any of that cool stuff into the master branch.

As an aside, I just realized that I didn't add a license to the repository. I'll take care of that pronto. For the record, I'm releasing this as public domain, with the exception of the Derelict bindings in the import directory which are released under the Boost Software License.
I'm overdue for part 5 of my Binding D to C series. I'll be sure to get that at some point over the next couple of weeks. Thanks for reading.


Binding D to C Part Four

Posted by , in D, Binding to C 05 August 2012 - - - - - - · 3,447 views

This is the fourth part of a series on creating bindings to C libraries for the D Programming Language.

In part one, I introduced the difference between dynamic and static bindings and some of the things to consider when choosing which kind to implement. In part two, I talked about the different linkage attributes to be aware of when declaring external C functions in D. In part three, I showed how to translate C types to D. Here in part four, I'll wrap up the dscussion of type translation with a look at structs.

A D Struct is a C Struct
For the large majority of cases, a C struct can be directly translated to D with little or no modification. The only major difference in the declarations is when C's typedef keyword is involved. The following example shows two cases, with and without typedef. Notice that there is no trailing semi-colon at the end of the D structs.

// In C
struct foo_s
{
	int x, y;
};

typedef struct
{
	   float x;
	   float y;
} bar_t;

// In D
struct foo_s
{
	int x, y;
}

struct bar_t
{
	float x;
	float y;
}

Most cases of struct declarations are covered by those two examples. Sometimes, a slight deviation may be encountered. Such as a struct with two names, one in the struct namespace and one outside of it (the typedef). In that case, the typedefed name should always be used.

// In C
typedef struct foo_s
{
	int x;
	struct foo_s *next;
} foo_t;

// In D
struct foo_t
{
	int x;
	foo_t *next;
}

Another common case is that of what is often called an opaque struct (in C++, more commonly referred to as a forward reference). The translation from C to D is similar to that above.

// In C
typedef struct foo_s foo_t;

// In D
struct foo_t;

Member Gotchas
When translating the types of struct members, the same rules as outlined in Part 3 should be followed. But there are a few gotchas to be aware of.

The first gotcha is relatively minor, but annoying. I've previously mentioned in this series that I believe it's best to follow the C library interface as closely as possible when naming types and functions in a binding. This makes translating code using the library much simpler. Unfortunately, there are cases where a struct might have a field which happens to use a D keyword for its name. The solution, of course, is to rename it. I've encountered this a few times with Derelict. My solution is to prepend an underscore to the field name. For publicly available bindings, this should be prominantly documented.

// In C
typedef struct
{
	// oops! module is a D keyword.
	int module;
} foo_t;

// In D
struct foo_t
{
	int _module;
}

The next struct gotcha is that of versioned struct members. Though rare in my experience, some C libraries wrap the members of some structs in #define blocks. I find this practice rather annoying (libpng, I'm looking at you), because it can cause problems not only with language bindings but also with binary compatibility issues when using C as well. Thankfully, translating this idiom to D is simple. Using it, on the other hand, can get a bit hairy.

Here's an example.

// In C
typedef struct
{
	float x;
	float y;
	#ifdef MYLIB_GO_3D
	float z;
	#endif
} foo_t;

// In D
struct foo_t
{
	float x;
	float y;
	// Using any version identifier you want -- this is one case where I advocate breaking
	// from the C library. I prefer to use an identifier that makes sense in the context of the binding.
	version(Go3D) float z;
}

Then, to make use of the versioned member, the '-version=Go3D' is passed on the command line when compiling. And this is where the headache begins.

If the binding is compiled as a library, then any D application linking to that library will also need to be compiled with any version identifiers the library was compiled with, else the versioned members won't be visible. Furthermore, the C library needs to be compiled with the equivalent defines. So to use foo_t.z from the example above, the C library must be compiled with -DMYLIB_GO_3D, the D binding with -version=Go3D, and the D app with -version=Go3D. And when making a binding like Derelict that loads shared libraries dynamically, there's no way to ensure that end users will have a properly compiled copy of the C shared library on their system unless it is shipped with the app. Not a big deal on Windows, but rather uncommon on Linux. Also, if the binding is intended for public consumption, the versioned sections need to be documented.

Read more about D's version conditions in the D Programming Language documentation.

The final struct member gotcha, and a potentially serious one, is bitfields. The first issue here is that D does not have bitfields. In D2, we have a library solution in std.bitmanip, but for a C binding it's not a silver-bullet solution because of the second issue. And the second issue is that the C standard leaves the ordering of bitfields undefined.

Consider the following example from C.

typedef struct
{
	int x : 2;
	int y : 4;
	int z: 8;
} foo_t;

There are no guarantees here about the ordering of the fields or where or even if the compiler inserts padding. It can vary from compiler to compiler and platform to platform. This means that any potential solution in D needs to be handcrafted to be compatibile with a specific C compiler version in order to guarantee that it works as expected.

Using std.bitmanip.bitfields might be the first approach considered.

// D translation using std.bitmanip.bitfields
struct foo_t
{
	mixin(bitfields!(
		int, "x", 2,
		int, "y", 4,
		int, "z", 8,
		int, "", 2)); // padding
}

Bitfields implemented this way must total to a multiple of 8 bits. In the example above, the last field, with an empty name, is 2 bits of padding. The fields will be allocated starting from the least significant bit. As long as the C compiler compiles the C version of foo_t starting from the least significant bit and with no padding in between the fields, then this approach might possibly work. I've never tested it.

The only other alternative that I'm aware of is to use a single member, then implement properties that use bit shift operations to pull out the appropriate value.

struct foo_t
{
	int flags;
	int x() @property { ... }
	int y() @property { ... }
	int z() @property { ... }
}

The question is, what to put in place of the ... in each property? That depends upon whether the C compiler started from the least-significant or most-significant bit and whether or not there is any padding in between the fields. In otherwords, the same difficulty faced with the std.bitmanip.bitfields approach.

In Derelict, I've only encountered bitfields in a C library one time, in SDL 1.2. My solution was to take a pass. I use a single 'flags' field, but provide no properties to access it. Given that Derelict is intended to be used on multiple platforms with C libraries compiled by multiple compilers, no single solution was going to work in all cases. I decided to leave it up to the user. Anyone needing to access those flags could figure out how to do it themselves. I think that's the best policy for any binding that isn't going to be proprietary. Proprietary bindings, on the other hand, can be targeted at specific C compilers on specific platforms.

Conclusion
I believe that's all I wanted to say about structs. In Part 5, which I'm quite certain will be the final installment, I'll talk about how to declare functions for both dynamic and static bindings and some of the issues that need to be considered when doing so. I'll also tie off any loose ends I think of.


Uniform Function Call Syntax in D

Posted by , in D 29 June 2012 - - - - - - · 10,080 views

Uniform Function Call Syntax (UFCS) is a feature of the D Programming Language that was finally implemented in all its glory in a recent compiler release. It has been available for use with arrays for quite some time, since the early days of D1. But now it is available for every type imaginable.

On the one hand, UFCS is nice syntactic sugar for those who hate free function interfaces (a group to which I do not belong). But it's more than that. It's also an easy way to extend the functionality of existing types, while maintaining the appearance that the new functionality actually belongs to the type.

Here's how it works. Given a free function that accepts at least one parameter, the function can be called on the first argument using dot notation as if it were a method of that type. Some code will make it clear.


import std.stdio;

void print(int i)
{
	writeln(i);
}

void main()
{
	int i = 10;
	i.print();
	8.print();
}

Notice that it works on both variables and literals (see the output over on DPaste, where you can compile and run D code on line).

For a long time, I was rather ambivalent about UFCS. I didn't see the need. After all, I have no problem with free functions. Then I found a situation where it's a perfect fit.

I'm using SDL2 in one of the many projects I've managed to overload myself with. The SDL rendering interface has several methods accepting SDL_Rect objects as parameters. While implementing a simple GUI, I wanted to maintain bounds information using a rect object. But I also need functionality SDL_Rect doesn't provide out of the box, like routines to determine the intersection of two rects, or if a rect contains a point. And despite not having a beef with free functions, it really does make a difference in the appearance of code when you have a bunch of free function calls mixed in with object method calls. So I started implementing my own Rect type, giving it an opCast method to easily pass it anywhere an SDL_Rect is expected. Then I realized how silly that is when I've got UFCS.

So I scrapped my Rect struct and reimplemented the methods as free functions taking an SDL_Rect as the first parameter. And now I can do things like this.

SDL_Rect rect = SDL_Rect(0, 0, 100, 100);
if(rect.contains(10, 10)) ...

auto irect = rect.intersect(rect2);

And so on. I also had need of a Point type, which SDL doesn't have. But it was ugly mixing 'Poin't and 'SDL_Rect', so I aliased the SDL_ bit away and it's now just 'Rect'. With the combination of aliasing and UFCS, it's possible to hide implementation details without using a full-on wrapper to do so. Of course, it's not entirely hidden as the SDL_Rect is still directly accessible and you can still use the type by name. But it certainly can come in handy.


Binding D to C Part Three

Posted by , in D, Binding to C 19 May 2012 - - - - - - · 4,518 views

This is the (long overdue) third part of a series on creating bindings to C libraries for the D programming language.

In part one, I introduced the difference between dynamic and static bindings and some of the things to consider when choosing which kind to implement. In part two, I talked about the different linkage attributes to be aware of when declaring external C functions in D. Here in part three, I'm going to begin discussing how to translate C type declarations into D. I'll continue the discussion in part four.

First off, I want to mention a particular page over at dlang.org called Converting C .h Files to D Modules. This is required reading for anyone planning to work on a D binding to a C library. This series should be considered a companion to that page.

Dealing With Types

When translating C types to D, the large majority take only a handful of forms:

* typedefs
* #defines
* struct declarations
* function parameters

All of the translation guidelines discussed in this post cover all four situations, but there are special cases regarding function parameters that will be covered in part five when I talk about function declarations. It's also possible to find global variable declarations in a C header. But, in my experience, they aren't generally encountered when creating library bindings.

Typedefs, Aliases, and Native Types

D used to have typedefs. And they were strict in that they actually created a new type. Given an int typdefed to a Foo, a type Foo would actually be created rather than it being just another name for int. But D also has alias, which doesn't create a new type but just makes a new name for an existing type. In D2, typedef was deprecated. Now we are left with alias.

alias is what should be used in D when a typedef is encountered in C, excluding struct declarations (more on that in part four). Most C headers have a number of typedefs that create alternative names for native types. For example, you might see something like this in a C header.

typedef int foo_t;
typedef float bar_t;

In a D binding, it's typically a very good idea to preserve the original typenames. The D interface should match the C interface as closely as possible. That way, existing C code from examples or other projects can be easily ported to D. So the first thing to consider is how to translate the native types int and long into D.

Fortunately, on the dlang page I mentioned above, there is a table that lists how all the C native types translate to D. If you look it up, you'll see that an int is an int, a float is a float, and so on. So to port the two declarations above, simply replace typedef with alias and all is well.

alias int foo_t;
alias float bar_t;

One thing I'd like to point out about that table, though. It lists the D int as equivalent to the C long. In most cases, this is true. But there is a possibility that the C long type could actually be 64-bits on some platforms, whereas D's int type is always 32-bits and D's long type is always 64-bits. As a measure of protection against this possible snafu, it's prudent to use a couple of handy aliases on the D side that are declared in core.stdc.config: c_long and c_ulong.

// In the C header
typedef long mylong_t;
typedef unsigned long myulong_t;

// In the D module
import core.stdc.config;

// Although the import above is private to the module, the aliases are public
// and visible outside of the module.
alias c_long mylong_t;
alias c_ulong myulong_t;

One more thing. If you are translating typedefs that use types from C's stdint.h, you have two options for the aliases. You can use native D types, since the sizes are fixed, or you can include core.stdc.stdint, which mirrors the C header, and just replace typedef with alias. For example, here are some types from SDL2 translated into D.

// From SDL_stdinc.h
typedef int8_t Sint8;
typedef uint8_t Uint8;
typedef int16_t Sint16;
typedef uint16_t Uint16;
...

// In D, without core.stdc.stdint
alias byte Sint8;
alias ubyte Uint8;
alias short Sint16;
alias ushort Uint16;
...

// And with the import
import core.stdc.stdint;

alias int8_t Sint8;
alias uint8_t Uint8;
alias int16_t Sint16;
alias uint16_t Uint16;
...

Enums

Translating anonymous enums from C to D requires nothing more than a copy/paste.

// In C
enum
{
	ME_FOO,
	ME_BAR,
	ME_BAZ
};

// In D
enum
{
	ME_FOO,
	ME_BAR,
	ME_BAZ,
}

Note that enums in D do not require a final semicolon. Also, the last member may be followed by a comma.

For named enums, you may want to do just a bit more than a direct copy/paste. Named enums in D require the name be prefixed when accessing members. Example:

// In C
typedef enum
{
	ME_FOO,
	ME_BAR,
	ME_BAZ
} MyEnum;

// In D
enum MyEnum
{
	ME_FOO,
	ME_BAR,
	ME_BAZ
}

// In some function...
MyEnum me = MyEnum.ME_FOO;

There's nothing wrong with this in and of itself. In fact, there is a benefit in that it gives you some type safety. For example, if a function takes a parameter of type MyEnum, you can't just pass any old int in its place. The compiler will complain that int is not implicitly convertible to MyEnum. That may be acceptable for an internal project, but for a publicly available binding it is bound to cause confusion because it breaks compatibility with existing code samples. One work around that maintains type safety is the following.

alias MyEnum.ME_FOO ME_FOO;
alias MyEnum.ME_BAR ME_BAR;
alias MyEnum.ME_BAZ ME_BAZ;

// Now this works
MyEnum me = ME_FOO;

It's obvious how tedious this could become for large enums. If type safety is not important, there's one more workaround.

alias int MyEnum;
enum
{
	ME_FOO,
	ME_BAR,
	ME_BAZ
}

This will behave exactly as the C version.

#defines

Often in C, #define is used to declare constant values. OpenGL uses this approach to declare values that are intended to be interpreted as the type GLenum. Though these values could be translated to D using the immutable type modifier, there is a better way.

D's enum keyword is used to denote traditional enums and also manifest constants. In D, a manifest constant is an enum that has only one member, in which case you can omit the braces in the declaration. Here's an example:

// This is a manifest constant of type float
enum float Foo = 1.003f;

// We can declare the same thing using auto inference
enum Foo = 1.003f; // float
enum Bar = 1.003; // double
enum Baz = "Baz!" // string

For single #defined values in C, these manifest constants work like a charm. But often, such values are logically grouped according to function. Given that a manifest constant is essentially the same as a one-member enum, it follows that we can group several #defined C values into a single, anonymous D enum.

// On the C side.
#define FOO_SOME_NUMBER 100
#define FOO_A_RELATED_NUMBER 200
#define FOO_ANOTHER_RELATED_NUMBER 201

// On the D side
enum FOO_SOME_NUMBER = 100
enum FOO_A_RELATED_NUMBER = 200
enum FOO_ANOTHER_NUMBER = 201

// Or, alternatively
enum
{
	FOO_SOME_NUMBER = 100,
	FOO_A_RELATED_NUMBER = 200,
	FOO_ANOTHER_NUMBER = 201,
}

Personally, I tend to use the latter approach if there are more than two or three related #defines, and the former if it's only one or two values.

But let's get back to the manifest constants I used in the example up above. I had a float, a double and a string. What if there are multiple #defined strings? Do you have to declare a seperate manifest constant for each one? No, not if you don't want to. D's enums can be typed to any existing type. Even structs.

// In C
#define LIBNAME "Some Awesome C Library"
#define AUTHOR "John Foo"
#define COMPANY "FooBar Studios"

// In D, collect all the values into one enum declaration of type string
enum : string
{
	LIBNAME = "Some Awesome C Library",
	AUTHOR = "John Foo",
	COMPANY = "FooBar Studios",
}

Neat, eh? Again, note the trailing comma on the last enum field. I tend to always include these in case a later version of the C library adds a new value that I need to tack on at the end. A minor convenience.

More to Come

I think that's about enough for this session. In part four, we'll take a look at structs. There are a couple of pitfalls to be aware of when porting them over to D and I'll give you some advice to get around them.


Setting Thread Affinity on Windows in D

Posted by , in D 03 April 2012 - - - - - - · 1,979 views

When working with D's standard library, it is sometimes necessary to work around missing declarations in the core.sys.windows.windows module. It's a fairly big module as is, but it isn't all-inclusive. If you are doing any heavy-duty Windows development, you'll likely want a third-party Win32 API binding. But if you just need to call a function or two, that's overkill.

A good example of this is the need to lock the timing thread to one core when dealing with the Windows timer on a multi-core machine. This is a well-known solution to the problem of erratic timing results returned by QueryPerformanceCounter. If you are using a library like SDL, which uses QPC under the hood for its timing calls, this is something you ought to consider. Unfortunately, Phobos is currently missing declarations for a couple of function calls and one type that you need to lock the timing thread down.

For many cases, this can be overcome by adding the appropriate declarations where you need them. In other cases, you'll find (particularly when using DMD) that the Win32 import libraries that ship with the compiler are outdated. In that case, you'll either need to generate them yourself or load the function symbols manually via LoadLibrary and friends. Luckily, for setting the thread affinity the solution is easy. Here's a complete example, ripped right out of the module where I use it.

version(Windows)
{
	private
	{
		import core.sys.windows.windows;
		import std.windows.syserror;

	    // Declarations missing from the windows module.
		alias size_t DWORD_PTR;
		extern(Windows)
		{
			DWORD SetThreadAffinityMask(HANDLE,DWORD);
			BOOL GetProcessAffinityMask(HANDLE,DWORD_PTR*,DWORD_PTR*);
		}

		void setThreadAffinity()
		{
			void doThrow(string msg)
			{
				auto err = GetLastError();
				throw new Exception(msg ~ sysErrorString(err));
			}

			DWORD_PTR procMask, sysMask;
			if(!GetProcessAffinityMask(GetCurrentProcess(), &procMask, &sysMask))
				doThrow("GetProcessAffinityMask failure: ");

			DWORD_PTR mask = 1;
			if(mask & procMask)
			{
				if(!SetThreadAffinityMask(GetCurrentThread(), mask))
					doThrow("SetThreadAffinityMask failure: ");
			}
			else
			{
				throw new Exception("Unexpected affinity mask mismatch.");
			}
		}
	}
}


Drop this into module or class scope and add the following to an init method somewhere in the same module (or another module if you move setThreadAffinity out of the private block).

version(Windows) setThreadAffinity();


Notice also that I imported std.windows.syserror, which exposes the sysErrorString function. I'm sure I wasn't the only one who overlooked that module. After years of using D, I only noticed it recently. If you're going to be making Win32 API calls, it will come in handy.






December 2016 »

S M T W T F S
    123
45678 9 10
11121314151617
18192021222324
25262728293031

Recent Entries