Like the title suggests I'm wondering if there is any standard way to go about this?

I could of course write a really simple program that reads an external file and generates a source file that sets up a byte buffer corresponding to it, but is there any other ways about this? Maybe something less ugly?

The reason I need to do this is that I'm developing a dll file that needs some external data, but it wouldn't do to clutter the plugin folder of the program where it will eventually end up with additional files.

Thanks for any suggestions,

Husbjörn

This article will do the trick ->
http://www.codeproject.com/Articles/4221/Adding-and-extracting-binary-resources

Like the title suggests I'm wondering if there is any standard way to go about this?
... The reason I need to do this is that I'm developing a dll file that needs some external data, but it wouldn't do to clutter the plugin folder of the program where it will eventually end up with additional files.

The nice thing about standards is that you have so many to choose from.

Two were already mentioned:

Embedding binary resources is good, you can access the resources using standard Windows functions, and do so from any program that loads the dll. The steps to get a resource and load it into a buffer aren't hard, but because of the ease you can end up with lots of duplicates in memory if you don't pay attention to what you are doing. If this is bad or not depends on your app.

Creating a big buffer in the data segment can be nice. You get a simple array with all your data conveniently present at load time. There are many great tools out there like xxd that can turn your data into buffers, and assorted tools to convert your data directly into object form without an intermediate step. The downside is only your app gets access.

The biggest difficulty with both of these is that they become part of the executable and require space at all times the DLL is loaded. This can be great if your data is small and if it will never change once the program is built.

But we don't know those for your data. We don't know how much data you have, creating a 600MB DLL is going to put a serious strain on load time. We also don't know if the data has a separate lifetime from the DLL, if you want other people to be able to modify it without rebuilding a DLL, or if the resources come from only one file or many files.

You might be better off with a single separate file. Some options there might be:
* zip up all the stuff you need and extract as you need it
* dump it into a memory-friendly format and memory map into the file
* pack them up in a physfs or similar archive. This allows you to work as a normal file system during development, but pack it to a single compressed archive on release.

You might instead want to have a configuration file that accompanies the dll --- this is easy with the dot net configuration libraries --- so that you can keep the files wherever you want on the disk in a more traditional directory tree but only have one file in your plugin directory, "foo.dll.config", that anyone can modify and that can even be unique for each user of the machine.

If you are feeling non-traditional you can add a separate data stream to your file and write the data there. The OS uses alternate data streams all the time, as do many programs wanting to attach metadata to user created content. It can frustrate casual copiers and confuse people trying to understand disk space numbers, but it does keep the data in a single file.

There are many more options, really limited only by your creativity.


Each option solves different variations of the problem, and each has its own list of pros and cons.

I wrote a C# visual studio extension that takes a source file with a custom extension and creates an .obj/COFF file out of it that is linked in directly.

I should clarify, wasn't trying to boast. Rather, if that was an option you wished to pursue, I can provide links and/or advice on how I approached it.

I didn't think anyone would be interested TBH, but sure I'll post it sometime today when I get the chance.

Thanks for the suggestions. Didn't know you could just store binary data in a resource; thought you would have to pack it into a string table or a bitmap in that case. That's good to know :)

But we don't know those for your data. We don't know how much data you have, creating a 600MB DLL is going to put a serious strain on load time. We also don't know if the data has a separate lifetime from the DLL, if you want other people to be able to modify it without rebuilding a DLL, or if the resources come from only one file or many files.

You're right, I should have described it a bit better. Let's blame it on me making that post just before going to bed.

Anyway, I'm simply storing some default shaders and a bitmap font image that will be used unless the user specifies his own. So it's just a couple of small files.

Rather, if that was an option you wished to pursue, I can provide links and/or advice on how I approached it.

I think I'm fine for now but it would still be interesting to look at sometime if you don't mind posting it.

I uploaded what I have to here: https://sourceforge.net/projects/rfx/. Its my first C# program so it may be a bit rough around the edges, but I think you should get the idea. The COFF.cs file handles COFF exporting. PassData.cs stores data to be saved and handles C++ name mangling, though its easy to remove if you want to use extern "C" style declarations.

The 'addon' compiles .fx files and instead of saving them to a .cso file, wraps them up into a .obj/COFF object file to be directly linked into the executable. The rfx.targets/.xlm/.props files are used to actually 'link into' Visual Studio. You'd want to copy these 3 files and the compiled .dll into the appropriate MSBuild build customization folder. Then add it via the Build Customization menu much like you would MASM or similar customizations.

Anyways, let me know if you have any questions, feel free to use it for whatever...