  
Welcome to Ventspace! Most posts here are delayed copies of posts from the real Ventspace.
  Building the SlimDX Installer | 
Posted - 6/19/2008 7:04:17 PM | The spy museum in DC is pretty cool, but it's not part of the Smithsonian. That is why:
1) It isn't on most of the Smithsonian maps, making the task of finding it somewhat irritating.
2) It charges $18 at the door.
I don't mind 1 that much, but 2 was really obnoxious.
Building the SlimDX Installer
Developing the SlimDX installer turned out to be a massive headache. I believe there is good installer creation technology out there; however, it would appear that I do not actually have access to it. When it comes to the world of install technology on Windows, there are basically two groups:- Installers that create Microsoft Installer (MSI) files. This includes the Setup Project in Visual Studio, and stuff created by InstallShield, Wise, WiX, and so on.
- Installers that do their own thing. In this group are NullSoft, InnoSetup, and some others.
Now, the first headache is simply the task of selecting which installation system to use. I threw out InstallShield and Wise because they cost money. WiX was a huge pain and everybody seems to complain about it, so I didn't give it much time of my own. That left Visual Studio, NullSoft/NSIS, and InnoSetup.
The SlimDX installation had three basic requirements. First, it had to install the basic native prerequisites that SlimDX apps always require. For the June release, that meant the Visual C++ 2008 (version 9) runtime, and the DirectX redistribable files for June 2008. Second, it had to register SlimDX in the Global Assembly Cache (GAC). Lastly, it had to do an installation to a normal Program Files directory, which is necessary for VS to find the XML and PDB files, and for people to distribute SlimDX as a private assembly.
The first sticking point came when I was looking into registering SlimDX in the GAC. One would expect that this is fairly simple. Not so much. See, the usual method developers are shown for registering their stuff in the GAC is to use the gacutil.exe utility to do it. It turns out that this is only for developers. Normal users don't have gacutil, and you aren't allowed to redistribute it as part of the installer. So how do you register in the GAC? MSI can do it; if you're not MSI, you can go to hell. (There's a class in the .NET Framework that can do it, so if your installer can handle plugins written in .NET, then you can pull it off.) At this point, NSIS and InnoSetup dropped out of the running. No GAC registration was a non starter, and I didn't want to use a hack for something so simple and fundamental.
Okay, so now I was down to VS, which I didn't think was a bad thing. It generated these hideous, amateurish looking installers, but I found out that if you know where to look, you can reconfigure that stuff so that it actually looks like a decent bit of software. I mean it's VS, right? Stuff might be a little weird, but it's basically good at what it does? Not exactly. Visual Studio is a terrible installer authoring system. There doesn't seem to be any specific reason for it, either. It's just all around shoddy, and omits stupid things that should be simple. For example, there is an interface for working with special folders like the Windows directory without having to give them explicit paths. That's great...except there's no support for getting the temporary directory. There's no reason not to, because the MSI format supports it, and if you know the property name, you can write it explicitly in VS. You just can't get it automatically, so you have to check the documentation for what the properties are.
After tweaking the project for a while, I got back around to GAC registration. It turns out that Visual Studio doesn't look at the GAC when adding references. It actually reads a set of registry keys (HKLM or HKCU, SOFTWARE\Microsoft\.NETFramework\AssemblyFolders) and uses their default values to get a list of search directories, and then it loads assemblies that it finds out of those directories. So to install a managed assembly, you put it somewhere on the file system apart, then register it with the GAC and also add the assembly folders entry. VS even has a Registry view that allows you to do this sort of thing...except it doesn't let you edit the default value of a key. Again, this is something that is trivially supported by MSI, VS just doesn't do it. Unfortunately, there's no hack-around in VS itself. You have to install a tool called Orca to modify the MSI after VS finishes building it, and write in the registry value yourself. Orca does support automation, but I have yet to figure out exactly how.
Oh, and on Windows XP 64 or Vista 64, VS reads from HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432node\Microsoft\.NETFramework\AssemblyFolders instead, because it's a 32 bit app and it gets redirected. That took a little figuring out, because the installer runs as 64 bit and doesn't write under WOW6432node. What I haven't figured out, however, is why Windows/msiexec completely ignored me when I tried to write the key under WOW6432node. That's why 64 bit users need to browse to the SlimDX directory manually.
Installing the VC9 redist turns out to be rather easy, but the results are clunky. VS allows you to ask it to create a bootstrap EXE that installs requirements like the CRT and .NET -- but not DX for some reason -- automatically as necessary before launching your installer. All you have to do is check the boxes. Problem is, it doesn't package the MSI or the CRT redist into the EXE, like you'd want. It just leaves them hanging there, loose. I couldn't find any useful information on changing this behavior, or really doing much of anything about it. I ended up writing a simple EXE file that embeds the EXE, the MSI, and the VC9 redist folder inside itself. That's the published SlimDX installer; when you run it, it will stage those files to the temporary folder (typically Documents And Settings\USER\Local Settings\Temp), and invoke setup.exe there. The sad part is, despite being a total hack, this is probably far easier than more maintainable than any other way out there for packing things together, self extracting ZIPs included. (Although for this June release, the DX team finally got around to making their installer not a POS zip-inside-a-self-extracting-zip.)
There was also the question of installing the DirectX redistributable. There's a web installer...but that's not redistributable and so it's worthless to us. The normal redistributable package is about 70 MB, and includes every redist package since the D3DX library was made dynamic. That was in 2005. Somehow, I didn't really feel that SlimDX users needed DirectX from October 2005. It's just a waste of space and bandwidth. Luckily, the DX setup program is flexible, and allows you to reconfigure it and simply delete old components. I cut it down to a minimal set that doesn't include DX 9.0c core install files (which are huge), and that subset was fairly compact -- about 8 MB. Not bad. Now I just had to actually run that minima installation as part of the SlimDX installer, and everything would be good. It shouldn't be a surprise that there was precisely zero built in support for doing this.
After some research, it was clear that the only way to really do this was to write a "custom action" for the install. That doesn't sound so bad...except custom actions are either JScript/VBScript (highly unrecommended), or DLLs. (Managed DLLs can be used, and are handled via a native proxy DLL. Blech.) I decided to just follow this guide (which is poorly written, frankly) and build a C++ DLL called DXSetupAction to handle this for me. That way I'd even get to use the MS DirectSetup API, which can be used to easily and transparently install the DX redist. (Although the 1.5 MB dxsetup.exe file is required to be redistributed even if you do use DirectSetup. What the hell.) It was here that I realized that there was no clear way to access any of the DX redist files if they were embedded in the MSI. It sounded possible, but I couldn't really see how. (Later, I realized that the custom actions only run AFTER your application installation, so I could have simply installed the files as part of normal installation, even if that is rather contorted.) I decided to simply embed all of the redist files inside my DLL, just like I did with the main EXE, stage them to the temporary directory, and call DirectSetup from there.
I would love to say that worked, but it didn't. When testing on my machine, everything seemed fine. It was only when I moved to clean VM tests that I noticed that DirectSetup wasn't doing a goddamn thing. It would load out of the DLL. It would find the function. It would run the function, and it'd get a success code back. But it didn't actually accomplish anything. I decided to simply CreateProcess on dxsetup.exe with the /silent flag instead, which did work as expected. Maybe that's why you're required to redistribute the executable -- it saves you time once you notice their API is broken. On the bright side, I now have a single DLL which you can easily add to anything that can load it, MSI or otherwise, and it will quietly and neatly install DirectX for you. (The version's pretty easily configured, and it can do either x86 or x64.)
In any case, I finally had it working. Let's look at the layout: all of the DX files are embedded in DXSetupAction.dll. DXSetupAction.dll, and SlimDX.dll/.pdb/.xml are all embeddded in SlimDXSetup.msi. SlimDXRedist32.msi, setup.exe, and a vcredist_x86 folder are all embedded in SlimDXSetup.exe. I personally find this whole thing to be fairly ridiculous...but it works. This was a fairly simple install, too. I can't imagine what kind of horrors I'd have to endure if I was doing something more complex, like Visual Studio or Office. As I said before with the docs, I can't help but think that they must have something better than what the non-paying public has been given access to.
| |
| SlimDX Installer Teaser | |
Posted - 6/13/2008 5:37:50 PM | I've been experimenting with driving to improve my mileage lately. I managed to gain about 4.5 mpg, from 23 to 27.5. It was really boring though, so I'm not sure if I will do it again. I prefer driving like a nutcase to saving a couple dollars...
SlimDX Installer Teaser
I was going to write up a beautiful entry on the process of creating an installer for SlimDX. However, that process took me three long days, and has been a thoroughly exasperating process. Not to mention I still need to test the damn thing. I thought the docs were bad, but this was much worse.
Plus it's freaking Friday evening. So I'm giving you guys this instead.
I'll throw up the write up on creating that thing in a few days. I'll tell you one thing though -- if any of you are building a Windows installer for a DX based program and want to include the redist, come to me. I can hook you up.
| |
| Building Documentation for SlimDX | |
Posted - 6/9/2008 4:48:24 PM | I'm writing this entry while waiting for a partially frozen Visual Studio using exactly 25% CPU (spread evenly across 4 processors, mind you) to do whatever the hell it is so intent on doing between building docs and allowing me to interact with it again. I believe it has something to do with the 6500 HTML files that are referenced by this project. Maybe VAX is involved. Thankfully though, I fixed this problem by the time I finished writing this journal entry.
Oh, and before you do anything else, watch this video. The Eastern Bloc has clearly been hiding some amazing music from us.
Building Documentation for SlimDX
One of the major pieces of work on SlimDX for the last release cycle has been documentation. We've written more of it than is humane, and I've been working hard on tweaking and perfecting the presentation. Like most things dealing with SlimDX, this turned out to be a hell of a lot harder than it should have been. I really wish I knew what Microsoft was doing with MSDN, because they can't possibly be struggling with the same problems I am and still be getting anything done.
Like many codebases nowadays, and nearly all managed ones, SlimDX docs start out as specially formatted doc comments embedded in the source code. The compiler reads off all these comments and produces an XML file alongside the output binary that encodes all of the documentation information. This part is a little obnoxious in C++/CLI, because the compiler has to look up and resolve all of the names referenced in the doc comments for cross linking. If it can't see the relevant identifier, it can't link it up correctly, and there are other problems with referencing specific members as well. These are all basically artifacts of applying the managed doc compile system to C++ code, and they make a mess of things. In most places we hack around it with forward declares or extra includes. Other than that insanity, though, this step basically works just fine.
The next step in producing documentation is to convert this XML file and the SlimDX DLL to actual readable documentation. That's done using a process of reflecting on the binary to find all of the types, members, etc and applying the correct bits of XML and XSL transforms to produce an output document. The format of the output document can be basically anything you want; just use XSL transforms that get you where you want to go. Microsoft provides a build engine for doing all of the reflection and transforms and everything called Sandcastle. Sandcastle is, for the most part, a pretty slick tool. Combined with a GUI to save you the annoyance of dealing with bat files and config files and junk, it produces some pretty neat looking documentation in HTML 1.x (CHM), HTML 2.x (VS), and online web format. I even patched the transforms to generate an extra little bit on pages which identifies the native equivalents to SlimDX types. Look at the SlimDX documentation for March 2008 as an example of the online documentation.
So far so good. Except for one detail -- this isn't good at all. The page loads slowly, because the index.html file is about 1400 KB. It uses frames, so you can't link to specific topics inside it, and the navigation frame doesn't update as you navigate. And as a final kicker, it took close to five minutes to build the docs each time, on my fast as hell work machine. I'm not willing to accept this level of quality, except temporarily while I make something better. So I've spent some time trying to make things better. My first idea was to try the other themes available out of the box with Sandcastle. There are three -- vs2005, Prototype, and hana. The one used currently in the SlimDX docs is vs2005. I tried hana, but it looks crappy and is even bigger. Prototype was much more promising; it compiled in around 3 minutes, index.html dropped by 300 KB, and the total size of all the HTML pages dropped to one third of the size of the vs2005 version. That was pretty good, with one caveat: the home page for Prototype was not displaying, either online in a browser or in the CHM file. It doesn't look good when the front page of your documentation is freaking blank.
After some web searching and a massive dose of good luck, I discovered Sandcastle styles. I initially though that it was a set of new styles for Sandcastle, rather than the ones provided by MS. That is not the case. It's actually a set of patches for the MS styles that fix their problems...including problems in the last version, released on May 30, 2008. Yes, that's right. Sandcastle's styles are so broken that people started another project just to make them usable. This is about where I began to realize that something is very wrong with the Sandcastle team and their product. (It's an amusing coincidence that Sandcastle's CodePlex site vanished while I was writing this.)
The patches did work, though, and allowed me to generate a fully working, non blank Prototype based set of documentation. Here it is. Except this hasn't solved anything. Thanks to expansion in SlimDX, the index.html file is back up to 1330 KB, and still loads slow. The layout is still frame based and you still can't link to specific topics. I've given up on the built in Sandcastle stuff at this point, because it's clear that it doesn't scale. Instead, I went looking for something that would handle all this more robustly.
I found DocProject. This thing is also Sandcastle driven, but does a lot of the heavy lifting by itself. It also integrates with VS 2005 or 2008. It also provides an external UI in case you're using the Express editions. One thing in particular caught my eye: Easily build an AJAX-enabled ASP.NET documentation website. Finally! AJAX based dynamic loading of topics, instead of a gigantic static list. This was clearly worth spending time with.
Right off the bat, VS 08 support doesn't work. It just crashes the IDE every time I try to build, on step 7. Fine, whatever; we just finished re-introducing VS 2005 support in SlimDX anyway. DocProject works with VS 2005, and creates a DocSite project (builds both online and CHM docs) using a project reference to SlimDX to do its work. I run a build, which churns away for a while, finishes building everything...and then freezes. It does this pretty much any time I do anything, just after finishing its work. It looks like the delay is in adding the HTML pages for the individual topics to the project, of which there are about 6500 currently. This process takes somewhere in the vicinity of an hour or two; I haven't had the patience to wait for it to finish instead of simply leaving for a while. Oh, and the external UI doesn't work, because it uses MSBuild and MSBuild can't understand project references to C++ projects. Fantastic.
Anyway, I can actually see the docs being produced by this thing, even if my iteration times are two hours or worse. This is all ASP.NET and AJAX hotness. Not bad. The HTML is even more compact. The navigation tree stays in sync with what you're looking at (doesn't autoscroll though). It allows you to save, print, bookmark, or email links to specific topics (but doesn't let you easily copy the link). Visually it's pretty snappy in default setup, and it can be reskinned like any other ASP.NET website. There's an admin page, index (which doesn't qualify names and so everything looks identical), and search that works well (but operates fully in memory and so mucks up the server somewhat). It extracts keywords (although it apparently picks up everything, including numbers). Okay, so it's not ideal. Still, we're getting much closer to the mark. Hey, it even has a space to put a logo in.
Now then, about those iteration times. I don't like waiting two hours between doing stuff. Like I said before, though, I think it's because all of the output HTML files are being added to the project file, which I just realized is optional. After turning it off...success! Things are fast! Thank god. So yeah, don't allow DocProject to add the Html directory to your project and everything will be fine. Builds are pretty snappy overall, and changing web page settings doesn't cause it to rebuild all of the docs. Unfortunately, its deployment option (which is supposed to copy just the files you need to upload out) doesn't actually work properly, so I'm building it partially by hand after the build finishes. I only need to do that when uploading though, which is not too bad.
Right now, I'm simply working on tweaking and tuning the current DocProject based documentation. It's an ASP.NET site after all, so there's a lot of configurability available. (Quite unlike the original Sandcastle stuff, which basically just offered a few dozen bool options.) These new docs aren't online yet due to some ASP.NET configuration issues on the server, but they should be visible soon, and I expect that they'll be the ones used for the June release. Here's a screenshot in the meantime.
| |
|
| S | M | T | W | T | F | S | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | | 10 | 11 | 12 | | 14 | 15 | 16 | 17 | 18 | | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | | | | | |
OPTIONS
Track this Journal
ARCHIVES
October, 2009
September, 2009
August, 2009
July, 2009
June, 2009
October, 2008
June, 2008
May, 2008
April, 2008
March, 2008
February, 2008
January, 2008
December, 2007
November, 2007
October, 2007
September, 2007
August, 2007
July, 2007
June, 2007
May, 2007
February, 2007
|
