1)You can use D3DReadFileToBlob or just read the file as a byte buffer (more flexible if you have a sort of big-and-unique binary archive for shaders, instead of many multiple files, for cache reason).

2) I don't see any particular benefit of compiling HLSL shaders on a end user machine since Microsoft give us a HLSL compiler, it just add extra time at runtime (and it could be a lot of time) before the game starts.

This link explains the different ways to compile/load shaders. By the way, there are several search engines such as google, dogpile, etc., that can provide you a lot of information within seconds.

Ninja'd.

EDIT: During development of a shader and it's associated setup (constant buffers, input layout, etc.), it can be well worthwhile to compile shaders in your app for debugging purposes. Particularly if you're only editing the shader without needing to recompile the rest of the app, you can text edit the shader, run the app, edit, run, etc.

there will always be the specific case when it bites you in the posterior

Most of these chipsets can have problems with precompiled shaders, ...

For everyone's benefit, would you share specifics (rather than a general warning) about what problems you've had with the Microsoft shader compiler, which chipsets are incompatible with such precompiled shaders, examples of HLSL syntax or keywords that cause problems, etc.?

You have to be careful with precompiled shaders, in the general case it's probably a very good thing, but there will always be the specific case when it bites you in the posterior.

I personally have much, much more expierience in OpenGL than DX, and in OpenGL you should never use precompiled shaders. IMHO

The shader compiler is either a device specific one (usually supplied by the GPU manufacturer, often buggy and out of date, .... have a little cry thinking about it) , or shipped as part of the device drivers. (which you can update ... little cheer)

Most of these chipsets can have problems with precompiled shaders, if you are working in the OpenGL ES world they often are not supported at all. I can think of only a couple of devices that support them.

Which makes me wonder how Microsoft have got it to work at all. The hardware is no different from the chipsets Microsoft are running on. I sometimes wonder if HLSL compiles to pseudocode which gets 'recompiled' when loaded. Don't know, would be interesting to know.

HLSL is compiled to bytecode (Microsoft's specification), so the GLSL issues (different device, different GPU ISA, different drivers, double compilation) are thrown away. The HLSL bytecode then is compiled once at runtime by the GPU drivers to enable optimization.

Yes, HLSL compiler produce a vector bytecode (current GPUs are "super-scalar"), and sometime it adds some useless optimization, but it works well in the large. Of course you need to profile and be careful with the operations and operations' order to achive the maximum performance, but it still works well.

So, compiling HLSL at runtime does not produce any particular benefits outside the development process.

Which makes me wonder how Microsoft have got it to work at all. The hardware is no different from the chipsets Microsoft are running on. I sometimes wonder if HLSL compiles to pseudocode which gets 'recompiled' when loaded. Don't know, would be interesting to know.

Like Alessio mentioned, the HLSL compiler produces hardware-agnostic shader assembly as output. It's basically an intermediate bytecode format, and it's JIT compiled by the driver at runtime into the native microcode ISA used by the GPU. It's rather similar to the process used by Java and .NET to generate runtime code. Compared to OpenGL it has a few advantages, namely that all of the language parsing and semantics is done through a unified front-end rather than having different implementations per-driver. It also can do a majority of optimizations (inlining, constant folding, dead-code stripping, etc.), which is nice since you can do that offline instead of having to do it at runtime. The downside is that the compiler can only target the virtual ISA, and the JIT compiler that produces the actual bytecode won't have full knowledge of the original code structure when performing optimizations.

FYI Nvidia also has PTX which serves a similar role for CUDA, and AMD has IL.

I personally have much, much more expierience in OpenGL than DX, and in OpenGL you should never use precompiled shaders. IMHO

OpenGL doesn't actually have precompiled shaders in the sense of Direct3D's.

Newer versions have allowed an application-controlled shader cache though that's typically built into the GPU drivers anyhow and not necessarily of any real benefit. Some people try to use these for distribution on certain mobile devices and the like, but it's a terrible idea since the GL shader cache blobs are NOT meant to be used on any machine aside from the one that generated them, period. The output of the shader cache is dependent on the specific hardware, the specific driver, the specific OS, and the specific versions of each of those. You might update a driver and your cache is invalid, or some mobile phone might switch chip suppliers a couple months after being released and hence have a slightly/severely different GPU.

The only real use for OpenGL's shader cache is to compile the shaders on the end-user's machine during installation so that subsequent launches don't need to recompile anything (and then be prepared to recompile them when the game starts in the event the user dares to upgrade a driver or install a different GPU). In this use case and this use case only, OpenGL's shader cache is not only useful but can actually result in better load times than Direct3D.

You can do something similar with Direct3D by shipping HLSL, compiling to bytecode, caching that, and then passing that in on each startup, but the final compilation steps will always be performed every time you call CreateVertexShader or friends. There's no way for your application to read or save the final hardware-specific shader code in Direct3D, much less load it. However, it's typically a better idea in Direct3D to ship the compiled bytecode, if for no other reason than that it removes a dependency on d3dcompiler_XX.dll from your end product (which is not part of the standard end-user Direct3D redistributable). By not requiring external DLLs you can even make standalone .exe files that launch the game without needing an install process by building your data into the .exe itself, which is handy for some types of games in some circumstances since Fewer Steps to Play == More Players (so more in-game sales in a f2p title, more chance for a reviewer or producer to try it out, etc.). There's a few off-the-shelf engines that have the ability to build stand-alone game executables like that.