Yep, passing a structure containing states is a good idea.

Even if you''re not using D3D as one of your APIs, I''d still recommend taking a look at D3DX Effects & Techniques and their files (.FX) for ideas - the concept is easily portable to OpenGL and even console APIs.

With .FX files you take the concept one step further, instead of going to the trouble of compiling each shader into a plugin DLL, the file is just a text file that that''s essentially a cross between a glorified .ini file and script.

The ability to change the operation of a shader in Notepad, while the app is running is really nice. If you add a bit of code to notify you when a shader file changes, the engine can do an automatic refresh and so the result of changes can be seen as you make them.


A look at other systems like ATIs RenderMonkey might be worthwhile too (i.e. let someone else write the editor tools for you...)

--
Simon O''Connor
ex -Creative Asylum
Programmer &
Microsoft MVP

if you are using d3d you can use state locks to record current renderer state.

I have considered the shader script approach in our engine, but I finally went with dll shaders for the reasons jamessharpe mentioned. Here a short overview, perhaps it can give someone a couple of ideas:

All user defined shaders are derived from an exported base class. The base class provides basic functionality, helper functions (vertex and pixel shader management, Cg connector, etc), gives access to general engine states, and auto-registers the new derived shaders at a central shader registry.

When a render display is opened, and the system has parsed the hardware caps of the 3D card, the registered shader classes are queried one by one. If the hardware doesn't offer the features required by a class, it is removed from the list. If it does, then the shader is queried for the number and type of passes it requires to create the effect on the current hardware. The info is stored. A shader class can dispatch or forward parts of the work to a different shader class, that allows for multipass setups. Those dependencies are resolved in the next step. This bouncing and forwarding of shader passes opens the possibility to express effects such as eg. shadowmapping (render from light, store in render texture, render geometry with depth texture) or multipass reflect/refractions through this simple interface.

Those pretty complex steps are only done once on setup. If a scene is loaded, each chunk of geometry has an effect signature that gives the system an idea about how it should look like. Eg: "render this geometry chunk with bumpmapping, local shadows and EMBM reflections. I don't care about the details, just do it as good as you can on the current hardware". Those effect classes are connected to the shader class dependency chain outlined above. The system chooses the shader class that is best suited to render the required effect, and sets a table of function pointers to the appropriate shader (could also work through virtual functions, but I thought function pointers might be a bit more efficient in this case).

So basically, while rendering, it all boils down to a few function pointer calls: setup_shader, enter_shader, shader_params, fill_shader_cache, exit_shader.

Edit: I'd like to point out a very important feature of that system (or similar systems): suppose your game shipped, and two months later, a top-notch 3D card with superb features hit the market. All you have to do, is write a couple of new shader classes for this hardware, and give them the appropriate effect signature, but with a higher priority than the ones you previously used in the game. Compile as a dll, put onto your website. A 20k download for the user, he puts it into his game directory, restarts the game, and voilà: brand new top of the line effects on the new 3D card, without changing anything in the game (as the shader dependency resolver recognizes the effect signature, and overides the old shader classes with lower priority).


[edited by - Yann L on July 20, 2003 4:51:48 PM]

I have both systems, a ''script'' approach with D3D/OpenGL compatibility, and a PlugIn/DLL approach too.
Altough I''ve not toyed with he PlugIns yet, and the script version is VERY fast.

I use a State approach (who doesn''t ?), TMUStates, VPStates, FPStates and GeneralStates (the later needs to get a better name BTW).
It works well.

Also those States objects are built by my renderer, which can be D3D or OpenGL, the only ''drawback'' atm, it that you cannot change Renderer during the game. You need to exit and relaunch it with the new API/Renderer...

Note however that it''s not a design problem, much more a lack of will to code it ^^ (it can be done, but I don''t see the point)

-* So many things to do, so little time to spend. *-

quote:Original post by jamessharpe
Is this a pre-process step of the render-queue? I can see that if multipass is required then there may be shader state switches that could be made redundant by sorting.

Actually, it is not done at render time, but as a one-time preprocess after a new scene has been loaded. It can take a couple of seconds, depending on the size of the scene, and the complexity of the shader dependencies. Essentially, for each geometry chunk in the scene, a best fit shader combination (with as few passes as possible on the current HW, and best possible quality) is evaluated, and stored with the mesh chunk.

Later on, in the actual render loop, this stored information is then simply used to call the appropriate shaders at the right time. An additional state-change optimizing sorting pass is applied each frame, after all visible chunks have been determined, and before the lists are dispatched to the shaders. It's a simple 48bit radix sort, sorting on shader ID and shader param pointer (so that shaders with the same parameter set are grouped).

quote:
Let me check I have this right:
[...]

Almost:

setup_shader - called once after the scene was loaded. Gives the selected shaders the opportunity to create some internal data, eg. normalization cubemaps, register vertex and pixel shaders with the engine, etc. It's pretty much the constructor of the shader. You can't put it into the real class ctor, since this one will be called at a time, where the render subsystem is not yet initialized. The destruction of those shader states is done through garbage collection, when the scene is closed.

enter_shader - sets any states that are required to render the effect

shader_params - called each time a primitive (geometry chunk) is drawn ( like glBegin), bind textures required and shader specific parameters. Called using lazy-evaluation, ie. only if the states changed from the last geometry chunk (supported by the radix sorting prior to mesh dispatching).

fill_shader_cache - Only called, if the geometry chunk is not yet cached in VRAM. Used to fill the vertices into VRAM, with the components required by the shader. If this data is still in the cache from the last frame, this function is not called.

exit_shader - Called, as soon as we are done with the shader, and a different one is selected. Typically, there is not very much in this function, but it can eg. pop back matrices that were pushed during enter_shader.

quote:
This means moving calls to stuff like glDrawElements into the shaders correct?

No, only the cache filling is done by the shader. Only the shader knows exactly, what components it needs in a vertex array. glDrawElements is called from the main render loop on the cache entry, outside of the shader.

quote:
I suppose the engine provides some basic shaders e.g. gouraud shading, single texture, two textures etc..

Yes, it provides basic fucntionality, but on a more abstract level (ie. not depending on the number of textures). For example: simple Gouraud, diffuse texture, diffuse and bump, diffuse + bump + specularity, etc. All in all, the engine provides approx. 50 basic shader types, but most of them reflection/refraction oriented. You can get basic geometry with good quality (ie. incl. specular maps + bump maps) by using, I'd say, 5 or 6 shaders.

quote:
I suppose that in the end all shaders(ignoring pixel and vertex shaders) could be implemented as multipasses of these 'basic' shaders.

Yep, that's the idea. And that includes pixel and vertex shaders, since the abstract base class provides vertex and fragment shader management. An external shader can register it's VP/FP (either as ASM or as Cg) with the system, and those will be activated as needed.

quote:
The advantage I see of this system over that of something like quake3's renderer codepaths is that it is more dynamic i.e. if a feature is supported it is used and also a fallback system can be made using a priority system - which would also enable us to have an easy way to adjust the render detail level.

Right. You don't really have to care about a fallback system, as it is implicitely given by selecting appropriate shader priorities.


[edited by - Yann L on July 21, 2003 1:56:57 PM]

quote:Original post by jamessharpe
So the shader dependancy chain for the effect is looked up and the geometry is duplicated in the render queue with each applicable shader, and this is actually contained within the mesh representation in the engine, correct?

The geometry itself is not duplicated, that would cost too much memory. It is simply instanciated for each shader pass, with a pointer onto the actual mesh data. So, for example, if geometry chunk A has bounced through 3 shaders (using 3 passes), the dependency manager will create 3 instances of that geometry, each containing a pointer to the original mesh chunk data.

quote:
So every geometry chunk passed to the renderer requires no more than one pass?

Each geometry chunk instance passed to the renderer requires exactly one pass by one specific shader, yes.

quote:
How are textures managed then? Are they specified in the shader params e.g. no of texture layers, handles to texture to be used, or is some other system used?

As you said, their handles (32bit GUIDs) are specified in the shader params. One for each texture type the shader might need: diffuse1 to 8, bump1 and 2 (for detail bumpmaps), specularity, opacity, reflectivity, refractivity, etc, depending on the effect type. Each selected shader can then extract and use the texture handles it needs.

quote:

quote:
Right. You don''t really have to care about a fallback system, as it is implicitely given by selecting appropriate shader priorities.

I realise that, but it is a useful tool for debugging and sometimes a player may want to switch off a particular feature in the game in order to achieve a higher framerate for a more responsive game.

Yes, of course. I didn''t mean that you wouldn''t be able to control it manually (you can), but that you don''t have to build in separate code paths. User defined quality params can simply be included as additional constraints in the dependency chain. Everytime the user has changed the visual parameters, simply resolve the dependencies, and continue rendering. Everything will automatically adjust to the new situation. That system could theoretically even handle hot-plugging of the 3D card

quote:
My hardware doesn''t support pixel or vertex shaders, so I won''t be implementing them for a while(perhaps in Directx with the software drivers(slow!)), but I suppose I ought to look in to how they are coded in order to make sure I don''t stop them being easily integrated.

That''s not so hard. You can simply treat VP/FPs as two additional shader states. In shader_setup, you register your VP and FPs, and get back a handle. When using a shader, you activate your VP/FP using that handle you got earlier. Just a simple additional state.