shader system implementation

Graphics and GPU Programming Programming

Started by Basiror September 15, 2005 08:15 AM

44 comments, last by zedzeek 18 years, 7 months ago

229

September 17, 2005 08:06 AM

Quote:Original post by Basiror
e.g.:
a grass shader with (normal grass, boring grass, animated grass)
lets say you work with an older graphics card and implement these shaders mentioned above
they all provide some properties used to render them
now you switched to a modern graphic card and you want to update the graphics quality you just release a new dll with a higher priority to provide the functionality used by the shaders, just in a modern manner maybe with enhanced features
the shader dll itself runs a test on you system to check if everything is available otherwise the system chooses the old dll

this allows you to specialize a generalized representation of a effect for certain graphic cards without rewriting and updating a ton of code

the shader dll itself just does the state processing
the effect properties are still seperated in a effects/shader file

People all over the world update their graphics, by releasing patches, which include executable and the art needed.
Making DLL system just for the ease of updaing the graphics is little pointless, because the typical executable size is nothing to download today, and provides a lot more freedom to implement visual features for modern cards (if we are talking about that) - no shader can ADD new features, nor modify quantities of older ones (say adding light shafts, fog layers, spider-webs over the walls, etc.).
IF you go that way, the executable patching is the way, DLL-patching is half-the-way, not-really-there.

A system with priorities looks dangerous. Tweaking parameters by hand, and overriding priorities looks dangerous.
We create shader, set a priority and hope it will be resolved.
Better is to explicitly put the new shader, in the effects we want with the priority we want - it is more stable.

LarsMiddendorf

122

September 17, 2005 08:19 AM

@Yann L:
How do you solve the problem with the big number of shader permutations?
With PS30 HW you can put 4 or more lights(with shadowmaps) into one rendering pass. But each light/fog should also have its own shader fragment: point light, spot lights, lights with a cubemap, volumetric fog. Combined with some material shaders the number of combinations grows exponentially.
Precompiling is impossible but creating on the fly is also slow. Cg's interfaces are nice, but they don't really help here, because the cause a recompilation. Generating asm shaders is doable, but you lose specific compiler optimizations(and it's even impossible with glsl).

Zemedelec

229

September 17, 2005 08:36 AM

Quote:Original post by LarsMiddendorf

One working system that I know of, just compiles all combinations in preprocess, search unique resulting shaders and removes all duplicates.

Basiror

241

Author

September 17, 2005 09:00 AM

Well as mentioned above i don t want to encapsulate the shader itself into the dll
i would rather implement a set of features in the dll which makes use of latest extensions available
and in a shader file you define the effects you d like to achive with a description language

at loadtime the system checks which shaders to load,
checks the dlls for supported features and availability on the current hardware(e.g.: multi texturing with 2 4 8 16 layers *just an example don t rant me*)
and links the shaders with the dlls

the dlls just provide some extended functionality you offer the artists which on the other side could not be implemented with a scripting language

of course i could recompile a executeable and upload it, but no modder can customize the rendering pipeline to fit his needs and i certainly don t want to release my engine source only the gameplay components which are encapsulated into a seperate dll as you know it from halflife1

light shafts, spider webs or what ever aren t features that are implemented by a shader, light shafts belong to your sky model implementation in my opinion

effects aren t shader related either,

for me a shader defines
- material properties, effect name on bullet hit
- texture names ....
- blending operations
- parallax mapping dot3 bumpmapping and such

effects:
- name of effect
- particle effects
- sound effects
- other effect types

to implement the functionality of the shader as i defined it in the context of my engine, the dll provides a interface for the capabilities your hardware offers

you can imagine it as a renderer plugin with the little addition that everyone can customize it specialize it for certain hardware ....

as for the supply of information to render you could implement another system that allows the creation of additional render information such as a color array for each vertex in a mesh
all you had to do is to implement a api into your mesh class, or whatever you use to represent your geometry, that allows a preprocess to be run that generates the additional data so you shader can handle it

you could implement a preprocess inside your shader that runs the preprocess on the mesh

this is just an idea to extent the whole think without touching the executeable's code
of course you can t implement each gimmick you wish to that way but i allows for a lot of customization without much of an effort

[Edited by - Basiror on September 17, 2005 9:00:25 AM]

http://www.8ung.at/basiror/theironcross.html

Zemedelec

229

September 17, 2005 09:29 AM

Quote:Original post by Basiror
of course i could recompile a executeable and upload it, but no modder can customize the rendering pipeline to fit his needs and i certainly don t want to release my engine source only the gameplay components which are encapsulated into a seperate dll as you know it from halflife1

The patching by releasing entire executable was to show that releasing a DLL will not enhance top-level graphics in a way that releasing whole executable will.

About modding - do you really want your possible modders can alter shader database?
It can be done, without messing with executable code completely, just keeping shaders as source (or ability to load them that way too), and possibly cache compiled results later.
Or (even better), release the tools that let your artists create shaders! :)

Basiror

241

Author

September 17, 2005 09:47 AM

you still mix the shader and the dlls

the shaders are just a description of the surface and maybe vertex and pixel shader programs,

the dll however implements a high more abstracted part to provide additional functionality just as if you d link the SDL library to your c++ program and make use of SDL's build in functions to setup a window

thats a huge difference in my eyes

http://www.8ung.at/basiror/theironcross.html

Zemedelec

229

September 17, 2005 09:52 AM

Quote:Original post by Basiror
you still mix the shader and the dlls

the shaders are just a description of the surface and maybe vertex and pixel shader programs,

the dll however implements a high more abstracted part to provide additional functionality just as if you d link the SDL library to your c++ program and make use of SDL's build in functions to setup a window

thats a huge difference in my eyes

Yes, we use different terminology. Give an example, of one DLL and shader usage, like you see them to be used.

Yann L

1,806

September 17, 2005 10:09 AM

Quote:Original post by Zemedelec
People all over the world update their graphics, by releasing patches, which include executable and the art needed.
Making DLL system just for the ease of updaing the graphics is little pointless, because the typical executable size is nothing to download today, and provides a lot more freedom to implement visual features for modern cards (if we are talking about that) - no shader can ADD new features, nor modify quantities of older ones (say adding light shafts, fog layers, spider-webs over the walls, etc.).

Of course the DLL approach can do that - that's the whole point of it. And obviously it is not always possible to patch the executable for simple reasons such as application deployment (games are not the only sector where shaders are used), extensibility without access to the application or game source code (what would Max or Maya be without plugins), etc. The whole idea behind the DLL approach is flexibility. Patching the executable completely takes away that flexibility.

Quote:Original post by Zemedelec
IF you go that way, the executable patching is the way, DLL-patching is half-the-way, not-really-there.

Executable patching is often not an option, and to be frank, it is a pretty primitive way of extending functionality of your software. See Plugins.

Quote:Original post by Zemedelec
A system with priorities looks dangerous. Tweaking parameters by hand, and overriding priorities looks dangerous.
We create shader, set a priority and hope it will be resolved.
Better is to explicitly put the new shader, in the effects we want with the priority we want - it is more stable.

I think you didn't quite understand the system. There are absolutely no stability concerns, the system is stable in itself. In fact, it is much more stable and reliable in selecting an appropriate shader than a manual approach could ever be. Especially on large scale system, with many shaders.

Quote:
How do you solve the problem with the big number of shader permutations?

They're generated and compiled on the fly right now. One must be careful about terminology here: many people use "shader" as an equivalent of a "shader program". That is incorrect. A shader is a structure, visual description and algorithm to simulate a certain visual appearance. Part of a shader can be a shader program (eg. a GLSL shader), but this is not mandatory.

Taking this into account, shaders will never suffer permutation problems. Shader programs will - and those can be auto-generated by their corresponding parent shader. On the fly compiling isn't ultra fast, but this was never a real issue in my experience. Takes at most a couple of seconds, when the effect system resolves the shaders.

Basiror

241

Author

September 17, 2005 10:36 AM

Quote:Original post by Zemedelec
Yes, we use different terminology. Give an example, of one DLL and shader usage, like you see them to be used.

shader <shadername>{  stream(vertex,normal,color,blendfactors);.....}

now lets say you simply render your scnene with pure vertex arrays and basic opengl lighting so the shader above fullfits your needs

however the auther wants to use dot3 bumpmapping so he has to tell the render what he needs

shader <shadername>{  stream(vertex,normal,color,blendfactors);  enable dot3_bumpmap.  normalmap "some.tga".....}

the DLL features dot3_bumpmap and normalmap and knows what to do with those keywords
in this case the dll and the shader are linked *hence: implicitly*
however another approach would be to specifiy the dll's name within the shader to make guaranteed use of a certain dll

so in your mesh you have to run a preprocess or you run the initialization the first time you render a mesh although this might lead to lag spikes

the TBN needs to be calculated ....

this is just an example on how i would use it in addition you could store seperate streams created by the preprocess of the dll
the whole concept of interaction of geometry with the additional functionality still needs to be outlined in a straight forward and clear way but thats the basic principle i thought of

http://www.8ung.at/basiror/theironcross.html

Zemedelec

229

September 17, 2005 10:55 AM

Quote:Original post by Yann LOf course the DLL approach can do that - that's the whole point of it. And obviously it is not always possible to patch the executable for simple reasons such as application deployment (games are not the only sector where shaders are used), extensibility without access to the application or game source code (what would Max or Maya be without plugins), etc. The whole idea behind the DLL approach is flexibility. Patching the executable completely takes away that flexibility.

I was talking about the shader-into-DLL approach, not general DLL concept.
No doubt, updating a part of a system, rather than whole system is more flexible, if implemented.

As for DLL updating as is, a rant:
Software, that do that and have plugin structure have serious requirements to do so - it is developed often by large teams, that create/update parts independantly. And for other concerns, like stability.
Nothing of that concerns the typical game scenario, games imho are not so big software.

Quote:Original post by Yann LExecutable patching is often not an option, and to be frank, it is a pretty primitive way of extending functionality of your software. See Plugins.

If we target system complexity, that could be good point to start... :)
My personal decidion would be not to trade the design/implement time for pluggable system that can update everything, for the sake of uploading 1.5M more. I would go for stable-old (KISS) solution... but its just me, yes.

Quote:Original post by Yann LI think you didn't quite understand the system. There are absolutely no stability concerns, the system is stable in itself. In fact, it is much more stable and reliable in selecting an appropriate shader than a manual approach could ever be. Especially on large scale system, with many shaders.

I agree with one - this system will ALWAYS produce an output for asked shader. Be it solid color, at the end.
The thing I don't like about it, is that problem: I want to manually control the fallback of given shader on lower systems for effect A, and for effect B differently and explicitly. Keep good specular on one of them, but remove AO for one, and opposite for the other.
What I understand from the system you described in the oh-so-long thread, was that it will break the description of the shader into a set of smaller compiled and running on that hardware shaders, with the highest priority.
Can't see how this is more predictable and easy-to-develop for a typical not-shader intensive game. For me, it's like having quite many renderpaths, and being unable to guarantee the visual quality in one of them. Maybe it is suitable for software, but for games - can't agree.

And for the sake of "graphic theory" nature of the forum, I totally agree with the post about shader abstraction beyond the passes and profiles.

shader system implementation

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

shader system implementation

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

Reticulating splines