so how do you know where might the concave portion be? the 8 top cubes?
Exactly. There is basically 8 flags (which you can conveniently store in a single byte) that shows which of the cubes are occupied.
I still need to know who to apply a gradient in HLSL on my pixel shader.
[/quote]
You mean how?
I have this function in my GLSL shader (translating to HLSL should be trivial, you probably only need to replace vec2 with float2)
float occlusion(vec2 tx, uint occ) {
float res = 1;
if((occ & 1u) > 0u) res*=1-pow(clamp(1-sqrt((tx.x)*(tx.x)+(tx.y)*(tx.y)),0,1),2);
if((occ & 2u) > 0u) res*=1-(1-tx.y)*(1-tx.y);
if((occ & 4u) > 0u) res*=1-pow(clamp(1-sqrt((1-tx.x)*(1-tx.x)+(tx.y)*(tx.y)),0,1),2);
if((occ & 8u) > 0u) res*=1-tx.x*tx.x;
if((occ & 16u) > 0u) res*=1-pow(clamp(1-sqrt((1-tx.x)*(1-tx.x)+(1-tx.y)*(1-tx.y)),0,1),2);
if((occ & 32u) > 0u) res*=1-tx.y*tx.y;
if((occ & 64u) > 0u) res*=1-pow(clamp(1-sqrt((tx.x)*(tx.x)+(1-tx.y)*(1-tx.y)),0,1),2);
if((occ & 128u) > 0u) res*=1-(1-tx.x)*(1-tx.x);
return res;
}
"occ" contains the occlusion flags and tx are essentially texture coordinates of the quad (so [0,1] on both axes). The result are gradients that are 0 on concave edges and 1 on the others. So you then can use that in your lighting calculation however you like. In the example images I just do "color *= 0.7+0.3*occlusion(...);"
hmm... so besides the HLSL conversion, which as you said is trival, I will need to modify it a little bit in order to change the input texture coordinate (since you are using a geom shader, you are building your color for each quad into a texture if I'm not wrong) to be used directly with my color coordinate, and since I'm not using quads, I'm using cubes, I can make a little trick here in order to use the normals buffer to get which quad I'm reading..... does it makes any sence?
Geometry shader and cubes shouldn't really figure into this... what you need to provide is a quad/face relative coordinate system as you would when texturing the quads, texture coordinates essentially. When I speak of quads is mean the faces of the cube which in my case as well as i guess in yours are made up of two triangles. You then use this function in the fragment shader the same way you would use a texture lookup when texturing individual quads, except that you have to provide the occlusion information about the neighboring cubes.
yeah I got that about the quad, its the same thing as I was thinking but I have some optimization of vertex in place that makes it a little bit hard for me, since if a cube is next to another that is the same "type" the vertex is shared with its neighbor cube and the index is re-used... funky code I came up, after i figured out that I cannot do a full optimiazation of vertexes to be unique in space, due to some coloring problems.
Besides I'm not texturing anything, just coloring, and I really was planning in leaving textures out of the picture here, which presents me with a problem.
But on the other hand, trying to see the big picture here... I could create a texture with each color I want, and use the full optimization code I have for vertexes to be unique, and just sample the texture on top of the corresponding vertex coordinates... hmmm... so that way colors wont mix, even when they are bases on color textures, ... tricky.
this is another pic of the full extent of my map.. fps are low, since I'm looking at all the map at once, normally the camera would be on the player which is on the middle of the map, so the view frustum will take care of the FPS on the screen.
On topic, I'm starting from scratch here, never give up, never surrender, I've already written the terrain generation 11 times using different approaches, to make it look better, faster, etc, the same goes for the light or everything else I need to do, every time I learn something new. This is what it looks with the normal diffuse light + ambient light that I have
[attachment=9460:shapes2.png]
This is what it looks like with the current diffuse + normals + kind of ambient, + ssao. No water here yet...
in my quest to get this working well, I had some bumps, my PSU died, so my new one is on its way, bad luck mostly. I have also made what I would like to call a test bench, were I can use properties to change parameters instead of having to change a number and and the compile again and so on.
I think I have SSAO working... not really entirely sure, and japro, I still want to apply your technique, I kind of made it working, but I have the issue that I'm using plain vertex colors not textures, so I couldn't find how to map the color on my quad with the what would look like a color in a texture... still trying to figure that out, I tried to cheat on the shader making the changes to the pixel color on the xna program instead of making the pixel change on the pixel shader, as I already had all the occlusion information, but xna mixes color very very bad, and so ... it sucks.
I finally got the hangs of using PIX to debug HLSL... what a change it makes... like night and day.
here is a screen of the bench, with and without SSAO. (as I'm using a cheap laptop on the time being that actually shuts down on video card overhead, I just render a very very small portion of the map)
Final render - no SSAO
[attachment=10520:Combined NO SSAO.png]
YAY!!!, so I managed to do that shadowing on concave edges... unfortunatedly XNA and DX9 don't have bitwise operators so I had to improvise a little bit there. And I might still need some adjustments. The other thing is that I had to drop my vertex optimizations due to texcoords. Here is a small shot. Just color and concave shadows.
