Jump to content

  • Log In with Google      Sign In   
  • Create Account

We're offering banner ads on our site from just $5!

1. Details HERE. 2. GDNet+ Subscriptions HERE. 3. Ad upload HERE.


Problem with vignette shader on PC


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
17 replies to this topic

#1 DarkRonin   Members   -  Reputation: 614

Like
0Likes
Like

Posted 03 April 2014 - 10:59 PM

Hi Guys,

 

I am having a problem witha vignette shader in Game Maker: Studio. The shader works perfectly in Codea on the iPad, but I don't get any fall-off in GM:S. All I get is a slightly darker screen.

 

I made the background whit to ease troubleshooting and the background turns to a perfectly even grey (checked in photoshop). So, I know there is not fall-off. It is like smoothstep() isn't happening on the PC.

 

I have even added a sepia effect to verify that the shader is actually being applied. So with that addition I get an even sepia screen.

 

// Fragment shader
 
//uniform sampler2D u_texture;
varying vec4 v_vColor;          //"in" attributes from our vertex shader
varying vec2 v_vTexcoord;
const float RADIUS=0.5;
const float SOFTNESS=0.45;
uniform vec2 resolution;
void main()
{
    vec4 texColor = texture2D(gm_BaseTexture,v_vTexcoord);
    // *** VIGNETTE ***
    vec2 pos = (gl_FragCoord.xy/resolution.xy)-vec2(0.25);
    float len = length(pos);
    float vignette = smoothstep(RADIUS, RADIUS-SOFTNESS, len);
    texColor.rgb = mix(texColor.rgb, texColor.rgb * vignette, 0.5);
   
    // *** SEPIA ***
    vec3 lum = vec3(0.299, 0.587, 0.114);
    float gray = dot(texColor.xyz, vec3(0.299, 0.587, 0.114));
    texColor = vec4(gray, gray, gray, texColor.w);
    texColor.rgb *= vec3(1.15,1.1,0.8);
   
    gl_FragColor = texColor * v_vColor;
}

 

Any help on this would be awesome. I am completely out of ideas.



Sponsor:

#2 Ashaman73   Crossbones+   -  Reputation: 7872

Like
1Likes
Like

Posted 03 April 2014 - 11:34 PM


vec2 pos = (gl_FragCoord.xy/resolution.xy)-vec2(0.25);

0.5 instead of 0.25 ?



#3 DarkRonin   Members   -  Reputation: 614

Like
0Likes
Like

Posted 04 April 2014 - 01:22 AM

 


vec2 pos = (gl_FragCoord.xy/resolution.xy)-vec2(0.25);

0.5 instead of 0.25 ?

 

 

Just tried 0.5 as well, but absolutely not difference, unfortunately.

 

The whole screen is just 147,140,102 (RGB) - sepia. Instead of sepia falling off to dark sepia (which my Codea project correctly does on the iPad).

 

This is what my shader was derived from - https://github.com/mattdesl/lwjgl-basics/wiki/ShaderLesson3



#4 Stainless   Members   -  Reputation: 1043

Like
1Likes
Like

Posted 04 April 2014 - 02:18 AM

It may be that you have transformed the coordinates.

 

When displayed the pixel coordinates are in the range -1 to 1, you are assuming 0-resolution

 

I would try changing this

vec2 position = (gl_FragCoord.xy / resolution.xy) - vec2(0.5);

to this

   vec2 position = (gl_FragCoord.xy  * 0.5f);

It is worth a try at least



#5 haegarr   Crossbones+   -  Reputation: 4438

Like
1Likes
Like

Posted 04 April 2014 - 02:22 AM

The GLSL specifications say:

    genType smoothstep (genType edge0, genType edge1, genType x)
    ...
    Results are undefined if edge0 >= edge1.

 

In your case edge0 = 0.5 and edge1 = 0.5 - 0.45 = 0.05, so you get a undefined result! Even it it works on one platform, the result may be totally different on another platform, and so happens here. Try to exchange the arguments RADIUS and RADIUS-SOFTNESS when invoking smoothstep.


Edited by haegarr, 04 April 2014 - 02:40 AM.


#6 haegarr   Crossbones+   -  Reputation: 4438

Like
1Likes
Like

Posted 04 April 2014 - 02:37 AM


When displayed the pixel coordinates are in the range -1 to 1, you are assuming 0-resolution

The x and y values of gl_FragCoord are window relative co-ordnates. The bottom left pixel has co-ordinates (0.5,0.5) and the upper right has (w-0.5,h-0.5) where (w,h) denotes the resolution in pixels. The range [-1,+1] instead is in normalized device co-ordinates (NDC).

 


Just tried 0.5 as well, but absolutely not difference, unfortunately.

That is okay w.r.t. the fact that the problem is probably based in the use of smoothstep, but Ashaman73 is right: A symmetric effect would require the subtraction of (0.5,0.5) there.



#7 Ashaman73   Crossbones+   -  Reputation: 7872

Like
1Likes
Like

Posted 04 April 2014 - 02:44 AM


In your case edge0 = 0.5 and edge1 = 0.5 - 0.45 = 0.05, so you get a undefined result! Even it it works on one platform, the result may be totally different on another platform, and so happens here.

Try to negate it:

    float vignette = smoothstep(-RADIUS, -(RADIUS-SOFTNESS), -len);



#8 DarkRonin   Members   -  Reputation: 614

Like
0Likes
Like

Posted 04 April 2014 - 03:36 AM

Thanks Guys, lots of replies here so I'll go through the suggestions one by one smile.png

 

Stainless - I just tried vec2 position = (gl_FragCoord.xy * 0.5f); No joy, I am afraid

haegarr - I just tried float vignette=smoothstep(RADIUS-SOFTNESS,RADIUS,len); so then egde0<edge1, still no good.

Ashaman73 - Tried float vignette = smoothstep(-RADIUS, -(RADIUS-SOFTNESS), -len); But still no luck sad.png

 

I am told that GameMaker user GLES, does this make a difference? All compiles without a problem though.

 

There is a slight difference in the vertex shaders though.

 

Although I don't know enough about shaders to know what is going on.

 

From the working shader on Codea

 

uniform mat4 modelViewProjection;
attribute vec4 position;
attribute vec4 color;
attribute vec2 texCoord;
 
varying highp vec2 v_vTexCoord;
varying lowp vec4 v_vColor;
 
void main()
{
    //Pass the mesh color to the fragment shader
    v_vColor = color;
    v_vTexCoord = texCoord;
    //Multiply the vertex position by our combined transform
    gl_Position = modelViewProjection * position;
}

 

and the non-working Game Maker shader

 

attribute vec4 in_Position;
attribute vec4 in_Colour;
attribute vec2 in_TextureCoord;
varying vec2 v_vTexCoord;
varying vec4 v_vColor;
void main()
{
    vec4 object_space_pos=vec4(in_Position.x,in_Position.y,in_Position.z,1.0);
    //Pass the mesh color to the fragment shader
    v_vColor = in_Colour;
    v_vTexCoord = in_TextureCoord;
 
    //Multiply the vertex position by our combined transform
    gl_Position =gm_Matrices[MATRIX_WORLD_VIEW_PROJECTION]* object_space_pos;
}

 

GM has a few of its own variables like in_Position etc. I am a bit dubios of the last line though as it was copied from another shader.

 

Any direction would be awesome!


Edited by lonewolff, 04 April 2014 - 04:05 AM.


#9 haegarr   Crossbones+   -  Reputation: 4438

Like
1Likes
Like

Posted 04 April 2014 - 04:11 AM

...

But the above code breaks the shader on the iPad (Codea) as well. So, obviously something not right with it.
...

"Breaks the shader" is somewhat unspecific. However, have you checked that the uniform resolution is up to date?

 

 

What do you see if you debug output the shape of the vignette like so: 

// Fragment shader

//uniform sampler2D u_texture;
varying vec4 v_vColor;          //"in" attributes from our vertex shader
varying vec2 v_vTexcoord;
const float RADIUS=0.3; // <<< changed here for debug output
const float SOFTNESS=0.1; // <<< changed here for debug output
uniform vec2 resolution;
void main()
{
    vec4 texColor = texture2D(gm_BaseTexture,v_vTexcoord);
    // *** VIGNETTE ***
    vec2 pos = (gl_FragCoord.xy/resolution.xy)-vec2(0.5); // <<< changed here for symmetry
    float len = length(pos);
    float vignette = 1.0 - smoothstep(RADIUS-SOFTNESS, RADIUS, len); // <<< changed here for correctness with spec.
//    texColor.rgb = mix(texColor.rgb, texColor.rgb * vignette, 0.5);
    texColor.rgb = vec3(vignette, vignette, vignette); // <<< debug output
    gl_FragColor = texColor;
}
 

If I made no mistake, the shader should show you an inner white disk, a relatively sharp white-to-black transition, and a black surrounding.


Edited by haegarr, 04 April 2014 - 04:12 AM.


#10 DarkRonin   Members   -  Reputation: 614

Like
0Likes
Like

Posted 04 April 2014 - 04:26 AM

Thanks again haegarr for your help.

 

With your code I get a completely black screen. If I swap the smoothstep() function back to the old one I had, we the get a completely white screen.

 

A bit frustrating - LOL.

 

Even at the point where I am now downloading the latest driver for my video card.

 

[edit]

Nope - drivers didn't give a miracle fix, unfortunately.


Edited by lonewolff, 04 April 2014 - 04:56 AM.


#11 TheComet   Crossbones+   -  Reputation: 1623

Like
1Likes
Like

Posted 04 April 2014 - 06:03 AM

Maybe try using gl_TexCoord[0].xy instead of gl_FragCoord.xy. I remember having a similar issue where gl_FragCoord would return the correct value in GLES, but in GL it would always be 0.


YOUR_OPINION >/dev/null

#12 DarkRonin   Members   -  Reputation: 614

Like
0Likes
Like

Posted 04 April 2014 - 03:32 PM

Maybe try using gl_TexCoord[0].xy instead of gl_FragCoord.xy. I remember having a similar issue where gl_FragCoord would return the correct value in GLES, but in GL it would always be 0.

 

Thanks for the suggestion.

 

I just tried that but it says that it is an undeclared identifier sad.png

 

Just saw this in the GLES manual. Does this mean it was removed from GLSL ES?

 

10.17 Unsized Array Declarations

Desktop GLSL allows arrays to be declared without a size and these can then be accessed with constant

integral expressions. The size never needs to be declared. This was to support gl_Texcoord e.g.

varying vec4 gl_TexCoord[];

...

gl_FragColor = texture2D (tex, gl_TexCoord[0].xy);

This allows gl_TexCoord to be used without having to declare the number of texture units.

gl_TexCoord is part of the fixed functionality so unsized arrays should be removed for GLSL ES

RESOLUTION: Remove unsized array declarations.


Edited by lonewolff, 04 April 2014 - 03:37 PM.


#13 DarkRonin   Members   -  Reputation: 614

Like
0Likes
Like

Posted 04 April 2014 - 04:15 PM

Getting pretty close now. Had a suggestion on another forum and this is pretty close to working.
 
 
precision highp float;   
uniform vec2 resolution;
varying vec4 v_vColor;
varying vec2 v_vTexCoord;
   
const float RADIUS=0.75;        // 0.75 is the good setting
const float SOFTNESS=0.45;
 
void main()
{
    vec4 texColor=texture2D(gm_BaseTexture,v_vTexCoord);
   
    vec2 center=vec2(0.5,0.5);
    vec2 aspect_center=vec2(0.0,0.0);
    aspect_center.x=(v_vTexCoord.x-center.x)*(resolution.x/resolution.y);
    aspect_center.y=v_vTexCoord.y-center.y;
    float len=length(aspect_center)*2.0;
       
    float vignette=smoothstep(RADIUS,RADIUS-SOFTNESS,len);  //use smoothstep to create a smooth vignette
    texColor.rgb=mix(texColor.rgb,texColor.rgb*vignette,0.50); //apply the vignette with 50% opacity
   
    gl_FragColor=texColor*v_vColor;
}
 
The only thing now is that it is positioned a bit too far across and down the screen. And I dont have the nice oval shape I had in Codea anymore. The Codea shader was centred and the vignette edge had an even amount of pixels from top, bottom, left, & right (oval shaped).
 
Codea original example (ignore the harsh black/white settings)
 
codea01.png

Current image from withing GameMaker (using above code). More the position and shape that I am looking to correct.
 
shader00.png

Thanks again for everyone's awesome help!

[edit]
I did some more testing. I think (using this method) it is positioning the vignette at the center of the texture page. As, the centre of the vignette is positioned at 1024, 512 (texture page would likely be 2048,1024 in this case).

[edit2]
Taking screen resolution 1600 x 900 and page size 2048 x 1024 into account, I have worked out that the centre should be
vec2 center=vec2(0.390625,0.439453125);

Edited by lonewolff, 04 April 2014 - 05:28 PM.


#14 haegarr   Crossbones+   -  Reputation: 4438

Like
1Likes
Like

Posted 05 April 2014 - 02:34 AM


Had a suggestion on another forum and this is pretty close to working.

So, what is wrong with using gl_FragCoord in the end? I've used it in a test implementation on a MacBook and it worked well. You also mentioned it is working on an iPad. And from a logical point of view it should work anyway...

 


The only thing now is that it is positioned a bit too far across and down the screen.

The shader script shown in the post above expects a full-screen quad rendered with texture co-ordinates [0,1] from left to right and [0,1] from bottom to top. If you supply the drawing with such a quad, then there should be no need to adapt the center variable.

 


And I dont have the nice oval shape I had in Codea anymore.

This is due to the aspect correction factor in the shader code. If you want the oval back, then remove the stroked part in the following line:

       aspect_center.= (v_vTexCoord.x-center.x* (resolution.x/resolution.y) ;



#15 DarkRonin   Members   -  Reputation: 614

Like
0Likes
Like

Posted 05 April 2014 - 05:30 PM

So, what is wrong with using gl_FragCoord in the end? I've used it in a test implementation on a MacBook and it worked well. You also mentioned it is working on an iPad. And from a logical point of view it should work anyway...


No idea why that doesn't work. Maybe it is a Game Maker: Studio bug or something. As GML is an 'interpreted' language and is based on DirectX9 when running under windows and GLSL ES is OpenGL based. So, maybe something is being lost in the translation?

#16 DarkRonin   Members   -  Reputation: 614

Like
0Likes
Like

Posted 07 April 2014 - 04:29 AM

Turns out that 'straight from the horses mouth', GLSL ES is not fully implemented in Game Maker: Studio.

This explains why things are inconsistent. Turns out my shader was right but GM, well, isn't.

gl_FragCoord, while is compiles ok in GM has been removed from GM's interpretation on GLSL ES in favour of their own 'broken' implementation. So, using their own implementation is returning coordinates for the entire texture page not just the texture.

Scary stuff...

I hate chopping and changing between engines, but I am getting to the point to where I figure I am safer with my own C++ framework.

I just want to get this project into the next IGF but, geez, even with another 6-8 months until submissions, the way things are going, I'll run out of time...

#17 TheComet   Crossbones+   -  Reputation: 1623

Like
1Likes
Like

Posted 07 April 2014 - 05:17 AM

Good job on finding the core of the issue. I hope all goes well!


YOUR_OPINION >/dev/null

#18 DarkRonin   Members   -  Reputation: 614

Like
0Likes
Like

Posted 07 April 2014 - 05:35 AM

Thanks for that.

Now trying to convince the GM devs that their shader support is broken.

Ah, us devs love challenges don't we? LOL

And thanks to everyone who helped out in this topic. It must have been pretty frustrating for you guys as well I'd imagine smile.png




Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS