Advertisement Jump to content
Sign in to follow this  

DX11 D3D11 - Pretransformed vertices? (GUI)

This topic is 937 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I'm currently implementing a DX11 renderer for my engine.
I have regular 3d stuff going, vertex buffers, texturing, vertex and pixel shaders all forced to do my bidding. I'm a bit struggling with 2d (known as pretransformed vertices in DX9).
My goal is to provide coordinates in screen pixel values.
As I couldn't find a proper analogy I simply used an off center ortho matrix and let the vertex shader only do that one transformation. This works fine for full screen viewports.
However in my GUI code I'm using viewports heavily to clip 2d output. With my ortho matrix approach it seems that after the vertex shader there's still viewport scaling applied to the result.
What is the proper way to do pretransformed vertices, that still allow clipping, in screen space?
Here's an image explaining a bit better I hope:
The top shows the "working" case, black being the screen, green the set viewport extents.
The lower part shows what I want to achieve. The top right what appears to happen (and what I don't want)
For clarification, my current "pretransforming" vertex shader looks like this:
// A constant buffer that stores the three basic column-major matrices for composing geometry.
cbuffer ModelViewProjectionConstantBuffer : register( b0 )
  matrix model;
  matrix view;
  matrix projection;
  matrix ortho2d;
// Per-vertex data used as input to the vertex shader.
struct VertexShaderInput
 float3 pos : POSITION;
 float4 color : COLOR0;
  float2 tex : TEXCOORD0;
// Per-pixel color data passed through the pixel shader.
struct PixelShaderInput
 float4 pos : SV_POSITION;
  float2 tex : TEXCOORD0;
 float4 color : COLOR0;
// Simple shader to do vertex processing on the GPU.
PixelShaderInput main(VertexShaderInput input)
 PixelShaderInput output;
  float4 pos = float4( input.pos, 1.0f );
  output.pos = mul( pos, ortho2d );
  // pass texture coords
  output.tex = input.tex;

 // Pass the color through without modification.
 output.color = input.color;
 return output;
? Edited by Endurion

Share this post

Link to post
Share on other sites

Yup, changing the viewport needs adjusted transformations, as Hodgman shows.


You could use scissors (ID3D11DeviceContext::RSSetScissorRects) for axis aligned clipping and keep the viewport as is. Probably the simplest to implement.


For completeness: There are other ways to clip:

- Look into SV_ClipDistance semantic (VS output)

- Similarily: Manually clip in the pixel shader with clip/discard

- For rectangles you could "clip" in the vertex shader (rect intersection calculation, adjusting tex coords, too) (*)

- Hint: SV_Position as input to the pixel shader delivers pixel coordinates (with a 0.5 offset), this can come in handy.


Edit: (*) Can be even a good idea to do this CPU-side. Instead of lots of pipeline state changes, one can batch everything.

Share this post

Link to post
Share on other sites

Thanks both of you!


Adjusting the ortho matrix to the new viewport did the trick, thanks a lot :)


Onward I stumble towards the next problem...  generic shaders are tough to do.

Share this post

Link to post
Share on other sites
Sign in to follow this  

  • Advertisement

Important Information

By using, you agree to our community Guidelines, Terms of Use, and Privacy Policy. is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!