[hlsl dx10]can't convert 23-bit int correctly
Author
[dx10, hlsl, vista, gpgpu]
Hi,
for a int x (x in [0, MAX)), I first pass (float)x/MAX into the layout
{ "POSITION", 0, DXGI_FORMAT_R32G32_FLOAT, 0, 0, D3D10_INPUT_PER_VERTEX_DATA, 0 },
and in vs
void vs(in float4 vert: POSITION, out float4 pos: SV_Position)
{
pos = vert * MAX / _screenSize;
}
I let x be (0,0),(1,1),(2,2),(3,3) and expect 4 points in a line on screen.
for MAX=4, i'm right.
for MAX=1<<23, which i think should be correctly converted between int and float, but they all overlap to 1 point in (0,0).
why's that? i'm dying. thanks!
1 << 23 puts a 1 in the 24th bit. Floats only have 23 bits of precision. Once you start doing calculations on them you will also lose some of that precision.
http://en.wikipedia.org/wiki/IEEE_floating-point_standard
http://en.wikipedia.org/wiki/IEEE_floating-point_standard
In addition to Adam_42's comment, you realise you have a mismatch between your IA<->VS linkage? Widening is not an error, but your VS should really be float2 rather than float4...
Also, a little more whitespace in your post might be beneficial - I had to read your question a couple of times before I picked up all the details [smile].
Jack
Also, a little more whitespace in your post might be beneficial - I had to read your question a couple of times before I picked up all the details [smile].
Jack
Author
Thanks Adam_42, 1<<20... all lead to the same result.
Thanks Jack, if I use
instead of float4 in VS, then i can't get 4 points in a line, but a overlapped point.
btw, i can't find reference saying that we should only use "float" rather than "int" as vs's positions, like
[Edited by - yk_cadcg on September 26, 2007 7:21:02 AM]
Thanks Jack, if I use
in float2 vert: POSITION, ..pos = float4(vert, 0, 0);instead of float4 in VS, then i can't get 4 points in a line, but a overlapped point.
btw, i can't find reference saying that we should only use "float" rather than "int" as vs's positions, like
in int4 vert: POSITION, Quote:Original post by jollyjeffers
In addition to Adam_42's comment, you realise you have a mismatch between your IA<->VS linkage? Widening is not an error, but your VS should really be float2 rather than float4...
Also, a little more whitespace in your post might be beneficial - I had to read your question a couple of times before I picked up all the details [smile].
Jack
[Edited by - yk_cadcg on September 26, 2007 7:21:02 AM]
Shader Semantics (DirectX HLSL) -> The 'System-Value Semantics' section lists SV_POSITION (==POSITION) as float4.
I'm not sure about this:
pos = float4(vert, 0, 0);
But try:
pos = float4(vert.x, vert.y, 0, 0);
instead. I'm not sure there is a (float2,float,float) overload to the float4 constructor - I thought it was only (float,float,float,float) or (float3,float). Could well be your odd results are due to some sort of silent casting/promotion error...
hth
Jack
I'm not sure about this:
pos = float4(vert, 0, 0);
But try:
pos = float4(vert.x, vert.y, 0, 0);
instead. I'm not sure there is a (float2,float,float) overload to the float4 constructor - I thought it was only (float,float,float,float) or (float3,float). Could well be your odd results are due to some sort of silent casting/promotion error...
hth
Jack
Author
thanks a lot!
i tried your suggestion and the same overlappe result.
i let color = pos in ps, staged rendertarget, and find that if we use your mentioned "widening" (IA is float2 but vs input is float4), then the final output pos is made up with "0, 1": (x, y, 0, 1).
this means, system default widening is "0,1". In light of this, i tried
pos = float4(vert.x, vert.y, 0, 1);
or
pos = float4(vert, 0, 1);
both get what i want (4 points in a line).
so i recalled there's a /w operation after vs. Digged out sdk entry "Rasterizer Stage (Direct3D 10)":
thanks a lot!
col = (int4)(pos.w, 255,255,255);
i tried your suggestion and the same overlappe result.
i let color = pos in ps, staged rendertarget, and find that if we use your mentioned "widening" (IA is float2 but vs input is float4), then the final output pos is made up with "0, 1": (x, y, 0, 1).
this means, system default widening is "0,1". In light of this, i tried
pos = float4(vert.x, vert.y, 0, 1);
or
pos = float4(vert, 0, 1);
both get what i want (4 points in a line).
so i recalled there's a /w operation after vs. Digged out sdk entry "Rasterizer Stage (Direct3D 10)":
Quote:the rasterizer always performs clipping, a perspective divide to transform the points into homogenous space, and maps the vertices to the viewport.
Vertices (x,y,z,w), coming into the rasterizer stage are assumed to be in homogenous clip-space. In this coordinate space the X axis points right, Y points up and Z points away from camera.
thanks a lot!
col = (int4)(pos.w, 255,255,255);
Quote:Original post by jollyjeffers
Shader Semantics (DirectX HLSL) -> The 'System-Value Semantics' section lists SV_POSITION (==POSITION) as float4.
I'm not sure about this:
pos = float4(vert, 0, 0);
But try:
pos = float4(vert.x, vert.y, 0, 0);
instead. I'm not sure there is a (float2,float,float) overload to the float4 constructor - I thought it was only (float,float,float,float) or (float3,float). Could well be your odd results are due to some sort of silent casting/promotion error...
hth
Jack
Author
thanks a lot!
i tried your suggestion and the same overlappe result.
i let color = pos in ps, staged rendertarget, and find that if we use your mentioned "widening" (IA is float2 but vs input is float4), then the final output pos is made up with "0, 1": (x, y, 0, 1).
this means, system default widening is "0,1". In light of this, i tried
pos = float4(vert.x, vert.y, 0, 1);
or
pos = float4(vert, 0, 1);
both get what i want (4 points in a line).
so i recalled there's a /w operation after vs. Digged out sdk, found the pipeline architecture diagram, found that vs->(gs)->raster->ps, then found the entry "Rasterizer Stage (Direct3D 10)":
thanks a lot!
i tried your suggestion and the same overlappe result.
i let color = pos in ps, staged rendertarget, and find that if we use your mentioned "widening" (IA is float2 but vs input is float4), then the final output pos is made up with "0, 1": (x, y, 0, 1).
this means, system default widening is "0,1". In light of this, i tried
pos = float4(vert.x, vert.y, 0, 1);
or
pos = float4(vert, 0, 1);
both get what i want (4 points in a line).
so i recalled there's a /w operation after vs. Digged out sdk, found the pipeline architecture diagram, found that vs->(gs)->raster->ps, then found the entry "Rasterizer Stage (Direct3D 10)":
Quote:the rasterizer always performs clipping, a perspective divide to transform the points into homogenous space, and maps the vertices to the viewport.
Vertices (x,y,z,w), coming into the rasterizer stage are assumed to be in homogenous clip-space. In this coordinate space the X axis points right, Y points up and Z points away from camera.
thanks a lot!
Quote:Original post by jollyjeffers
Shader Semantics (DirectX HLSL) -> The 'System-Value Semantics' section lists SV_POSITION (==POSITION) as float4.
I'm not sure about this:
pos = float4(vert, 0, 0);
But try:
pos = float4(vert.x, vert.y, 0, 0);
instead. I'm not sure there is a (float2,float,float) overload to the float4 constructor - I thought it was only (float,float,float,float) or (float3,float). Could well be your odd results are due to some sort of silent casting/promotion error...
hth
Jack
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
