Sign in to follow this  

Unity Ray picking without D3DXIntersect

Recommended Posts

I am having some problems with ray picking in DirectX. I am using it for a 3d map editor; the objects to be selected are not meshes, but simple vertex-defined quads with textures. I cannot fully emulate the pick example in the SDK; while it was useful for demonstrating the principles underlying the process, it is designed for meshes, and uses the D3DXIntersect function which takes a mesh as an argument. I have picked one of the quads as a test object and am trying to get the program to pick that object when the mouse is over it. I am using a plane intersection check, but the intersection vector is always wrong. I am coding it like this. Apologies for the slightly lengthy nature, but might as well make sure all the bases are covered:
POINT ptCursor;

ScreenToClient(g_hWndGalaxy, &ptCursor);
D3DXVECTOR3 vecMouse, vecFar, vecIntersect, vecScreen, vecObject(g_x, g_y, g_z); 

// g_x, etc. are the world matrix    
// coordinates of an object I am using
// as a test object

D3DXMATRIX matWorld, matProj, matView, matViewInv, matProjInv;

D3DVIEWPORT9 vpScreen;
g_pd3dDevice->GetTransform(D3DTS_WORLD, &matWorld);
g_pd3dDevice->GetTransform(D3DTS_VIEW, &matView);
g_pd3dDevice->GetTransform(D3DTS_PROJECTION, &matProj);

vecMouse.x = float (ptCursor.x);
vecMouse.y = float (ptCursor.y);
vecMouse.z = 0.001f;

D3DXVec3Unproject(&vecMouse, &vecMouse, &vpScreen, &matProj, 
                  &matView, &matWorld);
vecMouse.z = 0.99f;

D3DXVec3Unproject(&vecFar, &vecMouse, &vpScreen, &matProj,   
                  &matView, &matWorld);

D3DXPLANE plZplane;
D3DXVECTOR3 v1, v2, v3;
D3DXPlaneFromPoints( &plZplane, 
                       &D3DXVECTOR3( 0, 0, g_z ),
                       &D3DXVECTOR3( 0, -1, g_z ),
                       &D3DXVECTOR3( 1, 0, g_z ) ); 

// It seemed initially that drawing the plane at the z 
// coordinates of the object I'm testing would be best. 
// However, I have a feeling that might be wrong.

D3DXPlaneIntersectLine(&vecIntersect, &plZplane, &vecMouse,     

The problem is very simple: vecIntersect, even though the plane is being drawn at the same Z coordinate of the test object, is always way, way off from the object's actual world coordinates (g_x, g_y, and g_z). I haven't been able to notice any pattern in the discrepancy either, it's just way off. I can't for the life of me figure why. A vector is derived from the mouse position, another vector down the z axis with the same x y coordinates, a plane is drawn at the z coordinate of the test object I'm trying to check, and it checks for an intersection. But so far no dice, which is perplexing because I think the way I've done it is pretty much identical to the plane/intersection method for picking that I've seen in other threads and tutorials. It seems that, for some reason, the mouse coordinates are just not gelling with the world coordinates of the object; it's as if the mouse and the object I'm trying to check are in two different world transforms, but I have checked and made absolutely certain that the world transform is constant. I'm follow the method described by Zaei (I'm assuming from the signature) in the last post on this thread: Any ideas or suggestions? I am probably missing something very obvious and silly. Thanks, sayfadeen [Edited by - sayfadeen on September 21, 2004 4:10:08 PM]

Share this post

Link to post
Share on other sites
One obvious thing I have noticed with this is that x,y coordinates of the untransformed mouse position do not match up with the x,y coordinates of the object which is under it. Even with a perspective view, shouldn't the coordinates match up (I'm just thinking about it visually here. If I'm standing here and facing directly at an object at a distance and we are aligned horizontally, doesn't this by definition mean that my "x coordinate" and its "x coordinate" should be the same?). I have also noticed that the untransformed x,y of vecMouse and vecFar are different from each other, even though the original x,y is the same (ptCursor.x, ptCursor.y). Is this supposed to happen because of the projection/perspective?. If this is supposed to happen, then clearly I'm missing something about how projection works (I'm going to RTFM again anyways). But if this is not supposed to happen, then what to do to scale the untransformed mouse coordinates in such a way that when the mouse cursor is over an object, the mouse x,y will match the object's x,y?

[Edited by - sayfadeen on September 21, 2004 3:01:30 PM]

Share this post

Link to post
Share on other sites
Yeah, you should keep the projected x,y and z coords as you do.
What bothers me, is:


D3DXVec3Unproject(&vecMouse, &vecMouse, &vpScreen, &matProj,
&matView, &matWorld);

vecMouse.z = 0.99f;

D3DXVec3Unproject(&vecFar, &vecMouse, &vpScreen, &matProj,
&matView, &matWorld);

You are overwriting vecMouse in the first call, then you are using it again to compute vecFar. Reverse order of those calls and it shold be "much better" ;)

Also, your plane is perpendicular to XY plane in world coordinates, not in screen coords. What you could do to construct the plane you want, is:

- take a vector(0.0f, 0.0f, 0.5f) in the meaning of screen coords.
- unproject it, then it would represent the "center"(or so) of the original camera frustum.
- substract it from your camera's position, then you will get your camera's direction(or you can just take your camera's direction from somewhere else, that would be much simplier)
- normalize the direction and create the plane from that normal and (g_x, g_y, g_z).

I don't know if I understood your problem right, but hopes it is of some help.

Share this post

Link to post
Share on other sites
Yeah, that helps, and thank you very much for your response. I'm still a bit unclear about a couple points, being quite a newbie at this.

To get the camera position, should I just take the ._41, ._42, ._43 values from the view matrix (or possibly the inverted matrix)? And would I just use D3DXVec3Normalize on the camera position? I've tried it these different ways but no success, so I'm probably doing it quite wrong.

[Edited by - sayfadeen on September 22, 2004 4:47:21 PM]

Share this post

Link to post
Share on other sites

should I just take the ._41, ._42, ._43 values from the view matrix

Just out of curiosity, where did you get your view matrix from? I always construct it from camera's position and it's rotation(which gives me its look direction).
Anyway, yes you inverse the matrix first, so as you move in a direction, it's like all your objects were moving in the opposite. If you rotate left, it's like your world rotated right and you stood still the entire time(it is not _that_ simple, but taking translations and rotations independently, it can be done by hand like that).

Then you could take those values(fourth column). This would be _position_, so you didn't want to normalize this.
Another story with direction. You could also accuire it from inversed view matrix(it's third column, as it determines transformation of your Z axis to your new Z axis, which is, in the end, just where camera looks). You don's need to normalize it, as it is normalized already(if it wasn't your view in z-dir would be somewhat streched).
But much better(and cleaner) it would be if you tracked the way your view matrix is _born_, then you could have all the needed values directly.

Try it out.

PS: I never, ever had to use inversion of a matrix in any of my DX3D programs. Not that I _avoided_ it or so. It is just not necesarry, IMO.

Share this post

Link to post
Share on other sites
Guest Anonymous Poster
Well they both worked! I'm surprised there's two so different ways of going the same thing.

Excellent work, I really appreciate the help. I was starting to despair for a minute there.

Kind regards,

Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this  

  • Announcements

  • Forum Statistics

    • Total Topics
    • Total Posts
  • Similar Content

    • By JM-KinematicSoup
      Unity has put up an official release of EditorXR (formerly EditorVR).
      If you haven't tried it lately, you should. This release is actually very good,
    • By JM-KinematicSoup
      Hello everyone,
      I'm looking for some input. We created a tool called Scene Fusion to enable real-time collaboration while doing gamedev in Unity. In effect, it will replicate any scene changes one person performs for everyone else connected to the session. It works well, and we as well as our customers using it notice a drastic speedup getting work done, especially when building large complex levels.
      One limitation we have is that we can't support every single plugin and asset out there, so what we opted to do is provide an API that allows game developers to do that work as required.
      My question is: How many if you out there actually end up having to modify an external plugin/tool in order to get it to work with your project?
    • By toadvine
      I'd like to start building a 2D game, very small, just pixel art. I know python/javascript/tiny bit of HTML and I've started learning C++. I'm a long way from starting officially, I've just been making mini-projects to practice & drawing up concept art/gameplay storyboards. I'm just wondering if I'm off with the C++ for the game's programming, or if I should know certain things before starting.
      The game would be very small, I just want to get started trying out some actual work before college.
      Some other info: I'm working by myself on this, and going into game design/programming next year for college. I'm a high school student with a few years of experience just working by myself making small projects, drawing up stuff, so I really don't know much and I'd appreciate any tips.
      Thanks a bunch!
    • By Daerst

      SWARMED is a Zombie-themed RPG / RTS currently in development using Unity 3D. We love Dwarf Fortress (though we have no illusions that SWARMED will reach the same level of complexity), roguelikes, old-school point & click RPGs and real-time strategy games. We aim to cross genre-borders here and there and give some twists to the old Martinis every gamer has been drinking since the 1980s, metaphorically.
      Single player, 3D graphics and adjustable top-down camera - old-school RPG / RTS feeling Take control of a core group of survivors after the outbreak Encounter Zombies that are a real threat, no machine-gun massacre. Don't get swarmed! Build a safe zone anywhere with a highly flexible build system: campsite, lighthouse, school, or fence a whole village Grant asylum to other survivors that you meet and make them a part of your community Achieve sustainability in your safe zone and go on supply runs with your survivors
      The core team of recently founded indie studio Three Eyed Games currently consists of one writer, two artists and two programmers, based in Germany. We are in our mid-20s with professional experience in developing interactive 3D applications with Unity.
      SWARMED will feature both a 'free-play mode' and a campaign with mid-sized maps that leads the player through a story while explaining the gameplay and introducing him / her to the survivors: a core group a few 'hero' characters the player starts with (each one a detailed character with backstory, hopes and dreams), and more 'heroes' (total not more than 20, probably less) that the player can meet on the journey. In addition, randomly generated NPCs (less detailed and not directly controllable, similar to the way Dwarf Fortress handles its dwarves) can join your safe zone – if you let them.
      We plan to release a few 'Origin' prototypes that showcase individual gameplay systems and meanwhile give a gentle introduction to the characters you will meet in the game. Origin I, showcasing the build system and many fundamental elements like character controls and interactions, is finished and will be released soon. Next up, we're working on the dialog system to be presented in Origin II. Get in touch and we will provide more details and a playable version.
      We want you!
      We seriously think you should join the fun! We are looking for:
      Level Designers / Environment Artists, preferably with experience in Unity and procedural asset creation. Design and build maps with interesting visuals and proper pacing. 3D Artists. Our shacks, items and the dead guys' faces could use some plastic surgery. Can you do that? Writers. We have a bunch of characters to detail and a story to write ahead of us. Game Designers. We have a rough game design sketched out that needs improvement and completion. We need a balanced combat system, trees for constructions, workshops and character skills etc. PR & Community Managers, preferably with web development experience. We want to build a community around the game, and we need you to plan and manage this (with the help of the rest of the team, of course). 2D Artists / UX Designers, preferably with Unity UI experience. Our menus still look pretty dull, and we don't like that. We also need concept art for characters and iconic game moments to define their look and feel. Coders. If you know your way around Unity and C#, there are lots of challenging things to be done. You will work closely together with the two programmers already on the team to get going quickly. Please drop me a message or contact
    • By sveta_itseez3D
      itSeez3D, a leading developer of mobile 3d scanning software, announced today a new SDK for its automatic 3D avatar generation technology, Avatar SDK for Unity. The Avatar SDK for Unity is a robust plug-n-play toolset which enables developers and creatives to integrate realistic user-generated 3D avatars into their Unity-based applications. SDK users can allow players to create their own avatars in the application or integrate the SDK into their own production processes for character design and animation.
      “Virtual avatars have recently become increasingly popular, especially in sports games and social VR apps. With the advance of VR and AR, the demand to get humans into the digital world is only increasing”, said Victor Erukhimov, itSeez3D CEO. “Our new Avatar SDK for Unity makes it super-easy to bring the avatar technology into any Unity-based game or VR/AR experience. With the Avatar SDK for Unity now every developer can bring face scanning technology into their games and allow players to create their own personalized in-game avatars, making the gameplay much more exciting and immersive.”
      Key features of the Avatar SDK for Unity:
      Automatic generation of a color 3D face model from a single selfie photo in 5-10 seconds (!). Works best with selfies, but can be used with any portrait photo.
      Shape and texture of the head model are unique for each person, synthesized with a deep learning algorithm crafted by computer vision experts
      Head models support runtime blendshape facial animations (45 different expressions)
      Generated 3D heads include eyes, mouth, and teeth
      Algorithms synthesize 3D meshes in mid-poly resolution, ~12k vertices, and ~24k triangles
      Six predefined hairstyles with hair-recoloring feature (many more available on request)
      Avatar generation API can be used in design-time and in run-time, which means you can allow users to create their own avatars in your game
      Cloud version is cross-platform, and offline version currently works on PCs with 64-bit Windows (support for more platforms is coming soon)
      Well-documented samples showcasing the functionality.
      The Avatar SDK for Unity is offered in two modes - “Cloud” and “Offline”. The “Cloud” version is available at and the “Offline” version is available by request at
      About itSeez3D
      At itSeez3D, we are working on the computer vision technology that turns mobile devices into powerful 3D scanners. itSeez3D has developed the world's first mobile 3D scanning application that allows to create high-resolution photorealistic 3D models of people's' faces, bodies and objects. The application is available for iOS and Windows OS mobile devices powered with 3D cameras. In 2016 the company introduced Avatar SDK that creates a realistic 3D model of a face from a single selfie photo. To learn more about itSeez3D scanning software and 3D avatar creation technology, please visit and

      View full story
  • Popular Now