Credit for the concept of Pixel Perfect Picking in D3D11 belongs to gamedev member unbird, a wizard with shaders. However, since I'm basically lazy, I coined the acronym PPP, and adapted the idea to my needs.
N.B., some of the implementations below require SM5.
The PPP Approach
The idea is to pick objects (in my editor that's vertices and faces) with a mouse-click by writing mesh and vertex/face information to a texture during the rendering process; sample a single pixel of the texture at the mouse position; and interpret the components as data, rather than color, providing the needed information. If you need to know what object is under the mouse position, and you're rendering the scene anyway, get the information from the rendering process itself in a single pass.
When the user clicks the mouse to select something (or, actually, any time you want to know what's under the mouse position):
1. Clear and set a second render target view.
2. Set appropriate buffer data and shaders to both render to the primary render target and output the needed information to that second texture (mesh id's, vertex or face id's, etc.) HLSL provides the convenience of SV_VertexID and SV_PrimitiveID which can be passed through.
3. Render the scene, writing the color to SV_Target0 and the data to SV_Target1.
4. Use CopySubresourceRegion to get just the single pixel from the second texture at the mouse position.
5. Examine the pixel to see what (if anything) lies under the mouse position.
It's a Different Approach to an Old Concept
Picking objects by rendering them can be done in OpenGL. I haven't done it in ages, but, IIRC, OGL has the option to set the render mode to GL_SELECTION. You then get a list of object hits in the view volume by rendering. PPP works similarly but tests for an object hit at the pixel under the mouse-click position in a single pass.
The Need for a Picking Algorithm
A feature I wanted to incorporate in my mesh editor for picking or selecting vertices and faces is the option to limit selection to only visible vertices or faces. That is, "visible" in the sense of "appearing in the current view." It's one of the options Blender implements and I use it frequently when editing meshes. In general, it's more natural when selecting vertices or faces with a mouse-click to expect that the selection will include only the verts/faces I can see onscreen. Before I was mindful of the option in Blender, I would click a face or vertex and later find that hidden vertices or faces (verts or faces at a greater depth than the intended selection) were also selected. Editing what I assumed was only affecting the verts/faces I could see, was, in fact, altering stuff I hadn't intended.
A common method for picking verts/faces with a mouse-click is to do a raycast into the scene from the mouse position, search through verts/faces for hits, and select the object with the smallest hit distance. That picking algorithm can be extended to a list of hits with depth information (hit distance) if desired. That algorithm can be made more efficient with broad-phase culling (choose your favorite method), but still requires setting up the raycast and searching through data to find hits.
However, as with the OpenGL GL_SELECTION method, the very act of rendering does that depth testing (if depth testing is enabled). In particular, if the scene is being rendered anyway, why not use the rendering process to get the needed information, and skip the raycasting process altogether?
For my particular needs in a mesh editor, I have two picking/selection algorithms available - one using geometry shader stream-out to generate a list of visible objects for box selection; the other using the PPP method for mouse picking.
In addition, to give the user a reasonable target for clicking, I display the vertices as billboarded colored quads (say 3x3 or 4x4 pixels), with color indicating whether the vert is currently selected or not, with depth testing disabled so the full quad overdraws adjacent geometry. To do that with depth testing disabled requires culling using a depth map (I use a method similar to shadow mapping) to determine whether the quad is visible and should be rendered, or is not visible and should be discarded. That can be done in the pixel shader, or in a geometry shader. I do it in a geometry shader which can also be used, as mentioned, with stream-out to generate a list of visible objects. Maybe more on that in a later blog entry.