I think you need to use color picking. That is, you draw your objects in a unique color for each object. Don't show the result, but read out the color at the screen coordinate of the mouse. That color will then identify what object is behind the mouse.

You will probably need three different modes, for vertices edges and faces. For vertices, it may be an idea to draw small quads instead of dots, making it easier to hit them with the mouse. For edges, you can draw wide lines.

Do you want continuous detection of the object behind the mouse, or only when the user clicks with the mouse? Continuous detection can lead to performance problems.