If I have two triangles, such that one cuts through another (like if two objects overlap in space), the edge where they meet doesn't look right. It tends to look pixelated or otherwise has a strange zigzag effect. As I move closer to the object, sometimes this effect diminishes. Also, sometimes if I look at an object so that I can see the edge where the side meets the back, I may see a similar effect there. Does anyone know what causes this and how to avoid it?

Welcome in the land of limited depth buffer precision and non-uniform value distribution ;)

What you're experiencing is known as z-fighting - see Wikipedia here: http://en.wikipedia.org/wiki/Z-fighting

So the two triangles have similar depth values in the area of the intersection - there are a couple solutions but it is a very tricky problem.

The best solution is to try to not have the situation in the first place ;)

Then you can also try to have a greater angle between the triangles so their depth values differ more quickly.

Sometimes also using a depthbias is a viable option (e.g. you hint the renderer that the depth value should actually be closer of further from the original value), but itdepends on the situation if this is a usable solution.

Some background on the depth bias can be found for example here in the Microsoft documentation: https://msdn.microsoft.com/en-us/library/windows/desktop/cc308048%28v=vs.85%29.aspx

I suspected it was something like this. But I wouldn't think it would actually look better just because i get closer to the object. Or is that because floating points are floating, and therefore the precision is reduced at a greater range, because of a limited amount of significant digits?

Anyway, couldn't I just increase the resolution of the depth buffer or something? I can't increase the angle between the polygons, because they need to be whatever shape they must be. I don't know anything about a depth bias, but my guess is that in a situation where objects can be moved arbitrarily, I can't make such assumptions necessary to do that.

Edit: Also, in the case where I can see the back through the back/side corner, I would think that would happen a lot, because ALL objects should have such corners! What do people normally do in that case?

Like Buckeye wrote the ratio of your near to far plane is one of the most important tweaking possibilities - since the values are not distributed uniformly over the range you see that the artifact gets better when you get closer.

Out of curiosity: What are your near & far plane values? And are you using a D32F, D24S8 or D16 depth buffer format?

If I recall (I don't have the code here), I'm using a 32-bit buffer, but the depth has a pretty huge range, because I didn't realize it would be problematic. Can't I just use doubles instead of floats? D3D always wants to use floats for everything and it's so antiquated. Aren't there any 64-bit video cards, and if so, would 32-bit be totally incompatible with any code written for 64-bit, or is there some sort of emulation mode?

>You seem to be confusing texture format with the processor bit width. They aren't related. Your shader calculations may all happen in 32-bit, but you can read and write to a texture format of any bit-depth.

No I realize that the the number of bits we were talking about refers to the bus size, but I guess I just assumed that they made the data types and buffer resolutions that way because it matched the bus.

D3D unfortunately doesn't expose any 64 bit data formats (with a single component being wider than 32 bits) yet - so no dice with 64 bit buffers. Also I guess that most depth buffer compression schemes in use by the IHVs won't fit out of the box with 64 bit precision requirements etc. So for now you need to be careful with the ratio of near to far plane I'm afraid :)

All formats (potentially) supported by D3D11 are here: https://msdn.microsoft.com/en-us/library/windows/desktop/bb173059%28v=vs.85%29.aspx