Ok, guys (and girls), long post ahead. I experimented and that's what I got. First my setup - I don't duplicate vertices, because, well, that's what indexing is all about ;-)
Vertex buffer:
0: {X:-0.5, Y:-0.5} quad vertices
1: {X:-0.5, Y: 0.5}
2: {X: 0.5, Y: 0.5}
3: {X: 0.5, Y:-0.5}
4: {X: 0 , Y: 0.7} "the" triangle vertices
5: {X: 0.7, Y:-0.7}
6: {X:-0.7, Y:-0.7}
Index Buffer:
0: 0 quads 1st triangle
1: 1
2: 2
3: 2 quads 2nd triangle
4: 3
5: 0
6: 4 "the" triangles triangle
7: 5
8: 6
Position offsets and color come from the instance stream, but I will concentrate on the geometry now:
First, drawing all geometry. Not your intention, just to check if the overall setup is fine.
Ignore that this is C#/SlimDX code, the parameters are the same:
device.DrawIndexedPrimitives(PrimitiveType.TriangleList, 0, 0, 7, 0, 3);
Drawing just the quads:
device.DrawIndexedPrimitives(PrimitiveType.TriangleList, 0, 0, 4, 0, 2); // (A)
Drawing just the triangles:
device.DrawIndexedPrimitives(PrimitiveType.TriangleList, 0, 4, 3, 6, 1); // (B)
To elaborate:
In (A) we don't need more than the first four vertices, NumVertices = 4 instead of 7.
In (B) we select a subset for "the" triangle, namely vertcies 4,5,6 (so MinIndex = 4 and NumVertices = 3), the StartIndex is now 6.
An alternative I've found is this:
device.DrawIndexedPrimitives(PrimitiveType.TriangleList, 4, 0, 3, 6, 1);
BUT then the indices for "the" triangle have to be
...
6: 0
7: 1
8: 2
... since a BaseVertexIndex of 4 offsets them by 4.
You can also select all vertices:
device.DrawIndexedPrimitives(PrimitiveType.TriangleList, 0, 0, 7, 6, 1);
On my system, I can even do a:
device.DrawIndexedPrimitives(PrimitiveType.TriangleList, 0, 0, 0, 6, 1);
I don't even get a fail. The driver obviously ignores the vertex range altogether. But neither do PIX (index out of range) nor the reference device (nothing gets drawn). Also the debug runtimes should complain if fully enabled.
("Should", because I currently have yet to discover why they sometimes don't. I just recently switched to Win7 and VS 2010 Express. Sometimes they show up in the VS output, sometimes in DebugView (using C#/SlimDX), but mostly they don't, even if I start my app NOT from VS. As said, dunno why yet, so PIX is my reference here).
Conclusion: It is possible - and I would have been surprised if it wasn't: as if we were the first to get instanced subsets on a 10-year-old API ;-)
For why you get your behavior on the other hand I can still only guess: I don't see you check the result codes, so maybe the device is in some erroneous state after the first (failing) call which lets your second call fail. Do check them, enable full debug runtimes, run it through PIX, use the reference device... and then get the numbers right ;-)
And, yeah, you don't need to reset the streams or the frequencies (I didn't in my setup).
You could try - if you haven't already - this:
d3dDevice->DrawIndexedPrimitive(D3DPT_TRIANGLELIST,0,0,9,6,1); // full range
or
d3dDevice->DrawIndexedPrimitive(D3DPT_TRIANGLELIST,0,6,3,6,1); // subrange
if I got your numbers right. If you still can't figure out on your own, attach a minimal project and I'll take a look.
@iedoc: Using the "rich" editor ? Oh yeah, it's really bad. I disabled it pretty soon (can be done in your profile settings) and used tags again.
