Just now, Vilem Otte said:
Yes, it should be possible through this:
Nah... it's not right this way. The problem is that the advantage of fixed function hardware vanishes with custom intersection shader:
You have no control what rays process it an a shared CU. You do most work on regular shader cores. You might need to implement your own mini BVH (which is total nonsense at this point because rays have no shared memory) Conclusion: You better do it in just compute.
With compute you can batch all rays that may intersect a patch (or a number of patches), then you could tesselate these patches to LDS, and brute force all rays of the batch over it.
But this is maybe how RTX works under the hood, so what you do is re-implementing fixed function hardware, just because the existing one is too restricted / black boxed.
At the end you'll just pre-tesselate your patches to triangles and the potential advantage is lost.
I have exactly this problem with tracing my GI sample hierarchy, which is geometry, hierarchy and LOD on one data structure. My compute tracing here matches RTX performance - as far as i can compare algorithms with differing time complexity.
I want to use RTX for specular reflections, which is fine. But using my sample hierarchy should work too... strange situation. I'll wait how RT support on upcoming consoles will look like. On the long run i want to use my GI stuff as a fall back for 2nd. bounces with a path tracer - only then i'll join in peace with RTX
... would be very nice if you make a blog post about your RTX experience when you get at it.