Do you think it's a good idea to use more metrics than just average frame rate when testing render performance? I mean 95th percentile, minimium frame rate, etc.
Do you think it's a good idea to use more metrics than just average frame rate when testing render performance? I mean 95th percentile, minimium frame rate, etc.
Yeah average framerate alone definitely isn't enough. You really want a graph of frame times (don't use FPS, it's non-linear) so that you can clearly see if there's any spikes, fluctuations, or stuttering. A highly variable frame time results in a very bad playing experience, even if the average frame rate is still "high". So a game that's locked at a constant 33.3ms (30fps) will often be perceived as "smoother' than a game that averages close to 16.6ms (60fps) but fluctuates from frame to frame.
Profiling helps identify bottlenecks, when looking to optimize. Profilers tell you the average amount of time a function takes, and even more importantly, the number of times it is called during a specific program run. Saving 10ms on a function that is only called 20 times a frame isn't as good as saving 1 ms on a function that is called two hundred thousand times a frame.
Speaking of profiling and performance tracking, riting your own lightweight profiler is a great way to keep on top of how different bits of your game are performing. You set it to run per-frame and record timings for various bits of code. This is valuable for knowing where to look when you get performance drops during development - you might think the hit may be coming from what you're working on, but another part of the code has been affected.
A good example/tutorial is here:
http://www.rorydriscoll.com/2009/04/21/direct3d-11-multithreading/
I wrap certain functions up (culling, physics update, particles update, game code update, rendering, etc etc) in a profiling event and output a colour coded graph so I can quickly see what's going on. No where near as fine grained as a proper profiler but I feel it's much more useful!