Jump to content
  • Advertisement
Sign in to follow this  
jollyjeffers

OpenGL Unit Testing / TDD and real-time computer graphics

This topic is 3884 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Afternoon all, As I'd hope others here are, I'm a fan of good engineering practice. I've seen too much pain from cowboy coding in my time [grin] However I can't quite get into the mindset for TDD or unit-testing with real-time computer graphics applications (e.g. OpenGL or Direct3D stuff). From my online research it seems that for all the TDD or general testing guidelines out there graphics is really not covered in any sufficient detail. I've great respect for the regulars in this forum, so figured I'd come here on a fishing trip for any experience/comments on both the practical and conceptual side of testing graphics apps [wink] A few thoughts to get started:
  1. Real-world comparisons It's quite common in BRDF research papers for the authors to compare real-world measured results to computer simulated ones. For example Greg Ward's 1992 paper "Measuring and Modelling Anisotropic Reflection" details such a device. There is also the classic Cornell Box test-data as well. These tests work ok if you have the reference images (or the means to generate them) and are also striving for real-world realism (e.g. cel shading would be hard to test with this!).
  2. Standard unit-testing of an API It wouldn't be too hard to apply TDD or unit testing philosophies to normal D3D/OGL development. Testing HRESULT's and so on... but that'll only really test the correctness of your API usage and won't actually verify that the result is correct. You could quite easily write a completely correct D3D app that draws nothing or even utter garbage on the screen... You could probably write-up a test harness over an HLSL function that iterates over all inputs, but again you're probably stuck for reference test data. You could quite possibly go for known success/failure and fringe cases though.
  3. Testing against the reference rasterizer More of a sanity check than anything as you'd really be testing the driver/hardware implementation rather than any code you'd written.
Have I missed anything obvious in this list? Are there any other legitimate test strategies? If it wasn't clear already, I'm thinking more about the correctness of the image produced rather than the means by which it is created (e.g. legal/correct API usage). So if I were implementing a new BRDF, how could I write up some automated tests to verify that it was correct? How about a new post-processing effect - might be easier, but still doesn't seem to be covered in the test literature... All thoughts/comments welcome [smile] Jack

Share this post


Link to post
Share on other sites
Advertisement
I'd like to use more automated testing in my graphics programming too. some vague suggestions, ideas rather than experience:

tesing (image diff) against a reference image. This is useful when you already have the look correct, but need to optimise performance. You might be willing to live with say a maximum 25% overall pixel value difference. Obviously the allowable difference and the type of difference that's important to you depend on what you're doing. You may decide not to tolerate differences in shape (comparison of output images with an edge detect filter applied? comparison of simple white-on-black renders?), but to tolerate a certain amount of colour difference for example. IIRC you've posted images in your journal which show this kind of testing.

An approach that can be useful in OpenGL, and I'd guess would apply to D3D to some extent, is to use glGet* to verify that the state machine is in the state you expect. This is something gDEBugger is very useful for, but I'd consider doing it at runtime in debug mode as a class or function invariant.

I guess as graphics programming essentially boils down to numeric calculations, the main approach is to do testing on the numeric level rather than on the visual output.

Sorry, not much to add.

Share this post


Link to post
Share on other sites
Quote:
I'd like to use more automated testing in my graphics programming too
Good to hear its not just me [grin]

Quote:
tesing (image diff) against a reference image. This is useful when you already have the look correct, but need to optimise performance
I hadn't considered this aspect of testing, its is a good one!

Quote:
IIRC you've posted images in your journal which show this kind of testing.
Yup, I did exactly that. Although I didn't explicitly consider it to be a test at the time [lol]

Quote:
use glGet* to verify that the state machine is in the state you expect
Yup, that sort of thing can be a powerful test. I see quite a few threads in the DirectX forum where people think their state is correct but inspection via PIX reveals it isn't.

However I'd argue that that approach is far more suited to the old-style fixed function pipeline where you could express everything via the state machine. Now that we've moved to shaders it becomes much harder as, within reason, you haven't a clue what that shader might do with the pipeline state...

Quote:
I guess as graphics programming essentially boils down to numeric calculations, the main approach is to do testing on the numeric level rather than on the visual output.
True, so I wonder if launching HLSL code via some sort of GS+SO harness and effectively iterating over the range of inputs (e.g. for a BRDF you could have a 2D array of 'view' and 'light' vectors and the output can be compared against a set of results from MS Excel or something)...


Cheers,
Jack

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!