Jump to content
  • Advertisement
Sign in to follow this  
Techieman

Unit Testing multimedia classes

This topic is 661 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hey folks,
recently I got an enlightenment and I grasped,

how important unit tests actually are.

Also I understood how practically "Test Driven Development"

for solo developers can be.

In theory it sounds really great (yet another holy grail for me).

 

However, I don't know how to pull it off.

 

My current situation is the following:
I am developing a library, which provides facilities for cross-platform

game development. My way of testing 'till now is/was like that:
Does it display (or play the sound) as I expected, then it works.

How would one test for these kind of results? I.e. how do I know,

if I have a window actually being displayed on the screen?

I feel really lost about this. It quite easy to check, whether my "fibonacci algorithm"

calculates the proper results. But what about visual or aural results?

Hopefully someone can help me out on this. :-)

With best regards

 

Julien

 

Share this post


Link to post
Share on other sites
Advertisement

Well, you can't do that. Unit tests are not for that. Unit tests have strict rules: they have to run fast (all of the project's unit tests must run in a few seconds altogether), they can't access hardware (so no graphics, no sounds, no file I/O, no network I/O, no DB access), they have to test the functionality of a single unit separated from the others. Unit tests are used after every changes or at least before every check-in to repository. So you can't play sounds shows UI for minutes, otherwise your developing speed will decrease drastically.

 

I know there are GUI tester programs which hooks into the OS, and records keyboard and mouse events, replays them, and checks for window creation. I think they don't graphically test for the windows, but they check if Window creation calls are sent to the OS.

 

You can also use automation libraries. This is a library, that you can write, accesses your code from outside to start a task, and it can also checks the result the same way. Though it can't really check if the sound is played correctly, but it can check if the correct sound data is sent to the sound engine.

 

I don't heard of any automatized graphic/sound tester softwares.

 

Share this post


Link to post
Share on other sites

zaphod2  == :ph34r: :wink:

The answer is (which may not be to your liking :(): you don't!

The level of testing actually describe, what and how you test in your software. Unit testing means you test small sections (units, e.g.: functions) of your code, whether they do what the requirements and documentation tell. Testing ultra-high level concepts/code/functionality with unit testing is actually "bad" in the sense, that the higher level the functionality is (building on top of a bunch of other code and having a huge number of required/dependent components) becomes increasingly difficult. Less net win with the tests especially measured in time required to test and automate vs quality assurance return. Plus the tests (talking about a shit-load of unit tests with high coverage) may interfere a lot with code changes, and admit it, gameplay code is tweaked and changed a lot and even major modifications can occur.

Of course these problems can be mitigated with correct software design (software can be architected in a way to be easily testable and be flexible regarding changes, e.g.: plugable and composition based design + mocking comes to mind regarding testing tools), but I would advise to use the correct testing level for the correct feature level.

"Real" answer:

- When it comes to low-level code (e.g.: boiler-plate stuff on which a lot of other stuff will depend on in a game/software):
Unit test the hell out of it, make sure close to every statement, decision/branch and execution path of these functions work the way it is expected to work.
But again, you should not be opening files, loading assets, reading "real" sockets, playing sounds, or creating a window and drawing to the screen. These are much "harder" to test, and by their very nature, check the functionalities and integration of multiple software components (e.g.: not a simple "standalone" function call and its result but a huge amount of calls, system calls, and a bunch of setup/preliminary state).

When you reach a higher level, e.g.: I'm loading some assets from the disk and use them to produce graphics on the screen, or I'm loading an xml/yaml/json file and parse a list of components/entities out of it and check whether the thing end up with the correct scene at runtime, or I'm starting my network thread, open my sockets, listen on X and Y ports (maybe even simulate incoming packets just to see what happens), that is a new level called "integration" (if you go even higher you reach functional and system) testing.
This can be exercised using code the same way as with unit test, but you simply will not be able to write as many tests, as preliminary preparation parts may become huge (e.g.: I open a window, create a rendering context, load a texture, draw it on the screen etc...) and the actual tested part depends on a bunch of other parts already established/working/running. Also these tests usually become heavy in the time it takes to execute them, since a lot of things may happen in one compared to a unit test (e.g.: running 10K unit tests takes a minute, running 10K of these higher level ones may take hours or more...).

What usually happens, that a testing framework (just a small tool or a small library) is written/used-off-the-shelf for exercising your software for doing automated "integration"/"functional" testing. Some ideas about how this can/may work:
- The framework/tool sets up a window + a rendering context, runs some code (THE TEST-CASE), takes a screen-shot of the window and compares it to and old one taken as the reference.
- The framework/tool sets up the network system, starts simulating a bunch of packets previously captured with a tool like wireshark, runs some code (THE TEST-CASE + the captured packets are part of it too) to check whether the network system is in the expected state (e.g.: it ends up in a state where it believes two peers correctly connected to start a multiplayer match...)
- The framework/tool sets up a recording system, which records the audio output of the OS (e.g.: look up "Stereo Mix" for windows + there is already a bunch of free/open tools for this), runs some code (THE TEST-CASE), and checks whether it is silent or your test did actually make a sound.

Important:
- When testing, most of the principles apply as in the case of software development:
WORK SMART, your tests and test-framework is just as much a software as the software you are testing. It should be flexible, tidy, elegant. Individual tests themselves are dispensable, but the testing system(s) should be rock-solid and well architected.
- Find the important/critical parts and test the hell out of those:
Do not test every miniscule part, especially not on every level, e.g.: if you tested lower level systems correctly, you should already be trusting those parts, no need to test every single time in your game when you try to make a sound and draw a sprite if it works. If a sound cue is really important in the game-play logic, simply MOCK the sound system, and voila, write a simple unit test to check whether the given part of the game-play logic actually calls the sound system...
- Only test and automate stuff which will make a net return:
Testing a software can never be finished only abandoned, just as the development of the tested software. Check whether the testing you do will make a return regarding the time it takes to create and automate the tests. Usually this is a no-brainer in case of unit testing, as it takes only a fraction of the development and "manual" testing (trying out your fresh code with a debugger) time to write the unit tests for a bunch of functions, but with higher level tests it is not the case. These abominations may take a lot of time to make/write :lol:, but may not catch any bugs during their life-time. That is why in case of higher level testing risk analysis and stuff like that is done up-front, and mostly the critical/important and positive outcome (the common use-cases) is tested.

Shameless self-promotion:
I made a record & replay (input recording + replaying and screen-shot comparison) based testing framework for my games. Works like a charm, but took some time to pull it off, but as I know there are off-the-shelf solutions too for this problem, though may not be tailored for games.
If you are interested in the topic, more about it here: Part 1, Part 2

I hope this helped!
Br.

Edited by Spidi

Share this post


Link to post
Share on other sites

I am developing a library, which provides facilities for cross-platform game development. My way of testing 'till now is/was like that: Does it display (or play the sound) as I expected, then it works. How would one test for these kind of results? I.e. how do I know, if I have a window actually being displayed on the screen?


There is a great book called XUnit Test Patterns, also available as a web site although I'm on mobile right now so no hyperlink. Edit: Now enhanced with hyperlink, go read, the reference has about 1000 printed pages covering everything you ever wanted to know about unit tests plus some other kinds of tests.

Unit tests for multimedia depend on the unit being tested.

If you would call a system to give it data, pass in a mock object and known data and verify that it makes the right call with the right data. If you expect to give the object a StreamingVideoPin object called svp, then you expect svp->drawframe(data) with specific data, provide a mock object and validate the results.

You may provide a mock object that ensures a windowing system has a window object as part of a unit test, but creating an actual window is part of an integration test if it is automated by screen scraping or similar, or a manual test to see if there is a window on the display.

You may have an audio stream and provide a mock object that represents the audio player as part of a unit test, but ensuring the sound is hitting the audio hardware is an integration test or a manual test to see if the human hears the sound.

Generally for automated tests everything except the final presentation can be covered by unit tests. That last layer of actual hardware doing the hardware jobs doesn't get unit tests, but may have integration tests if you really want it, or you can wait until QA or people complain. That last layer is usually a really tiny shim, and if you've done everything well then everything up until that layer has full automated tests behind it.


Finally, remember the actual results of unit tests isn't to ensure the results are correct, but to ensure the results do not change. Consequently tests usually are best for data being transformed into other data, where the tests validate the transformation. Next best tests are data in motion, where you can write tests that verify data moves through the pipe as expected.

Edited by frob
It's later and I have a computer, edited with hyperlink.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!