Hi all!

It is time for some software testing framework talk again :)!

As you know, I'm a big quality/testing advocate (especially when it comes to software!!!), and I've been working for a while now, in my "lab", on a high-level testing and automation framework, since I've reached the limits of usefulness of unit testing my game projects.
I'm going to showcase the newest addition to this testing framework, so if you completely missed it and interested, here are the two older posts summarizing the design and some implementation details (with showcase video :wink:) about it:
Testing - part 1.
Testing - part 2.
A quick recap, if you would not like to read through the usual pile of text :), but still interested in this dev. log. entry:
It is a "capture and replay" based framework, where you can record (or edit/create) input events (or special ones, e.g.: network, debug events etc...) from the game to replay them later. For checking certain functionality, the replay files can be filled with assertions targeting the properties of any game-object within the game world at any given frame of the replay.

So the system served me well while developing the Operation KREEP update. It took less than two work days to reach a pretty high coverage of game-play features and the UI work-flow (around 80% code coverage in 12 hours). This test-suite helped me a lot while releasing the Steam version, besides keeping my sanity by covering and assuring, that most of the high level features work, it saved me from introducing a few pesky bugs while coding the new features!

After a while though, execution times grow as it is to be expected. The 67 replays for the final Steam build(s) which checked the UI work-flow, completed the tutorial, played till getting most of the achievements and so on and so on, requires around 11 minutes to fully execute ( more than a cup of hot beverage takes from inception to getting it into the belly :D). I knew, that after a while this is going to happen, worked before with similar frameworks, but also already had some ideas in the back of my mind to fight it if it may become a problem. Obviously this was not a big irritation yet, but for a larger game it may become a certain source of frustration.



The most simple and obvious solution was categorizing test-cases within a suite (UI, Options, Graphics, Tutorial etc...) and only execute immediately required categories. This took only a short time to develop and configure as NUnit already had support for it. I just had to put some extra properties here and there.
This was nice and worked well, taking less to test the crucial/modified parts faster, but of-course there was a much smarter idea there from where this one came from (a full test still required 10+ minutes :mellow:, no way I could not improve on that :wink:)!
Match Pause Restart Tests\ReplayMatchPauseRestart.xml Match Pause

So I investigated two common solutions to this problem (actually there is a third one which is manual: modify test-cases to make them simpler and shorter, but that is not a general solution, takes time and the gains are small), and I went with the "speeding up test-case execution" route. I haven't find any good/common name for it, although it is a known solution, so I called it the "unlocked" game-loop. The concept is simple: when replaying the test-cases a different game-loop is used, which runs as fast as it can ( no vsync no sleep nothing like that, exercising the CPU/GPU like a mad man :lol:), but the elapsed, total and accumulated time calculated and passed to the systems and game-objects of the game is mimicking the normal game-loop, so the game "believes" it is running at normal speed with the target 30 or 60 frames per second. I was certain that it is going to speed up execution, and at least cutting execution time in half with a simple game. I was wrong, it became much much faster :cool:. After the new game-loop, the full test set took not much more than 2 minutes instead of 11...

Take a look:

[color=#808080]Note: the first two minutes show the normal execution of 6 test-cases and the rest of the video show the same tests executed with the "unlocked" game-loop.[/color]

The approach has some down-sides (as with every route in software development), e.g.: the game may use system time for certain features (although I think this should be avoided, since the game-loop provides an elapsed total time of the game execution handled the same way as the elapsed/accumulated time) and another one is full-screen/windowed mode toggling which is not supported by this feature at all (maybe in the future, I guess it could be done, just it would require some hacks, don't know yet). For these problems I "cleverly" introduced a per-test-case setting to override the game-loop-unlocked configuration, so the execution speed-up can be disabled for "unstable" test-cases.
true Toggle Fullscreen Tests\ReplayToggleFullscreen.xml false

Another "I'm not so happy about it" thing is, that it is a bit hackish and fully platform dependent solution currently, but I guess in time I will solve this problem :).

As I mentioned there was another route I could take to speed up test execution. I think it is a somewhat superior solution, but would have taken much more effort both software and hardware wise, so I decided to go with the simple one. NUnit has an open parallel execution engine add-on, and I think it requires no explanation why that route is superior, since the limiting factor would only be the number of machines I could harness, but setup (and stability?!) would be a much more complex issue. In time I may try it out, since I'm interested in the actual setup time it takes + I'm certain with a couple of boxes execution time would match the time it takes to run a unit test set :), but the current solution fully satisfies my needs and my work-flow.

The testing framework is in an extremely stable and usable state for production by now. I'm going to make good use of it for my current game too. In time I'm planning to add more features to it, so a "testing - part 4" entry may happen :), but not anytime soon + most probably I will only focus on smaller, usability enhancements and additions.

I'm still working on some framework-y code from time-to-time (maybe next entry will be similar, mostly technical) and the current game project is not yet ready for announcement, but with this game I'm going to do a more open development. So starting from the working prototype till the finished product, I'm going to post (weekly maybe?) releases with limited content to get feedback and improve on usability, balance and overall features from the first days and to reach more players interested in the game, even before going to greenlight/itch.io/hopefully-Steam etc...
Expect a somewhat playable version soon (I think within weeks).

Take care!