Writing a robust function tester using templates.

Published March 05, 2016
Advertisement

My motivation was to write a test framework that I could run on functions to check whether they properly handle bad input. The below code sort of assumes the tested function does not throw, because that would defeat the purpose. Instead it should return gracefully and/or output a pretty error string. If you've done a bad job at coding, you'd, of course, expect it to crash.

The tester takes a set of reference arguments, which come in the form of hopefully valid input arguments that you provide. Code that runs all tests might then look something like this:void my_test_nocrash(int32 i32, char* str, testing::eTestType argtype, void* ptr, math::vec3 vec) { lout << i32 << " " << (void*)str << " " << uint32(argtype) << " " << ptr << " " << vec << endl; }void my_test_crash(int32 i32, char* str, testing::eTestType argtype, void* ptr, math::vec3 vec) { lout << i32 << " " << str << " " << uint32(argtype) << " " << ptr << " " << vec << endl; }....void* ptr = (void*)0x12345678;testing::test_function(my_test_crash, 33, "mystr1", testing::eTestType::eRandom, ptr, math::vec3(1, 2, 3));testing::test_function(my_test_crash, 45, "mystr0", testing::eTestType::eDefault, ptr, math::vec3(3, 2, 3));testing::test_function(my_test_crash, 33, "mystr1", testing::eTestType::eDefault, ptr, math::vec3(4, 2, 3));testing::test_function(my_test_nocrash, 34, "mystr0", testing::eTestType::eDefault, ptr, math::vec3(2, 2, 3));testing::test_function(my_test_nocrash, 22, "mystr0", testing::eTestType::eDefault, ptr, math::vec3(5, 2, 3));
Essentially I call [font='courier new']testing::test_function(your_func, ...args)[/font] with of arguments that I'd normally pass directly to [font='courier new']your_func[/font]. There are no glaring restrictions on argument types, but emulating bad input from non-trivial types takes a bit more work and requires modifying the return value modifiers. As such, the tester in its current form serves best to try out bad scalar, string and pointer values.


The above code runs five different types of tests on two functions (as the two last modes would crash if the [font='courier new']char*[/font] wasn't cast to a [font='courier new']void*[/font] and passed it on to the output stream as a string). The snippet also hilights the five profiles I wrote for myself, of which two are largely useless. Here's a brief rundown of what each one does:

[font='courier new'][color=#57a64a]/// [/color][color=#57a64a]eDefault[/color][color=#57a64a]: arguments are passed to the function without change. This is as if the function[/color]
[color=#57a64a]/// was called directly with the arguments provided.[/color]
[color=#57a64a]/// [/color][color=#57a64a]eNull[/color][color=#57a64a]: each argument is converted to the equivalent of NULL, which is often a valid,[/color]
[color=#57a64a]/// if undesirable input.[/color]
[color=#57a64a]/// [/color][color=#57a64a]eRandom[/color][color=#57a64a]: input values are "randomized", but their type is not compromised. This means:[/color]
[color=#57a64a]/// * a string will still passed to the function as a string, containing the word "[/color][color=#57a64a]<>[/color][color=#57a64a]"[/color]
[color=#57a64a]/// * pointer values are reverted to NULL so as to not cause unwanted crashes[/color]
[color=#57a64a]/// [/color][color=#57a64a]eUninitialized[/color][color=#57a64a]: arguments are assigned values you'd expect the debugger to assign[/color]
[color=#57a64a]/// uninitialized variables:[/color]
[color=#57a64a]/// * strings and pointers become "FDFDFDFDFD"[/color]
[color=#57a64a]/// * scalar values are assigned the equivalent of -1[/color]
[color=#57a64a]/// * floating point values are assigned NAN[/color]
[color=#57a64a]/// [/color][color=#57a64a]eGarbage[/color][color=#57a64a]: all argument values are fully randomized. If the function call expects pointers,[/color]
[color=#57a64a]/// this is pretty much guaranteed to crash the program.[/color][/font]

In most cases, running the function with eNull is sufficient, but eRandom might be useful to test how it behaves with ill-formed input. eUnitialized and eGarbage are likely to crash the program outright and frankly I can't really think of a use for them in normal circumstances. Of course you can always modify or add different test types to match your criteria.

Here's the output for the above test run. It's not pretty, but hey - it's not coding that's about the looks - it's the final product.

[quote]


[Testing] Running eDefault: 1/1

33 Everybody <3 unicornses. 4 12345678 (1.00, 2.00, 3.00)

[Testing] Running eNull: 1/1
0 00000000 2 00000000 (3.00, 2.00, 3.00)

[Testing] Running eRandom: 1/3
-721371808 <> 2 00000000 (4.00, 2.00, 3.00)
[Testing] Running eRandom: 2/3
-477658674 <> 2 00000000 (4.00, 2.00, 3.00)
[Testing] Running eRandom: 3/3
-842655645 <> 2 00000000 (4.00, 2.00, 3.00)

[Testing] Running eUnitialized: 1/1
-1 FDFDFDFD 2 FDFDFDFD (2.00, 2.00, 3.00)

[Testing] Running eGarbage: 1/1
1073937667 B73BBFE2 2 A06BD497 (5.00, 2.00, 3.00)[/quote]


With that aside, here's a quick rundown of how it all works:


  • wrap the function call up in a template and patch into the argument list using an initializer list. A [font='courier new']std::vector[/font] works nicely for this.
  • the argument list never leaves scope, so what the vector does is use a proxy container ([font='courier new']value_proxy[/font]) to store a pointer to each argument. Each argument is referenced into a [font='courier new']T*[/font] in [font='courier new']value_proxy[/font]'s templated constructor and assigned to an internal [font='courier new']const void*[/font] storage.
  • call the function, using the vector as a new proxy argument list. Since partial specialization of functions is not allowed, use a proxy class ([font='courier new']call_modified[/font]) to achieve that.
  • use the cast operator in [font='courier new']value_proxy[/font] to intercept the value when the argument is being forwarded and change the value types you want to what you want. You can write these for any type by modifying the __return_XXX macros. eDefault mode should work out of the box for most things, although I haven't tested it all that heavily.
  • the whole thing is header-only, depends only on and optionally whatever fancypants logging aggregate you might be using. I modified the attached code to use [font='courier new']cout [/font]so it should compile and run out of the box. For the provided code you'll also have to drop in your own rng functions, which are currently dummies.
  • the number of arguments a function can have is limited by how many overloads of [font='courier new']call_modified[/font] are implemented. I've implemented templates for up to 9 arguments, but you can add as many as you like.
  • return values are not handled
  • it's not fast, but that's not the point


This is a fairly specific piece of code that I wrote with some, but ultimately no particular purpose in mind, but I imagine there are people whom this might benefit a lot more. It was a fun exercise and a way practice template voodoo. There are a few macros in there as well to drastically cut down code bloat. They do occasionally affect readability a bit, but also improve it in different ways.


One thing that might likely make sense as an extension would be an incremental tester, which replaces one argument with an ill-formed value at a time.

PS - hmm, GD doesn't allow attaching header files, so I zipped it.

1 likes 0 comments

Comments

Nobody has left a comment. You can be the first!
You must log in to join the conversation.
Don't have a GameDev.net account? Sign up!
Advertisement
Advertisement