Jump to content

  • Log In with Google      Sign In   
  • Create Account

FREE SOFTWARE GIVEAWAY

We have 4 x Pro Licences (valued at $59 each) for 2d modular animation software Spriter to give away in this Thursday's GDNet Direct email newsletter.


Read more in this forum topic or make sure you're signed up (from the right-hand sidebar on the homepage) and read Thursday's newsletter to get in the running!


Unit Testing in C++


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
20 replies to this topic

#1 EnlightenedOne   Members   -  Reputation: 142

Like
0Likes
Like

Posted 05 April 2011 - 03:46 PM

I have developed a game engine in C++ in Visual Studio 2010. I decided to run unit tests on my Engine class and some of its sub classes. Unfortunately c++ unit tests in an unmanaged environment are apparantly impossible and as c++ is inherently an unmanaged language I do not know why this isnt by default supported. I was expecting a Nebteans/Java stlye unit test code generation practice. I do not believe that it is impossible to generate unit tests, I have just failed to find any logical explanation from MSDN or any practical application the closest being (http://unittest-cpp.sourceforge.net/) I have not looked at boosts incantation.

I do not care what incantation I use but I do not have all week to write out every test scenario for several classes. I am in a situation where time is most certainly against me and unit testing is the absolute bottom feeder on the list of things I had expected/can afford to encounter in a professional development environment as having appauling support. Can any one guide me toward a rapid unit testing solution for C++ or show me a guide to hooking up the unit tests through VS2010.

I don't (have time to) care about version control, I don't (have time to) care about massive flexibility in tests, I do care about rapidly demonstrating some simple test cases in VS2010 or otherwise with my .sln and C++ files.

Please help me if you know any way to generate dummy test cases or any test cases in VS2010 when using C++ I dont even care if I have to run it with specific settings to demonstrate the test (I know that its a bad thing to do) so long as I can make it run I am happy.


Many thanks,

EnlightenedOne

Sponsor:

#2 frob   Moderators   -  Reputation: 22779

Like
0Likes
Like

Posted 05 April 2011 - 07:55 PM

Unfortunately c++ unit tests in an unmanaged environment are apparantly impossible

Very much false.

The xUnit suite and clones have been implemented in many languages, including C++. A few of them are CxxUnit, CppUnit, and CppUnitLite. There is also CUnit which is for C but works just fine in C++.

Reflection, the ability for something to peek inside the program and figure out what is a test, is a feature in just a few languages. This is probably the missing feature that causes you to believe it isn't supported by default.

In most languages you need to explicitly list the functions needed for your unit tests. This applies to C++ unit tests.

I have not looked at boost ... I do not have all week to write out every test scenario ... unit testing is the absolute bottom feeder on the list of things I had expected/can afford to encounter ... I don't (have time to) care about version control, I don't (have time to) care about massive flexibility in tests ... EnlightenedOne




They are not something you can rush.


Unit tests generally have a roughly 1:1 correspondence in code size. If you have a 5000 line class, you should expect to have roughly 5000 lines of test code to properly exercise that class. Adding tests as an afterthought tend to increase the size and complexity of the tests. Adding it after expect to have 6000+ lines to test it, or to have large blocks that are not tested.


It is well researched that writing unit tests takes roughly the same amount of time as the initial implementation, and also that implementing code and test at the same time causes nearly double the time of a test-free implementation. You will be hard pressed to back-fill tests in less time than it took to write initially.


I'm not sure what version control has to do with this, it seems irrelevant.

And as for flexibility in tests, well, that's very telling.

Good luck Enlightened one. I hope you become so.

Check out my book, Game Development with Unity, aimed at beginners who want to build fun games fast.

Also check out my personal website at bryanwagstaff.com, where I write about assorted stuff.


#3 Slavik81   Members   -  Reputation: 360

Like
0Likes
Like

Posted 05 April 2011 - 08:15 PM

What do you mean by "Java style unit test code generation"?

#4 Hodgman   Moderators   -  Reputation: 31920

Like
0Likes
Like

Posted 05 April 2011 - 08:21 PM

What do you mean by "Java style unit test code generation"?

I'm intruiged too... How do you test classes without writing test cases? How does a code generator know the usage and expected behaviour of a class automagically, in order to write a test for you?

#5 wqking   Members   -  Reputation: 756

Like
0Likes
Like

Posted 05 April 2011 - 11:25 PM


Unfortunately c++ unit tests in an unmanaged environment are apparantly impossible

Very much false.

The xUnit suite and clones have been implemented in many languages, including C++. A few of them are CxxUnit, CppUnit, and CppUnitLite. There is also CUnit which is for C but works just fine in C++.


I use cxxtest which uses Perl to generate the test case.
I'm happy with it. But maybe it's not under active development.

http://www.cpgf.org/
cpgf library -- free C++ open source library for reflection, serialization, script binding, callbacks, and meta data for OpenGL Box2D, SFML and Irrlicht.
v1.5.5 was released. Now supports tween and timeline for ease animation.


#6 EnlightenedOne   Members   -  Reputation: 142

Like
0Likes
Like

Posted 06 April 2011 - 06:17 AM

I have error handling code within all classes and threaded classes I wish to test (including handling of various threaded classes having to do an emergency shutdown and others taking over their behaviour as best as is possible). I can just switch the error handling components to report unit tests failing using the strings I already have being output to report errors on a per class basis. Having unit tests on top of this is all to give evidence that the error handling actually works.

Unit testing would just be a matter of generating an empty method for every function WITH ACCESS TO THE CLASSES USED IN MY PROJECT (Which I cannot do in Visual Studios testing project for god knows what reason) and then making them do a list of generic tasks to put the engine class into the right state to run the tested function. The engine is all wrapped up so getting the state/context right for functions would not be hard. Every class has an error integer so its state can be tracked by an error handling thread, RETURNSUCCESS or RETURNFAILURE already exist and I have a std::string which retains every error recorded, I basically have all the functionality of HRESULT for my engine in place. So yes unit testing is a serious buisness but with the error handling deeply in place I have expected to be able to rapidly assemble unit tests.

This is where I am:
I have managed to find something. Executing "Test->New Test->Unit Test" via the menu I end up with a blank test program which I linked up to have all the resources that my actual engine class has. I then made it dependent on all the other projects being compiled and tried to include the EngineClass.h. It reached the dependencies the engine class includes and threw out an error.

[size="1"][size="1"]1>C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\include\vadefs.h(85): error C4956: 'va_list *' : this type is not verifiable

1>C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\include\vadefs.h(86): error C4956: 'va_list *' : this type is not verifiable

1>C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\include\vadefs.h(86): error C4956: 'void *(va_list *,...)' : this type is not verifiable

1>C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\include\vadefs.h(87): error C4956: 'va_list *' : this type is not verifiable

1>C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\include\crtdefs.h(543): error C4956: 'const wchar_t *' : this type is not verifiable

1>C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\include\crtdefs.h(543): error C4956: 'const wchar_t *' : this type is not verifiable

1>C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\include\crtdefs.h(543): error C4956: 'const wchar_t *' : this type is not verifiable

1>C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\include\crtdefs.h(550): error C4956: 'const wchar_t *' : this type is not verifiable

1>C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\include\crtdefs.h(550): error C4956: 'const wchar_t *' : this type is not verifiable[/size][/size]

Every seperate include gave errors. Everything from strings to my physics library are pumping out several thousand errors for anything that is remotely like a pointer. How can I test classes in this test project if I cannot include any of the classes dependencies for the thing I wish to test?

This is lame because Java provides an easy to build off of framework from your code at a class level for JUnitTests you can then provide produce the contextual state for the test case prior to the test. You do not have to worry about dependencies what-so-ever and you do not have to write out every method per class function you wish to test. You do have to provide the intial context before each test case but this is a game engine its all wrapped up and ready to rock, I just need a test method for every classes function I have to run and the rest will be rapid. In Java I would not be hassled to reproduce all of the includes, can anyone explain what would need changing to get these test cases to be capable of including the classes/dependencies my project uses, would I need a dummy version of the entire project with different compilation configuration?

This is the worst thing I have ever encountered in VS because getting to the point where test cases are ready to be written would be instaneously done for me by Netbeans.

Thank you wqking in lack of any idea of what VS wants me to do to test my classes I will try cxxUnit or one of the others after I try one or two more things with VS method to see if I can tempt it to let me test my classes.


#7 Bregma   Crossbones+   -  Reputation: 5481

Like
0Likes
Like

Posted 06 April 2011 - 07:09 AM

... can anyone explain what would need changing to get these test cases to be capable of including the classes/dependencies my project uses, would I need a dummy version of the entire project with different compilation configuration?

You need to either redesign the C++ language to be interpreted and provide runtime reflection capabilities (like Java), or write a compiler that generates test code as a back end (and modify MSVS to use that compiler). The latter is the easier route, mostly because it's at least possible. Try starting with GCC, all you need to do is write a back end for it.

Sounds like an engaging, worthy project. Sorry it's not the solution you're looking for.
Stephen M. Webb
Professional Free Software Developer

#8 EnlightenedOne   Members   -  Reputation: 142

Like
-1Likes
Like

Posted 06 April 2011 - 07:25 AM

Here is evidence that I am not going to get a fair trial for unit testing http://msdn.microsoft.com/en-us/library/ms243171.aspx no C++ libraries appear to be managed hence the errors I was being thrown earlier trying to use any libraries. Simply put I just found out what /clr does (ish) I am fresh out of university and the idea of any language not having unit testing capacity in it is moronic. Now I have shot myself in the foot in that I planned to do unit testing in my reports and I have no functionality that will enable me to do so therefore I have been skrewed by a technicality. I don't even know how to turn /clr behaviour on.

So in short I would have to use an external library to get some simple behaviour.

#9 ApochPiQ   Moderators   -  Reputation: 16413

Like
1Likes
Like

Posted 06 April 2011 - 07:55 AM

Why are you so insistent with this completely incorrect notion that unit testing only works in managed languages? The two are utterly orthogonal.

#10 frob   Moderators   -  Reputation: 22779

Like
3Likes
Like

Posted 06 April 2011 - 10:12 AM

Why are you so insistent with this completely incorrect notion that unit testing only works in managed languages? The two are utterly orthogonal.


Rereading his posts, it seems pretty clear:

First post, my summary:
I made a game in VS2010. I can't make VS2010 unit tests work on my code, and I heard Eclipse had these awesome tools. I don't know what I'm doing. It is impossible. I'm DOOMED. I need to get a full suite of unit tests written immediately or I'm DOOMED. Here are a bunch of things irrelevant to my problem: Boost, Version Control, VS2010, Java. It's urgent, I need a tool. Do it for me or I'm doomed. I don't care if I need weird settings in the tool, I need the tool to do it for me because I don't know what I'm doing.

Follow up post, seemed to confirm that panic:
I need to give evidence to somebody that my code actually works. I don't know what unit tests actually do, I clearly don't understand what I'm doing, and I need "to be able to rapidly assemble unit tests", urgently. I have compiler errors, I don't know what I'm doing, and I need a tool to do it for me. I have incorrect assumptions about how JUnit works: It is like magic and sex and drugs combined. In fact, I think JUnit and Java are the same thing. Why doesn't C++ work like my misunderstanding of Java? VS2010 is dumb because it doesn't do what I think it should. If the tool doesn't do it for me, I'm DOOMED. Now I'm going to insult people who tried to offer help.

Third post, more of the same:
It is their problem, not mine. I did everything right. "I am not going to get a fair trial for unit testing", it is their fault. I just graduated so I know everything, and "the idea of any language not having unit testing capacity in it is moronic. Now I have shot myself in the foot". I am doomed, "skrewed", and don't know how to be a programmer. I can't take any responsibility and it is their fault and I'm doomed.




Yes it is a bit jaded, but that's how I see it.

Unit testing is a parallel work effort that is equal in size to the initial development. It requires significant work, and significant engineering effort. It requires skills in the testing theory and tools, no different than graphics programming requires skill in math and graphics libraries, or database programming requires understanding of databases and database libraries.

Several languages have some features that make writing and running unit tests a little easier. Sometimes people merge the good features and ignore the problems combining them to create an idea of unit testing that is dramatically different than current technology.

He is not the first or last to assume that Unit Testing is a magic pixie dust that gets sprinkled over code by the IDE and some other tools, and you can apply it liberally at the end of a project. You see it every few months on the TDD and xUnit newsgroups. Usually people there don't have the same whining and "I'm DOOMED" overtones, but they sometimes do have the "it is urgent because I agreed to a contract and didn't know what unit testing is" viewpoint.

Check out my book, Game Development with Unity, aimed at beginners who want to build fun games fast.

Also check out my personal website at bryanwagstaff.com, where I write about assorted stuff.


#11 zerothrillz   Members   -  Reputation: 152

Like
1Likes
Like

Posted 06 April 2011 - 10:55 AM

Umm... Am I wrong in thinking that unit testing, in it's simplest, doesn't require a huge unit testing software suite to generate and run the tests for you? I know that kind of software would be necessary in production of a large system with many parts utilizing many languages, but if you're only using C++ on a small project, whats wrong with writing your own test functions and running tests in the same process before executing your actual program?

#12 frob   Moderators   -  Reputation: 22779

Like
1Likes
Like

Posted 06 April 2011 - 11:27 AM

Umm... Am I wrong in thinking that unit testing, in it's simplest, doesn't require a huge unit testing software suite to generate and run the tests for you? I know that kind of software would be necessary in production of a large system with many parts utilizing many languages, but if you're only using C++ on a small project, whats wrong with writing your own test functions and running tests in the same process before executing your actual program?


That is correct, and it is how the current xUnit libraries evolved.

People who had done it frequently just consolidated much of the reusable bits into a library, and also created fancy test runners that let you pick and choose subsets of the test, prettify the results, and make it easier to interpret.


Even with those libraries you are writing your own test functions, outputting events for success or failure (along with the reason for the failure), and then running the tests. Some tools can help automate the creation of function prototypes, but don't know how you intended to use the code. Using either route, using a library of tools or doing it yourself, the work of writing the tests remains, and it is significant.

Check out my book, Game Development with Unity, aimed at beginners who want to build fun games fast.

Also check out my personal website at bryanwagstaff.com, where I write about assorted stuff.


#13 EnlightenedOne   Members   -  Reputation: 142

Like
-1Likes
Like

Posted 06 April 2011 - 05:43 PM

So far we have established that the functionality exists in a convoluted form and there is no rapid implementation available, there is perhaps no benefit from implementing boost tests in order to check values are equal. I might aswell completely write out my own function for each method which just runs every function and identifies the outputs validity completely from scratch. In fact that is exactly what I will have to do. I was only reiterating exactly what MSDN had told me in relation to the lack of plausibility for unit testing unmanaged code, I even had a reference pointing to something which implictly stated that unmanaged code and unit tests were not going to fly.

To clarify I do not want someone to do anything in my project for me, I find that particularly insulting I am (well was/am) concerned/insulted by the poor quality of support for what should be a normal functionality in any language when reflecting on the ease of functionality provided for basically all other languages in all APIs. I never used the terminology "doomed" although I am in a squeeze and while I am aware that unit testing is usually something you would perform as you develop it would easily over double the development time for this project because as my understanding of the components I am modifying has evolved so has the architecture I have used in an Agile fashion.

It is inefficient to start with debugging when your doing something that is going to evolve and go through essentially paradigm shifts. A prime example is learning that direct x resources are exclusive to their creating threads and that the direct x device must be initialised from the same thread as the thread which created the windows window it renders in. I had to alter the entire operation of multiple threaded classes to change the engine and make that work, It would have been a waste of maybe a hundred hours development if I had unit tested those unworkable classes. In a scenario where I started the project with the knowledge I have now and everything was less volatile I would certainly start with unit testing as I now have the prinicple understanding as to how to use the components I am using. It remains a fact that I am better off unit testing now as its good practice to have unit tests in place when you are confident the fundamental architecture for a project is sound. The unit tests are actually part of my own requirements for my project submitted in my last interim report I have done unit testing in other simpler projects/languages but in this case I had to prototype and the prototype evolved into something stable and worth unit testing.

Unit testing is not pixie dust neither is having the capacity to run unit tests which show failed or succeeded, how hard would it have been to have used the functionality that generates a class diagram to generate a class full of stubs for unit testing based on the classes I have in use autonamously, probably about as hard as it is to generate stub c# unit test cases as it already does. It is something I do expect as a standard for any language and while the capacity does exist to integrate unit testing through seperate third parties or reptitive "manual" labour my opinion of Visual Studio has been greatly smacked by this time delay.

I have decided unit testing becomes a lot less valuable when it consumes time and you already have error handling code to flag anomalies that are not obvious from the visual/audio/physics behaviour. With the time cost added from the lack of stub generation and the need to produce an entirely new project unit testing right now seems a massively cost inefficient additional burden. From my understanding now I have to produce a new project dependent on my engines project and produce a set of unit test sets that are just as likely to fail silently as the actual project when being run. Perhaps I will use the std string library to state no tests failed... what if I generate an error in the thing that registers the error handling is working properly? I guess that is why the third party libraries exist. Still seems like laziness on VS part wasting developers time. I do have to argue it might be faster to test my own tester than attempt to process the huge scope of components involved in the unit tester boost provides, as a library it is of collossul proportions.

That (last paragraph mainly) is my argument with this forum and why I have been angry I have not misunderstood unit testing as a magical concept, everyone has stigmatised my perception of unit tests and failed to acknowledge how many man hours VS have wasted by causing all these deviations of libraries that perform unit testing to manifest. Not to mention all the additional time everyone else has to spend learning how to use them, its unfathomably massive. You cannot argue with that its pretty solid logic :P

Anyway we are done now thank you for routing me to the behemoth unit testing libraries I will now have to make a selection from in order to proceed. I hope you have come to gain some insight in that I am not whining for the sake of pixie dust in unit testing but for the sake of fundamental practices existing on an API commercially worth roughly a grand. I will not look for a new API when I am done getting myself out of the mess I have been dropped in by this unfortunate lack of foresight on both my side and my API's side because no other API comes close. If you know an API that provides this base functionality off the bat for C++ I might transfer in order to perform unit tests and I would love to hear about it, if not we are done here and you can wish me luck or debate the validity of my argument on unit testing an evolving prototype before its stable :)



#14 EnlightenedOne   Members   -  Reputation: 142

Like
0Likes
Like

Posted 06 April 2011 - 05:46 PM

"Using either route, using a library of tools or doing it yourself, the work of writing the tests remains, and it is significant." I cannot argue with that which is why I will only unit test some of the core components such as the engine class (not all of its sub threaded classes) before my next deadline so as to cover the bases.

#15 Bregma   Crossbones+   -  Reputation: 5481

Like
-4Likes
Like

Posted 08 April 2011 - 12:25 PM

You went wrong when you decided to use MS Visual Studio. You should use emacs. Emacs has a plugn to write unit tests for you.
Stephen M. Webb
Professional Free Software Developer

#16 e‍dd   Members   -  Reputation: 2105

Like
2Likes
Like

Posted 08 April 2011 - 04:58 PM

Still seems like laziness on VS part wasting developers time.

I'm left wondering after reading your posts what it is exactly you expect Visual Studio to do for you. Setting up a new project should take you a couple of minutes and many lightweight frameworks require very little boilerplate per test. For example, the testing library I use means my tests look like this:

TEST("The discombobulator assembles the widgets")
{
    discombobulator d;
    CHECK(d.assemble_widgets() == true);
}

I guess Visual C++ could in theory generate the 5 or 6 characters that aren't boilerplate, but I really can't see a lot to complain about, on balance.

That (last paragraph mainly) is my argument with this forum and why I have been angry I have not misunderstood unit testing as a magical concept, everyone has stigmatised my perception of unit tests and failed to acknowledge how many man hours VS have wasted by causing all these deviations of libraries that perform unit testing to manifest. Not to mention all the additional time everyone else has to spend learning how to use them, its unfathomably massive.

Pick one. Read the manual. You'll be writing tests within no time. FWIW, I rolled my own. The introductory page on the wiki will have you up and running in 5 minutes flat should you choose it. The same is true of many other libraries.

If you know an API that provides this base functionality off the bat for C++ I might transfer in order to perform unit tests and I would love to hear about it

What is this "base functionality" you desire? In your first post, you say you'd like the IDE to generate "an empty method for every function WITH ACCESS TO THE CLASSES USED IN MY PROJECT". This isn't necessarily a good approach to writing unit tests because having one test per method will often mean:

  • All the behaviours for a method are crammed in to one big test. This granularity makes it hard to narrow-down failing code.
  • Interaction between methods on an object aren't tested. As a contrived example, std::vector<>::push_back() should change the size() of the associated vector.

Writing good tests, or even half-way-sane tests requires some thought.

#17 LorenzoGatti   Crossbones+   -  Reputation: 2774

Like
1Likes
Like

Posted 11 April 2011 - 04:00 AM

I'm left wondering after reading your posts what it is exactly you expect Visual Studio to do for you. Setting up a new project should take you a couple of minutes and many lightweight frameworks require very little boilerplate per test. For example, the testing library I use means my tests look like this:

TEST("The discombobulator assembles the widgets")
{
    discombobulator d;
    CHECK(d.assemble_widgets() == true);
}


This example shows the two most important things that a unit testing library can do for you:
  • help you make assertions (here the CHECK macro) and report their results nicely as test failures
  • collect tests to be run (here the TEST macro), which is really the only use of reflection in JUnit (looking for patterns in function names) and can be perfectly replaced by other appropriate mechanisms in other languages (like macros here)
These features, together with other framework-style support for writing tests (mainly factoring out text fixtures and enforcing a regular structure) are what makes an unit test library better than writing everything on your own.

The best an IDE can do is running your tests and reporting the results; integration can be very nice and convenient, like Eclipse with JUnit, but a stack trace is a stack trace even if you get it in a command prompt or an ad-hoc test runner after compiling your test suite as a regular program: don't waste time on researching low-value tools, and specifically don't reject unit test libraries because an IDE is only integrated with something else.

Nothing meaningful that can be generated automatically. Python or Java tools can inspect code to create piles of empty tests for each class or method for you, which are not only misleading and useless without you actually testing something, but often mismatched to the structures of tests you actually need. Just copy and paste empty or existing tests instead, without creating graveyards of dead code.
Produci, consuma, crepa

#18 Jhon Qiki   Members   -  Reputation: 100

Like
0Likes
Like

Posted 15 February 2012 - 01:54 AM

You can use /clr compiler switch to fix these errors.That force the compiler to generate mixed assemblies contain both managed and native parts, allow you to apply .net features  in unmanaged code. turn off /clr:safe switch by going to project->properties->Configuration Properties->General, in configuration page look for &amp;quot;Common Language Runtime Support&amp;quot;, select /clr option.
A common way test your code(native code) in managed C++ is to use P/Invoke.
but if ur project is not integrated with TFS, U should choice better test suite to accomplish ur work.
hope this info can help u.

#19 Antheus   Members   -  Reputation: 2397

Like
0Likes
Like

Posted 15 February 2012 - 08:25 AM

Is this about IDE? The visual studio?

The fact it does not have File->New Unit Test? Which in Eclipse would generate a test stub?

#20 Slavik81   Members   -  Reputation: 360

Like
0Likes
Like

Posted 15 February 2012 - 11:22 AM

This topic is nearly a year old. The OP is probably not still around.




Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS