Use a logging system instead of a debugger

The next obstacle on your way while developing systems for processing large data sizes is that you are likely to reconsider your methodology of work with the debugger or even refuse to use it at all.

Some specialists offer to refuse the debugging methodology because of some ideological reasons. The main argument is that the use of the debugger is the use of cut and try method. A person noticing the incorrect behavior of the algorithm at some step of its execution brings corrections into it without examining why this error occurred and without thinking about the way he corrects it. If he didn’t guess the right way of correction he will notice it during the next execution of the code and bring new corrections. It will result in lower quality of the code. And the author of this code is not always sure that he understands how it works. Opponents of debugging method offer to replace it with more strict discipline of algorithm development, with the use of functions as small as possible so that their working principles are clear. Besides, they offer to pay more attention to unit-testing and to use logging systems for analysis of the program’s correct work.

There are some rational arguments in the described criticism of debugging systems but, as in many cases, one should weigh everything and not run to extremes. The use of the debugger is often convenient and may save a lot of effort and time.

Causes why the debugger is not so attractive

Bad applicability of the debugger while working with systems processing large data sizes is unfortunately related not to ideological but practical difficulties. We’d like to familiarize the readers with these difficulties to save their time on fighting with the debugging tool when it is of little use, and prompt them to search for alternative solutions.

Let’s study some reasons why alternative means should be used instead of a traditional debugger (for example, one integrated into Visual C++ environment).

1) Slow program execution

Execution of a program under a debugger processing millions and billions of elements may become practically impossible because of great time costs. Firstly, it is necessary to use debugging code variant with optimization turned off and that already slows down the speed of algorithm’s work. Secondly, in the debugging variant there occurs allocation of larger memory size for control of going out of the arrays’ limits, memory fill during allocation/deletion etc. This slows down the work even more.

One can truly notice that a program may be debugged not necessarily at large working data sizes and one may manage with testing tasks. Unfortunately, this is not so. An unpleasant surprise consists in that while developing 64-bit systems you cannot be sure of the correct work of algorithms, testing them at small data sizes instead of working sizes of many GB.

Here you are another simple example demonstrating the problem of necessary testing at large data sizes.

#include <vector>
#include <boost/filesystem/operations.hpp>
#include <fstream>
#include <iostream>

int main(int argc, char* argv[])
{
	std::ifstream file;
	file.open(argv[1], std::ifstream::binary);
	if (!file)
		return 1;
	boost::filesystem::path
		fullPath(argv[1], boost::filesystem::native);
	boost::uintmax_t fileSize =
		boost::filesystem::file_size(fullPath);
	std::vector<unsigned char> buffer;
	for (int i = 0; i != fileSize; ++i)
	{
		unsigned char c;
		file >> c;
		if (c >= 'A' && c <= 'Z')
			buffer.push_back(c);
	}
	std::cout << "Array size=" << buffer.size()
		<< std::endl;
	return 0;
}

On the 64-bit system we’d like to process files of larger size as there is no limit of the array’s size of 2 GB. Unfortunately, the program is written incorrectly from the point of view of LLP64 data model (see table 1) used in the 64-bit Windows version. The loop contains int type variable whose size is still 32 bits. If the file’s size is 6 GB, condition "i != fileSize" will never be fulfilled and an infinite loop will occur.

This code is mentioned to show how difficult it is to use the debugger while searching for errors which occur only at a large memory size. On getting an eternal loop while processing the file on the 64-bit system you may take a file of 50 bites for processing and watch how the functions works under the debugger. But an error won’t occur at such data size and to watch the processing of 6 billion elements under the debugger is impossible.

Of course, you should understand that this is only an example and that it can be debugged easily and the cause of the loop may be found. Unfortunately, this often becomes practically impossible in complex systems because of the slow speed of the processing of large data sizes.

To learn more about such unpleasant examples see articles “Forgotten problems of 64-bit program development” [3] and “20 issues of porting C++ code on the 64-bit platform” [4].

2) Multi-threading

The method of several instruction threads executed simultaneously for speeding up the processing of large data size has been used for a long time and rather successfully in cluster systems and high-performance servers. But only with the appearance of multicore processors on market, the possibility of parallel data processing is being widely used by application software. And the urgency of the parallel system development will only increase in future.

Unfortunately, it is not simple to explain what is difficult about debugging of parallel programs. Only on facing the task of searching and correcting errors in parallel systems one may feel and understand the uselessness of such a tool as a debugger. But in general, all the problems may be reduced to the impossibility of reproduction of many errors and to the way the debugging process influences the sequence of work of parallel algorithms.

To learn more about the problems of debugging parallel systems you may read the following articles: “Program debugging technology for machines with mass parallelism” [5], "Multi-threaded Debugging Techniques" [6], "Detecting Potential Deadlocks" [7].

The difficulties described are solved by using specialized methods and tools. You may handle 64-bit code by using static analyzers working with the input program code and not demanding its launch. Such an example is the static code analyzer Viva64 [8].

To debug parallel systems you should pay attention to such tools as TotalView Debugger (TVD) [9]. TotalView is the debugger for languages C, C++ and Fortran which works at Unix-compatible operating system and Mac OS X. It allows to control execution threads, show data of one or all the threads, can synchronize the threads through breakpoints. It supports also parallel programs using MPI and OpenMP.

Another interesting application is the tool of multi-threading analysis Intel® Threading Analysis Tools [10].

Use of a logging system

All the tools both mentioned and remaining undiscussed are surely useful and may be of great help while developing high-performance applications. But one shouldn’t forget about such time-proved methodology as the use of logging systems. Debugging by logging method hasn’t become less urgent for several decades and still remains a good tool about which we’ll speak in detail. The only change concerning logging systems is growing demands towards them. Let’s try to list the properties a modern logging system should possess for high-performance systems:

• The code providing logging of data in the debugging version must be absent in the output version of a software product. Firstly, this is related to the increase of performance and decrease of the software product’s size. Secondly, it doesn’t allow to use debugging information for cracking of an application and other illegal actions.
• The logging system’s interfaces should be compact not to overload the main program code.
• Data writing should be as fast as possible in order to bring minimal changes into time characteristics of parallel algorithms.
• The output log should be understandable and analyzable. There should be a possibility to divide information gained from different threads and also to vary the number of its details.

A logging system which meets these requirements allows to universally solve the task of debugging parallel algorithms and also to debug algorithms processing large data arrays.

We won’t give a particular example of a logging system’s code in this article. It is hard to make such a system universal as it depends on the development environment very much, as well as the project’s specificity, the developer’s wishes and so on. Instead, we’ll touch upon some technical solutions which will help you to create a convenient and effective logging system if you need it.

The simplest way to carry out logging is to use the function similar to printf as shown in the example:

  int x = 5, y = 10;
  ...	
  printf("Coordinate = (%d, %d)\n", x, y);

#ifdef DEBUG_MODE
  #define WriteLog printf
#else
  #define WriteLog(a)
#endif

  WriteLog("Coordinate = (%d, %d)\n", x, y);

Unfortunately, when compiling the un-debugging version in Visual C++ environment we get a warning message: "warning C4002: too many actual parameters for macro 'WriteLog'". We may turn this message off but it is a bad style. We can rewrite the code as follows:

#ifdef DEBUG_MODE
  #define WriteLog(a) printf a

#else
  #define WriteLog(a)
#endif
  WriteLog(("Coordinate = (%d, %d)\n", x, y));

#ifdef DEBUG_MODE
  #define WriteLog printf
#else
  inline int StubElepsisFunctionForLog(...) { return 0; }
  static class StubClassForLog {
  public:
    inline void operator =(size_t) {}
  private:
    inline StubClassForLog &operator =(const StubClassForLog &)
      { return *this; }
  } StubForLogObject;
  
  #define WriteLog \
    StubForLogObject = sizeof StubElepsisFunctionForLog

#endif
  WriteLog("Coordinate = (%d, %d)\n", x, y);

• This code looks complicated but it allows to write single brackets. When the DEBUG_MODE is turned off this code doesn’t matter at all and you can safely use it in critical code sections.

The next modification is to add such parameters to the logging function as the number of the information’s details and its type. The number of details may be defined as a parameter, for example:

enum E_LogVerbose {
  Main,
  Full
};

#ifdef DEBUG_MODE
  void WriteLog(E_LogVerbose,
                const char *strFormat, ...)
  {
    ...
  }
#else
  ...
#endif

WriteLog (Full, "Coordinate = (%d, %d)\n", x, y);

We mentioned that the writing of the debugging information must not influence the speed of the algorithm’s work too much. We can reach this by creating a system of gathering messages, the writing of which occurs in the thread executed simultaneously. The outline of this mechanism is shown on picture 2.

Picture 2. Logging system with lazy data write.

As you can see on the picture the next data portion is written into an intermediate array with strings of fixed length. The fixed size of the array and its strings allows to exclude expensive operations of memory allocation. This doesn’t reduce the possibilities of this system at all. You can just select the strings’ length and the array’s size with spare. For example, 5000 strings of 4000 symbols will be enough for debugging nearly any system. And memory size of 20 MB which is necessary for that is not critical for modern systems, I think you’ll agree with me. But if the array’s overflow occurs anyway, it’s easy to provide a mechanism of anticipatory writing of information into the file.

The described mechanism provides practically instant execution of WriteLog function. If there are offloaded processor’s cores in the system the writing into the file will be virtually transparent for the main program code.

The advantage of the described system is that it can function practically without changes while debugging the parallel program, when several threads are being written into the log simultaneously. You need just to add a process identifier so that you can know later from what threads the messages were received (see picture 3).

Picture 3. Logging system while debugging multithread applications.

The last improvement we’d like to offer is organization of showing the level of nesting of messages at the moment of functions’ call or the beginning of a logical block. This can be easily organized by using a special class that writes an identifier of a block’s beginning into the log in the constructor, and an identifier of the block’s end in the destructor. Let’s try to show this by an example.

Program code:

class NewLevel {
public:
  NewLevel() { WriteLog("__BEGIN_LEVEL__\n"); }
  http://images.gamedev.net/features/programming/vcppIntenseAppNewLevel() { WriteLog("__END_LEVEL__\n"); }
};

#define NEW_LEVEL NewLevel tempLevelObject;
void MyFoo() {
  WriteLog("Begin MyFoo()\n");
  NEW_LEVEL;
  int x = 5, y = 10;
  printf("Coordinate = (%d, %d)\n", x, y);
  WriteLog("Begin Loop:\n");
  for (unsigned i = 0; i != 3; ++i)
  {
    NEW_LEVEL;
    WriteLog("i=%u\n", i);
  }
}

Begin MyFoo()
__BEGIN_LEVEL__
Coordinate = (5, 10)
Begin Loop:
__BEGIN_LEVEL__
i=0
__END_LEVEL__
__BEGIN_LEVEL__
i=1	
__END_LEVEL__
__BEGIN_LEVEL__
i=2
__END_LEVEL__
Coordinate = (5, 10)
__END_LEVEL__

Begin MyFoo()
    Coordinate = (5, 10)
    Begin Loop:
        i=0
        i=1
        i=2
    Coordinate = (5, 10)

Page 4