Combining logging and statistics collection
I have an application which logs notable events out to the console and a text file. Obviously these logs are important as they are both human-readable and easily searchable in a text editor or using a tool like grep on *nix platforms.
However, I also want the ability to derive meaningful statistics from these events, and to be able to show those statistics from within the program. For example, to show how many times a certain event took place between 2 dates.
My first thought was to separate these 2 events, by logging the human readable stuff to a file as normal and putting the events into some sort of separate statistics database (maybe relational,maybe not). But this seems somewhat redundant and difficult to cross-reference.
I could probably write some small tools to perform specific queries on the logfiles, but obviously these wouldn''t be easily extendable.
So my next thought was to perhaps use XML: the ''traditional'' logging would be the tag content, but the tag attributes could hold specific values that could be easily processed in an unlimited number of ways. And it could still be viewed as a normal log file with a decent style sheet. But I expect it would be a bit cumbersome to browse from the *nix command line.
Has anyone had to do anything like this before, and do you have any insights or tips on good ways to combine readable logfiles with effective stat-gathering?
[ MSVC Fixes | STL | SDL | Game AI | Sockets | C++ Faq Lite | Boost | Asking Questions | Organising code files | My stuff ]
How about prefixing each line with a certain number. So prefix lines with ''1'' for event A. Prefix ''2'' for event B..etc...
1 Event A happened.
1 Event A happened.
2 Event B happened.
1 Event A happened.
.
.
..
1 Event A happened.
1 Event A happened.
2 Event B happened.
1 Event A happened.
.
.
..
Or store the data as a CSV file with extra fields to make it more human readable, something like
7163, "Error Message to Human", Propery1, Propert2, etc
1204, "Window Creation Succesfull", X, Y, Purpose, etc
[edited by - Extrarius on March 16, 2003 11:40:05 AM]
7163, "Error Message to Human", Propery1, Propert2, etc
1204, "Window Creation Succesfull", X, Y, Purpose, etc
[edited by - Extrarius on March 16, 2003 11:40:05 AM]
yazan3k: I already have some basic prefixes, as text between 2 square brackets, eg:
[DEATH] Player ''Superman'' died in room 1002.
[QUIT] Player ''Batman'' has quit the game.
[BUG] NULL argument passed to dumb_function(): ignoring
Obviously these are helpful, but I need to be able to filter on other criteria too, and the problem is actually generating the numerical data from them on demand.
Extrarius: CSV format is decent as far as storing the data goes, but I wouldn''t agree it''s very human-readable compared to a standard logfile, even with an explicit message field in there. Therefore I''d say it tips the balance towards being easy to process (and therefore get stats from) but more difficult to use as a plain logfile, and therefore may not be too much help.
[ MSVC Fixes | STL | SDL | Game AI | Sockets | C++ Faq Lite | Boost | Asking Questions | Organising code files | My stuff ]
[DEATH] Player ''Superman'' died in room 1002.
[QUIT] Player ''Batman'' has quit the game.
[BUG] NULL argument passed to dumb_function(): ignoring
Obviously these are helpful, but I need to be able to filter on other criteria too, and the problem is actually generating the numerical data from them on demand.
Extrarius: CSV format is decent as far as storing the data goes, but I wouldn''t agree it''s very human-readable compared to a standard logfile, even with an explicit message field in there. Therefore I''d say it tips the balance towards being easy to process (and therefore get stats from) but more difficult to use as a plain logfile, and therefore may not be too much help.
[ MSVC Fixes | STL | SDL | Game AI | Sockets | C++ Faq Lite | Boost | Asking Questions | Organising code files | My stuff ]
I'm simply adding a 128 bit identifier before each line. It encodes the error in machine readable form, incl. source file (table lookup) and source line. After the identifiers follows the date and time (both machine and human readable) and finally a human readable description. A log line look like this:
[0x12345678, 0xABCDEFAB, 0x12345678, 0xABCDEFAB] {16.03.03 - 19:05:10} [render.cpp, 520] CRenderCore ctor: can't connect to device driver.
It creates pretty long log lines, but contains all info you need, and is both machine and human readable. Since the ID field is always the same width (in characters), you can easily strip it away.
[edited by - Yann L on March 16, 2003 1:11:22 PM]
[0x12345678, 0xABCDEFAB, 0x12345678, 0xABCDEFAB] {16.03.03 - 19:05:10} [render.cpp, 520] CRenderCore ctor: can't connect to device driver.
It creates pretty long log lines, but contains all info you need, and is both machine and human readable. Since the ID field is always the same width (in characters), you can easily strip it away.
[edited by - Yann L on March 16, 2003 1:11:22 PM]
Thinking about it, I expect the best methods are just going to be some sort of prefix (as also used in FTP and HTTP, now I think about it), or to use XML if I can be sure of having a browser available. Any truly versatile data handling is going to have to be preprocessed into some sort of database anyway. Any other suggestions are still welcome though.
[ MSVC Fixes | STL | SDL | Game AI | Sockets | C++ Faq Lite | Boost | Asking Questions | Organising code files | My stuff ]
[ MSVC Fixes | STL | SDL | Game AI | Sockets | C++ Faq Lite | Boost | Asking Questions | Organising code files | My stuff ]
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement