Archived

This topic is now archived and is closed to further replies.

Kylotan

Combining logging and statistics collection

Recommended Posts

Kylotan    10002
I have an application which logs notable events out to the console and a text file. Obviously these logs are important as they are both human-readable and easily searchable in a text editor or using a tool like grep on *nix platforms. However, I also want the ability to derive meaningful statistics from these events, and to be able to show those statistics from within the program. For example, to show how many times a certain event took place between 2 dates. My first thought was to separate these 2 events, by logging the human readable stuff to a file as normal and putting the events into some sort of separate statistics database (maybe relational,maybe not). But this seems somewhat redundant and difficult to cross-reference. I could probably write some small tools to perform specific queries on the logfiles, but obviously these wouldn''t be easily extendable. So my next thought was to perhaps use XML: the ''traditional'' logging would be the tag content, but the tag attributes could hold specific values that could be easily processed in an unlimited number of ways. And it could still be viewed as a normal log file with a decent style sheet. But I expect it would be a bit cumbersome to browse from the *nix command line. Has anyone had to do anything like this before, and do you have any insights or tips on good ways to combine readable logfiles with effective stat-gathering? [ MSVC Fixes | STL | SDL | Game AI | Sockets | C++ Faq Lite | Boost | Asking Questions | Organising code files | My stuff ]

Share this post


Link to post
Share on other sites
yazan3k    122
How about prefixing each line with a certain number. So prefix lines with ''1'' for event A. Prefix ''2'' for event B..etc...

1 Event A happened.
1 Event A happened.
2 Event B happened.
1 Event A happened.
.
.
..


Share this post


Link to post
Share on other sites
Extrarius    1412
Or store the data as a CSV file with extra fields to make it more human readable, something like

7163, "Error Message to Human", Propery1, Propert2, etc
1204, "Window Creation Succesfull", X, Y, Purpose, etc

[edited by - Extrarius on March 16, 2003 11:40:05 AM]

Share this post


Link to post
Share on other sites
Kylotan    10002
yazan3k: I already have some basic prefixes, as text between 2 square brackets, eg:

[DEATH] Player ''Superman'' died in room 1002.
[QUIT] Player ''Batman'' has quit the game.
[BUG] NULL argument passed to dumb_function(): ignoring

Obviously these are helpful, but I need to be able to filter on other criteria too, and the problem is actually generating the numerical data from them on demand.


Extrarius: CSV format is decent as far as storing the data goes, but I wouldn''t agree it''s very human-readable compared to a standard logfile, even with an explicit message field in there. Therefore I''d say it tips the balance towards being easy to process (and therefore get stats from) but more difficult to use as a plain logfile, and therefore may not be too much help.

[ MSVC Fixes | STL | SDL | Game AI | Sockets | C++ Faq Lite | Boost | Asking Questions | Organising code files | My stuff ]

Share this post


Link to post
Share on other sites
Yann L    1802
I'm simply adding a 128 bit identifier before each line. It encodes the error in machine readable form, incl. source file (table lookup) and source line. After the identifiers follows the date and time (both machine and human readable) and finally a human readable description. A log line look like this:

[0x12345678, 0xABCDEFAB, 0x12345678, 0xABCDEFAB] {16.03.03 - 19:05:10} [render.cpp, 520] CRenderCore ctor: can't connect to device driver.

It creates pretty long log lines, but contains all info you need, and is both machine and human readable. Since the ID field is always the same width (in characters), you can easily strip it away.

[edited by - Yann L on March 16, 2003 1:11:22 PM]

Share this post


Link to post
Share on other sites
Kylotan    10002
Thinking about it, I expect the best methods are just going to be some sort of prefix (as also used in FTP and HTTP, now I think about it), or to use XML if I can be sure of having a browser available. Any truly versatile data handling is going to have to be preprocessed into some sort of database anyway. Any other suggestions are still welcome though.

[ MSVC Fixes | STL | SDL | Game AI | Sockets | C++ Faq Lite | Boost | Asking Questions | Organising code files | My stuff ]

Share this post


Link to post
Share on other sites