Skip to main content
GameDev.net gamedev.net
🔒 Locked

Loggers acceptable use of global variables?

Started by hymerman Nov 28, 2006 at 3:13 PM 23 replies 13.5k views
Original Post
hymerman
hymerman
Hi all, The next two paragraphs are waffly background story, if your time is far too valuable to waste just skip them :) I've been using a logging system in previous projects that uses a singleton, along with macros for easy access to the logger. This worked a charm, was easy to use and allowed me to compile out low-level messages easily due to the macros. I generally hate both macros and singletons, but was informed that both are acceptable when it comes to loggers. Now, I've just plonked it into a new project that uses dynamically linked code, and have found that singletons just don't work across shared object boundaries. I figured one option was to make the logger an instance variable of some persistent object (we have a Core class that seems suitable), but this makes the macros completely useless; there is no guarantee that the logger can be accessed the same way everywhere. I figured the next best thing would be lots of static methods in the logger, but my spider sense tells me this would suffer the same problems as the singleton implementation. So, my final chance seems to be a global variable to hold 'the' instance of the logger. This way, I can use macros to make life easier, and access is easy. Now, my question is: is a logger an acceptable use of a global variable? Secondary to that, will it work across shared object boundaries? If not, what would everyone suggest to do in my situation? Lastly, would static methods suffer the same fate as a singleton implementation, and if not are they preferable to global variables? Note that when I say 'better' I mean in a general sense (usual case - I'm sure you've all used loggers before), since I know it'll be a subjective thing. Thanks in advance for your input, folks!
ApochPiQ
ApochPiQ
Eurgh. Loggers seem to be a universal excuse to regress back to sloppy coding practices. I for one don't buy the argument that a logger is somehow magical and immune to the reasoning that makes macros and singletons disgusting evil [smile]


First of all, objects across dynamic link boundaries are dangerous no matter what. Unless you can guarantee that the code was compiled the exact same way in each linked module, you will run into icky problems. That said, if you can make such a guarantee, passing around object pointers/references shouldn't cause you any trouble. Obviously the shared code has to be available to each linked module at compile time.

This is going to be nasty. Honestly, the question you need to ask yourself is does a logger need to be an object in the first place. The compulsion to cram anything and everything into a class is not necessarily good. Frankly, I rarely have any logging capability so sophisticated that a class makes sense. At the most, I have a bitmask that filters the output, and a single function (possibly overloaded) that accepts strings. This can be trivially accomplished using a single global function. I'd recommend putting it into a namespace for organizational purposes, but that's up to you.

Turning off logging is easy:
void Log(const std::string& line){#ifdef _DEBUG  // Write line#endif}


Any decent compiler will have the sense to optimize away the empty function call in a release build. Substitute _DEBUG with ACTIVATE_LOGGING or whatever suits your fancy. Obviously in a real situation it's more convenient to overload Log to allow C strings and other data types in certain cases, but that's easy enough.

I typically have a Debug namespace that contains the LogOutput functions. A static variable at file-scope in the Log function module tracks the filter bitmask. (Obviously if doing dynamic linking you'd need an alternate method.)


If you really need shared logging state between dynamically linked address spaces, for instance if you have multiple threads or need to track filtering options at runtime, I'd recommend having a separate dynamically linked blob that does nothing but logging. Then, use the typical dynamic-link "export/import" interface to use the logging from the rest of your code. This will ensure you don't have to sync state across address boundaries or other such messes, and really dramatically simplifies locking for multithreaded logging.
hymerman
hymerman
Get rid of the logger class... well, that's certainly one way of tackling the problem ;) Can you honestly say that there's no point implementing an all singing, all dancing XML based logger with stylesheets and lots of other buzzwords, over a function that writes lines to a text file? I appreciate that the benefit may be less than the effort required to make it, but let's say I've already implemented this super duper logger, and now just want to make it work with dynamically linked code? Is there any point replacing it with std::ofstream?

I'm intrigued about the last think you said though. Would it be possible to do this without having to compile a separate DLL/so? The logger class is actually currently getting compiled into a static library, which is then linked with another application which uses the code, it would be a pain in the backside to have to supply static and dynamic libraries together.
ApochPiQ
ApochPiQ
To be utterly honest, I've never seen any XML logging that I felt represented a real benefit over raw text. However, that's not to say that such a thing is impossible [smile]

I usually see overengineered logging in two particular cases: where huge statistical reports are generated, in which case a proper reporting tool is what should be used and not a logging tool; and deeply sophisticated traces of code flow which show call stack depth and other things with indentation, nested scopes (typical of XML), and so on. Occasionally there is some annotation added for filtering or whatnot.

In the case of traces, there is no magic easy solution. Doing reliable code traces across dynamic link boundaries is quite difficult; doing it across threads is orders of magnitude more difficult, at least if you want your code to run at a reasonable speed. This is IMHO best left to specialized debugging tools, or simply not done for dynamically linked projects. Have each linked module do its own tracing and include annotation as appropriate for calls across boundaries; in my experience the difficulty of getting a reliable trace across boundaries is simply too huge to justify the miniscule benefits.

As far as annotations go, e.g. tagging entries for filtering or whatever - this is just a matter of not using the right tools. A good log viewer can do regular-expression based filtering, and most logging systems can have runtime on-the-fly filtering using bitmasks or other schemes as I described before. So I don't really see XML as a huge benefit for annotation here. Maybe I'm just too used to the Unix style of handling such things, but to me it's easier to write raw text and then prettify it with some regexps or a postprocess tool, rather than muck about with XML logging, XSLT, and all the other acronym cruft involved. That's subjective, though, obviously.


If you already have the class... well, that changes the argument a bit, obviously. No, there is no way to get shared logging across multiple boundaries if you statically link the logger to each dynamic module. That's not the way link boundaries work. If your logger lives in a module, it lives there and only there - the only way to get state information into another module is to send it explicitly via function calls. This means you have to synchronize your logger's state across each loaded module every time the state changes, and I hope it is obvious why that is just not a nice thing to have to do.

The cleanest solution is to keep your logger in its own module entirely, and call into that module via dynamic linking from any other module that needs it. If you're spending the time to create XML loggers, you presumably have other debugging/shared stuff that would make sense in such a module.


Optionally, you could rethink the use of dynamic linking, or you could rethink the need to have all dynamically linked modules log into the same target. Hell, if you're using XML, you should have no problem taking multiple log files (one from each module) and merging them - that's even simple enough to do with a plain text log format if you use timestamping.

Simply put, the solution requires you to change something. The pieces you've got don't fit together nicely.


As a footnote: if you happen to be using purely Microsoft compilers, can guarantee that all modules are built by the same compiler version and with compatible settings, and can ensure that you have no threading issues, you can use __declspec(dllexport) to share state in certain limited cases. It has some serious caveats though, and obviously only works if you tightly control the way modules are built - which defeats much of the point of dynamic linking. A .DEF file can work around some of the nuisances of dllexport, such as C++ name mangling (which is notoriously vendor-dependent) but has some annoyances of its own. This still requires that all modules be built with the same class code; the only really portable and truly "dynamic" way to share state is still by function calls.
Extrarius
Extrarius
Quote:
Original post by ApochPiQ
[...]the only really portable and truly "dynamic" way to share state is still by function calls.
Of course, that doesn't mean you need an ugly oldschool-C API for sharing state - you could always take one of the very few good lessons from COM and share an abstract class and expose a function to create new or obtain existing instances of a concrete implementation. Since all the functions are referenced using vtables and the functions are all the 'other' places know about, you don't need to worry about every module knowing all the details (as long as the interface remains the same)
"Walk not the trodden path, for it has borne it's burden." -John, Flying Monk
hymerman
hymerman
Quote:
Original post by Extrarius
Of course, that doesn't mean you need an ugly oldschool-C API for sharing state - you could always take one of the very few good lessons from COM and share an abstract class and expose a function to create new or obtain existing instances of a concrete implementation. Since all the functions are referenced using vtables and the functions are all the 'other' places know about, you don't need to worry about every module knowing all the details (as long as the interface remains the same)


I don't know a massive amount about C++, and next to nothing about COM. I'd like to know more about this though, if it leads to a solution. I'd appreciate any links you could give me, or of course any code ;)

If not, I think I'm just going to ditch this thing, it seems to be more trouble than it's worth. I did like the support for attaching different logger subclasses, which would eventually tie in with the in-application console window, but I guess I'll just have to do without if it's going to be this much of a pain in the arse. Thanks for your help anyway, ApochPiQ. I'd rate you up but I've already done so from some other thread :)
Bregma
Bregma
Quote:
Original post by hymerman
Get rid of the logger class... well, that's certainly one way of tackling the problem ;) Can you honestly say that there's no point implementing an all singing, all dancing XML based logger with stylesheets and lots of other buzzwords, over a function that writes lines to a text file?


Okay, I'm going to have to don my evangelist hat here because this is one of my pet peeves.

Get rid of the logger class. There is no advantage to implementing an all-singing, all-dancing, full buzzword-compliant port of a Java-oriented logging facility to C-with-classes. In fact, it costs you a lot in terms of development time (and time is money) and support (support is a cost centre) and maintanability.

The C++ language, believe it or not, comes with the perfect logging facility. It's always there, it works well with most software, and it does the right thing when you need it. It's called std::clog and for error logging, std::cerr.

So you find it doesn't provide the fine-grained functionality you need? Simple solution: it's highly and easily estensible. You can write a suite of manipulators to allow you to set, say, the severity and category of a message. Lots of examples out there, many in the standard library itself (you don't have to go far for code to model).

Having problems getting the client code to determine the output format and destination? Redesign: This isn't something client code should be concerned with.

Want to redirect logging output to a window updated in realtime? Replace the streambuf with one of your own. Need to filter output based on some criteria? Use an external tool, that's more appropriate.

So, is it okay to use globals for logging? Yes, as long as it's one of the ones found in the standard library.

Some day I'll get around to publishing my set of extensions to the standard logging streams so others don't have to worry their fluffly little heads about implementation. If anyone's interested let me know sooner rather than later and I'll move it up in the priority queue.

--smw
Stephen M. Webb
Professional Free Software Developer
Nitage
Nitage
Compare and contrast:

Function:
Log(__FUNC__,
std::string("X = ") + boost::lexical_cast(X),
__FILE__,
__LINE__);

Macro:
Log("X = " << X);
jpetrie
jpetrie
Just to add my two cents:

Once, many years back, I wrote a very powerful C++ logging library. It supported raw text, RTF or HTML output. It had a hierarchical tree structure (you could log to /graphics or to graphics/vistests, et cetera) that you could dynamically define and update, switch on or off, et cetera. It had a lot of the functionality that's now in log4j or was in the (proposed but rejected) Boost Log Library. It didn't use singletons (in fact, it was the first time I wrote a singletonless logging system).

I used it for many years. And out of all that fancy functionality I implemented because I read about it or thought it would be cool or useful (especially that hierarchical cruft), do you know how much I used?

On average, two functions: the log scope constructor, and the overloaded operator<<. Something like five percent of the actual functionality.

I have since seen the error of my youthful ways, and now the logging functionality that I write (when I bother) is quite spartan. Like ApochPiQ, I think its silly that people see logs as an excuse to regress to sloppy development methodology, especially since most of the time that regression is to support features that tend to be useless anyway.
Extrarius
Extrarius
Quote:
Original post by Bregma
[...]It's called std::clog and for error logging, std::cerr.[...]
Is there a standard way to redirect it to send to a different stream? If not, you'll have to use OS- or compiler- specific code to make sure it gets to the console or to a file (since win32 gui programs, for example, don't have a console by default and don't redirect the streams anywhere useful and I imagine the same is true when dealing with n?x windowing systems)
"Walk not the trodden path, for it has borne it's burden." -John, Flying Monk
Zahlman
Zahlman
Quote:
Original post by Extrarius
Quote:
Original post by Bregma
[...]It's called std::clog and for error logging, std::cerr.[...]
Is there a standard way to redirect it to send to a different stream? If not, you'll have to use OS- or compiler- specific code to make sure it gets to the console or to a file (since win32 gui programs, for example, don't have a console by default and don't redirect the streams anywhere useful and I imagine the same is true when dealing with n?x windowing systems)


Shouldn't need any of that. Like Bregma said: just replace the streambuf.
hymerman
hymerman
Okay, I'm now very interested in this idea of using std::cerr and std::cout, could any of you point me in the direction of some kind of documentation that'll tell me how to modify them (replacing streams and whatnot)? Or better yet, Bregma, could you post a little code, if it's not too messy/big/much trouble/against your wishes? I see the error of my ways now, I'm going to ditch my current logging system. It was buggy anyway.
Bregma
Bregma
Quote:
Original post by hymerman
Okay, I'm now very interested in this idea of using std::cerr and std::cout, could any of you point me in the direction of some kind of documentation that'll tell me how to modify them (replacing streams and whatnot)?


Hmm, well, I've been intending to write a series of articles on my website but as usual time and the elements have conspired against me. I'll give a few basics, though.

First off, I use a custom streambuf. Here's the header file for a class that just wraps whatever is already in a standard stream and prepends a severity, a timestamp, and an optional facility. Just as an example.
#include <streambuf>#include <string>namespace Bregma{	/**	 * The DebugStreambuf provides a concrete basic_streambuf that appends a	 * timestamp and process ID (or thread ID) at the start of each new line.	 *	 * This streambuf is an unbuffered streambuf.  It is not capable of input.	 */	template<typename Char, typename Traits = std::char_traits<Char> >		class DebugStreambuf		: public std::basic_streambuf<Char, Traits>		{		public:			typedef typename std::basic_streambuf<Char, Traits>::traits_type traits_type;			typedef typename std::basic_streambuf<Char, Traits>::int_type    int_type;		public:			/**			 * Buffer construction.			 */			DebugStreambuf(std::basic_streambuf<Char, Traits> *pRealBuf);			void setLogLevel(const LogLevel &logLevel)			{ m_logLevel = logLevel; }			void setFacility(const std::string &facility)			{ m_facility = facility; }		protected:			/**			 * Function called by an ostream when it's time to send something out.			 *			 * @param c The value to be written out (generally a single character).			 *			 * @returns A value equal to traits_type::eof() on failure,			 * traits_type::not_eof() on success.			 */			int_type			overflow(int_type c = traits_type::eof());		private:			DebugStreambuf(const DebugStreambuf&);			DebugStreambuf& operator=(const DebugStreambuf&);		private:			std::basic_streambuf<Char, Traits> *m_pRealBuf;			bool                                m_bAtBeginningOfLine;			LogLevel                            m_logLevel;			std::string                         m_facility;		};} // namespace Bregma

The LogLevel is an enum in another header file. Don't worry about it for now. The important thing is the implementation. Keep in mind this is modified from production code, so any errors are mine during online editing (not sure if I got the braces right, this online editor is awkward).
#include "debugstreambuf.h"#include <sstream>using namespace std;/** * Constructs a basic DebugStream. */template<typename C, typename T>	Bregma::DebugStreambuf<C,T>::		DebugStreambuf(basic_streambuf<C,T>* pRealBuf)		: m_pRealBuf(pRealBuf)		, m_bAtBeginningOfLine(true)		, m_logLevel(kLOG_INFO)		{		}/** * Actual function to move bytes to the logging stream if appropriate. */template<typename C, typename T>	typename Bregma::DebugStreambuf<C,T>::int_type Bregma::DebugStreambuf<C,T>::	overflow(int_type c)	{		int_type retval = traits_type::not_eof(c);		if (!traits_type::eq_int_type(c, traits_type::eof()))		{			if (m_bAtBeginningOfLine)			{				m_bAtBeginningOfLine = false;				basic_ostringstream<C,T> ostr;				// Format and display the level tag.				char tag = '?';				switch (m_logLevel)				{					case kLOG_FATAL:						tag = 'F';						break;					case kLOG_ERROR:						tag = 'E';						break;					case kLOG_WARNING:						tag = 'W';						break;					case kLOG_INFO:						tag = 'I';						break;					case kLOG_VERBOSE:						tag = 'V';						break;					case kLOG_DEBUG:						tag = 'D';						break;					default:						tag = '?';				}				ostr << '-' << tag << '-';				// Format and display the time stamp.				time_t curTime = std::time(NULL);				char timestamp[32];				std::strftime(timestamp,					      sizeof(timestamp),                                              "%Y.%m.%dT%H:%M:%S",					      localtime(&curTime));				ostr << timestamp;				// Format and display the facility.				if (!m_facility.empty())				{					ostr << '[' << m_facility << ']';				}				if (!ostr.str().empty())				{					ostr << ": ";				}				// Send the prefix string out.				const basic_string<C,T>& str = ostr.str();				streamsize sz = m_pRealBuf->sputn(str.c_str(), str.length());				if (sz != str.length())				{					return traits_type::eof();				}			}			// Send the real character out.			retval =  m_pRealBuf->sputc(c);		}		// If the end-of-line was seen, reset the beginning-of-line indicator and		// the default log level.		if (traits_type::eq_int_type(c, traits_type::to_int_type('\n')))		{			m_bAtBeginningOfLine = true;			m_logLevel = kLOG_INFO;		}		return retval;	}


None of the above is publicly available and can live in its own little DLL if necessary (or not). The public interface would be in a header file.
namespace Bregma{	enum LogLevel	{		kLOG_FATAL,		kLOG_ERROR,		kLOG_WARNING,		kLOG_INFO,		kLOG_VERBOSE,		kLOG_DEBUG	};	/**	 * Convert an IOStream to a Bregma logging stream.	 *	 * This can be used to convert, for example, std::cerr into a Bregma logging	 * stream.	 *	 * @param ostr  [IN]  The IOStream to convert.	 * @param level [IN]  The default loglevel cutoff (default is INFO).	 * @param flags [IN]  Flags to toggle various output fields.	 */	void convertToBregmaLogger(std::ostream   &ostr);	/**	 * Setter for the log level cutoff.	 *	 * @param ostr  [IN]  The IOStream for which the log cutoff is to be set.	 * @param level [IN]  The new log level cutoff.	 */	void setLogCutoff(std::ostream &ostr, const LogLevel &level);	/**	 * Manipulator helper for setting the current log level on a debug stream.	 */	class LogLevelSetting	{	public:		LogLevel level() const { return m_level; }	private:		explicit LogLevelSetting(LogLevel level): m_level(level) {}		friend const LogLevelSetting logLevel(LogLevel);	private:		LogLevel m_level;	};	/**	 * Ostream manipulator for setting the current log level.	 */	inline const LogLevelSetting logLevel(LogLevel level)	{		return LogLevelSetting(level);	}	/**	 * Ostream inserter for the log level manipulator.	 *	 * @param ostr [IN] The output stream.	 * @param ls   [IN] The log level setting.	 */	std::ostream& operator<<(std::ostream& ostr, const LogLevelSetting ls);	/**	 * A manipulator helper for seeting the facility.	 */	class LogFacilitySetter	{	public:		const std::string &facility() const { return m_facility; }	private:		LogFacilitySetter(const std::string &facility): m_facility(facility) {}		friend const LogFacilitySetter logFacility(const std::string &facility);	private:		const std::string& m_facility;	};	/**	 * An ostream manipulator for setting the current facility in the log stream.	 */	inline const LogFacilitySetter logFacility(const std::string &facility)	{		return LogFacilitySetter(facility);	}	/**	 * Ostream inserter for the facility manipulator.	 *	 * @param ostr [IN] The output stream.	 * @param ls   [IN] The log facility setter manipulator helper..	 */	std::ostream& operator<<(std::ostream& ostr, const LogFacilitySetter ls);	/**	 * A handy stream to mark the entry and exit of a scope.	 *	 * @param ostr [IN] The output stream.	 * @param ls   [IN] A string.	 */	class ScopeMarker	{	public:		ScopeMarker(std::ostream &ostr, const std::string &what)		: m_ostr(ostr)		, m_what(what)		{ 			m_ostr << m_what << " begins\n";		}		~ScopeMarker()		{			m_ostr << m_what << " ends\n";		}	private:		std::ostream  &m_ostr;		std::string    m_what;	};} // namespace Bregma

Naturally you'll need an implementation for some of the above.
void Bregma::convertToBregmaLogger(ostream        &ostr){	ostr.rdbuf(new DebugStreambuf<char>(ostr.rdbuf()));}ostream &Bregma::operator<<(ostream& ostr, const Bregma::LogLevelSetting ls){	typedef DebugStreambuf<char> Dstr;	Dstr *pDstr = dynamic_cast<Dstr*>(ostr.rdbuf());	if (pDstr)	{		pDstr->setLogLevel(ls.level());	}	return ostr;}ostream &Bregma::operator<<(ostream& ostr, const Bregma::LogFacilitySetter ls){	typedef DebugStreambuf<char> Dstr;	Dstr *pDstr = dynamic_cast<Dstr*>(ostr.rdbuf());	if (pDstr)	{		pDstr->setFacility(ls.facility());	}	return ostr;}

At last, an example of how to use this.
#include "bregmalogger.h"#include <iostream>using namespace std;using namespace Bregma;int main(int, char*[]){  convertToBregmaLogger(cerr);  cerr << logFacility("TEST") << logLevel(kLOG_WARNING) << "This is a demo.\n";}


As I said, I have a whole lot more that needs to be turned into human-readable annotated code (and provided as a downloadable source). I just need a Round Tuit and another 25 hours in the day.

--smw
Stephen M. Webb
Professional Free Software Developer
hymerman
hymerman
Thank you very much, Bregma, you've been "Extremely helpful and/or friendly" and have been rated as such. I look forward to a proper release of this, but I shan't hold my breath; you've been incredibly helpful already. I'll scurry off and adapt this now, and try to explain to my team why I've chosen to drop XML/XSLT/other goodies in favour of std::cerr ;)

Again, thanks :)
hymerman
hymerman
Sorry for the double-post, but I do have one quick question: Is your implementation thread-safe, Bregma? I notice one of your comments seems to indicate it'll work with threads, but I'd like to hear it from the horse's mouth, so to speak :)
Bregma
Bregma
Quote:
Original post by hymerman
Sorry for the double-post, but I do have one quick question: Is your implementation thread-safe?


The implementation I posted, no it's not threadsafe. As it stands, you can run into two threading-related issues: first, output from multiple threads is intermingled and second, an application can actually deadlock during output if a thread has crashed due to double deletions (at least under Linux, because of the mutex in glibc's malloc()/free() implementation). I tell you this from direct experience.

I did not include any of the code that makes this threadsafe because I thought it already complicated enough considering there was no explanation of how it works and because by its nature threading is platform-specific. In the production code I'm using, I have a bunch of threading stuff wrapped in a library but that's just yet more code to pull in and have to explain in a quick example.

There are in fact a number of ways to make the code threadsafe. The simplest is to use a separate streambuf for each thread: as soon as a thread starts, you can replace the std::clog streambuf with a thread-specific ofstreambuf that sends its output to, say, a file with the threadid as a part of its name. No intermingling, no deadlocks, completely threadsafe.

By the way, my example code just wrapped the existing streambuf from stderr. There's no reason why you can't use a different streambuf entirely so you could redirect output to, say, a file, a pipe or a Window, and no reason why this can't be done based on settings in a configuration file. There's also no reason why you can't output in XML so you can later transform the log info using XSLT or whatever today's silver bullet is.

--smw
Stephen M. Webb
Professional Free Software Developer
hymerman
hymerman
Okay Bregma, I'm really sorry for being such a pesky ho of a noob, but I'm having trouble making this thread safe. I'm trying to splice your code together with the only reference I could find, this article:

http://uk.builder.com/programming/c/0,39029981,20279107-1,00.htm

It's horribly written and a bit heavyweight, but it's all I can find! Would it be possible to either post a quick example as to how to make your code thread safe, or to post your production code? I doubt it'll be more confusing than the article above ;)

Of course, I totally understand if you don't want to waste any more time on this, or if you don't want to post your code. Just say so and I'll bugger off :)
Bregma
Bregma
Quote:
Original post by hymerman
Would it be possible to either post a quick example as to how to make your code thread safe, or to post your production code?


Unfortunately I am not at liberty to post the production code at this time and it seems we just hit crunch mode at work so I don't have the time to do a reasonable treatment for public consumption. Solutions for thread safety are of necessity platform-specific, but if you need a hint I suggest using thread-local storage to hold a pointer to a local buffer for each thread and a single mutex to guard the wrapped filebuf, grabbing the mutex and flushing only when you encounter a newline (the latter is already there in the code above).

--smw
Stephen M. Webb
Professional Free Software Developer
hymerman
hymerman
That's okay, I figured my question was particularly cheeky :) You've been a great help anyway. Good luck with the crunch.
Washu
Washu
Quote:
Original post by Bregma
Hmm, well, I've been intending to write a series of articles on my website but as usual time and the elements have conspired against me. I'll give a few basics, though.

That's usually what happens.
Quote:

... bunch of stuff ...

Just a comment though: You should be restoring the original streambuf that cerr started with (rdbuf returns a pointer to the old streambuf).
In time the project grows, the ignorance of its devs it shows, with many a convoluted function, it plunges into deep compunction, the price of failure is high, Washu's mirth is nigh.

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.

Sign in to reply to this topic.