Archived

This topic is now archived and is closed to further replies.

Kylotan

Using streams without inheritance

Recommended Posts

Recently I''ve been thinking about how useful stringstreams are for formatting text. One of my projects is a MUD which is almost exclusively about sending text to the players, which then ends up sent down a socket. The problem is this: an ostringstream is 136 bytes and a stringbuf is 96 bytes, a 240 byte addition that almost doubles the size of my Character objects. Now, if this was just for players, that would be fine, as I don''t expect more than 100 players in the game at once. But the interface has to be standardised across Players and NonPlayers, so any streaming has to be done in the base Character class. This would mean incurring that 232 byte penalty on tens of thousands of characters, meaning we''re talking about several extra megabytes of additional memory needed for this ''convenience''. I have an idea (with 2 potential implementations) for how to approximate this. There would be a single templatised << operator for Character that takes any type, and passes it to an ostringstream in order to get the string representation of that type. This is appended to an internal string buffer. Basically, the ostringstream is just used to convert any type to a string, based on prior formatting rules set by stream manipulators. Implementation 1: Just use a single static ostringstream for all instances of Character. This would work reasonably fine now, but I intend adding some multithreading later. I would then need to add some sort of mutex/critical section to it, and I can imagine that might get slow? (I hear conflicting things on such issues, and I don''t know enough about synchronisation primitives to know for sure.) Implementation 2: Create the ostringstream on the stack each time it''s needed. I can imagine that the constructor and destructor for an ostringstream are not trivial and therefore this would be inefficient too. Bear in mind that sending text to a character is done very very often. So I can''t afford for it to be a slow operation. (It doesn''t need to be hyper-efficient, but I can''t disregard efficiency entirely). So does anyone have any other ideas for an interface that provides all the versatility of iostreams, without the large memory footprint? [ MSVC Fixes | STL | SDL | Game AI | Sockets | C++ Faq Lite | Boost ]

Share this post


Link to post
Share on other sites
Implementation 3: @$@#$#$ xxstream and implement your own thread-compatible unbloated objects?

Don''t you generally pass xxstream _references around, meaning there''s very few actual xxstream objects?

I''d construct one stream object for input and one for output, and have the socket class suck/stuff those streams.

Am I missing something?

Oh good lord, no you can''t have two streams for every PC and NPC.
One for the console output, and one for the console input.
Maybe a small pool of them on the server side if you really wanted to use them for socket streaming.

Try std::string or std::vector<char>

Share this post


Link to post
Share on other sites
The idea is that each character has its own buffer area that receives formatted output. I could write to thousands of characters before any of them have that buffer flushed to a socket (or simply emptied, in the case of an NPC). This buffer area would be analogous to the .str() member function of a stringstream. It''s useful to be able to use standard functionality like the width manipulators and so on, and implementing these all over again seems like a waste of time. Currently I just have a few overridden << operators that handle the basic types, but I have to manage the buffer myself, as well as having no support for setting the width of a field since I don''t have all the same functionality that a normal ostream would have.

[ MSVC Fixes | STL | SDL | Game AI | Sockets | C++ Faq Lite | Boost ]

Share this post


Link to post
Share on other sites
Could you overridd the stream operators in the PC/NPC classes write to one stream singleton used to transmit data on the socket?

That way you could get a pointer to the (N)PC you were talking to
pTalkingTo << whatTheyTyped;
and it''d be serialized into the single outgoing stream buffer with whatever additional information is need to identify the target

Share this post


Link to post
Share on other sites
Sorry if I misunderstood.

If your character needs to store an buffer, then it shouldn''t know about formatting at all.

A character should store the buffer using std::string (or whatever) and takes a input string to add to it''s internal buffer. Formatting the string is done at the application level rather on a ostringstream instance than writing directly to the Char class.

  
class Char
{
void Add( const char *str ) { buffer += str; }
private:
std::string buffer_;
};

// at application level, use a single instance of stringstream

// to format the string. Just remember to clear it appropriately

Share this post


Link to post
Share on other sites
The main problem with using a singleton stream is that it doesn''t store state. I can give each Character their own buffer represented as a string, that''s no problem. But iostreams store other state, such as the formatting. Although it''s true that a Character doesn''t really need to know about most formatting, this needs to be stored somewhere between calls. As I see it, this either has to be in the character (incurring a size overhead) or in the calling function (incurring a code overhead). I really don''t want to be declaring ostringstreams everytime I want to write data to a Character. That would be literally hundreds of instances in the code.

At the moment, I am doing my formatting via a function that takes a string and returns it padded it to the left or right with spaces. This works for most applications, but is ugly.

[ MSVC Fixes | STL | SDL | Game AI | Sockets | C++ Faq Lite | Boost ]

Share this post


Link to post
Share on other sites
  
#include <iostream>
#include <iomanip>
using namespace std;

ostream& my_manip( ostream &os )
{
int length = 3;
for ( int i = 0; i < length; ++i )
os << " ";
return os;
}

int main()
{
cout << my_manip << "Hello" << endl;
return 0;
}


I was thinking of having the manipulator taking an argument that control the formatting state( ie. take a variable length ).

But it is seems that creating manipulators with arguments is non portable. VC++ uses macros to accomplish the tasks, some platforms uses template specialization.

Since this is not ideal, perhaps each Char object holds a state object which holds formatting state and the state class overloads the << operator.

Share this post


Link to post
Share on other sites
I may not fully understand your description of the problem, but when do you need the formatting of the stringstream?

It seems to me that you can have the Character class store the data in any format it wants and then have it be streamable - eg some virtual write/read methods

reading and generating new objects on the receiving end can be troublesome, but is not that hard of a problem to overcome.

So why not apply the formatting during transmission - e.g. create a custom stream class that sends the data to the socket.

Then create a special form of streambuf that worries about ''packaging'' the data to effecient sizes for transmission.

Share this post


Link to post
Share on other sites
I need the formatting of the stringstream in cases like this:

// nicely formatted table
for (allCharacters)
myCharacter << setw(10) << bigInt << setw(5) << littleInt << endl;
end for

Apologies for the pseudocode. I might also find the fill manipulator, the left/right formatting ones, and precision useful.

Storing and transmitting the data is not the problem. For all intents and purposes, it is stored as plain text and sent as plain text. The problem is getting a clean interface to write to a character that allows as much formatting as I would like. Currently I have to do it this way :
myCharacter << ToString(bigInt, 10) << ToString(littleInt, 5) << "\n";

It would be nice to be able to use the standard streams method instead of having to write a load of functions that convert values to strings, especially since all the functionality I need is already there and working.

This isn''t a "I can''t get my code working" problem. This is just a "is there any way I can do it this way" issue. I prefer not to replicate standard library functionality if at all possible, so I thought I would ask to see if anyone had any ideas.

[ MSVC Fixes | STL | SDL | Game AI | Sockets | C++ Faq Lite | Boost ]

Share this post


Link to post
Share on other sites
Let me know if this helps:

  
// dependencies

template <T> class pool;
class socket;
class socket::multi;
using std::stringstream;
using std::istream;
using std::ostream;
using std::setw;
 
socket::multi socketmultibuffer;
 
class isocketstream
{
string& buffer;
stringstream stream;
 
public:
socketstream () : buffer (socketmultibuffer.get_input ()) {}
~socketstream () { socketmultibuffer.release_input (&buffer); }
 
operator istream& () { return stream; }
};
 
class osocketstream
{
string& buffer;
stringstream stream;
 
public:
osocketstream () : buffer (socketmultibuffer.get_output ()) {}
~osocketstream () { socketmultibuffer.release_output (&buffer); }
 
operator ostream& () { return stream; }
};
 
typedef pool <osocketstream> output_stream_pool;
typedef pool <isocketstream> input_stream_pool;
 
output_stream_pool output_streams (20);
input_stream_pool input_streams (20);
 
template <class T, pool <T>& P > class pooled_object
{
T& t;
 
public:
pooled_object () : t (P.create ()) {}
~pooled_object () { P.destroy (t); }
 
operator T& () { return t; }
};
 
typedef pooled_object <osocketstream, output_streams> pooled_ostream;
typedef pooled_object <isocketstream, input_streams> pooled_istream;
 
class character
{
int big, small;
 
public:
void send () { serialize (pooled_istream ()); }
void receive () { serialize (pooled_ostream ()); }
 
protected:
virtual void serialize (ostream& out)
{
out << setw(10) << big << setw(5) << small;
}
 
virtual void serialize (istream& in)
{
in >> big >> small;
}
};
 
class player
{
int x, y;
 
protected:
virtual void serialize (ostream& out)
{
character::serialize(out);
out << x << y;
}
 
virtual void serialize (ostream& in)
{
character::serialize(in);
in >> x >> y;
}
};


The pool template manages a pool of type T and allows you to reserve an element for your own use and release it back to the system when you are done. socket is a class that encapsulates socket operations. socketmultibuffer is an instance of a class that adapts the simple socket class to add support for multiple input/output buffers, then encapsulates the combining of the multiple input/output streams them into a single input/output stream for send/receive using the socket class interface. socketmultibuffer manages multiple buffers, which are exposed by the socketmultibuffer.get_input/get_output and socketmultibuffer.release_input/release_output functions.

The isocketstream and osocketstream abstract socketmulti's support for multiple buffers and provide a standard istream/ostream interface by containing a stringstream object and exposing its istream/ostream interface. The stringstream object gets its buffer from the socketmultibuffer class, and leaves the details of combining the multiple streams in socketmultibuffer's implementation. The isocketstream/osocketstream classes should be separate, because the socketmultibuffer class allows a buffer to be either input or output, but not both.

Creating a pool of socket streams is easy now, just use the typedef and instantiate the pool with the initial capacity. I used two pools, one for input and one for output. To encapsulate the fact that the streams are in a pool, I placed them into a pooled_object template class which simply creates a pool object of type T from pool instance P, and then created typedefs so that the template parameters were hidden behind type names. The pooled_object's destructor takes care of destroying the object.

Retrieving one of these pooled streams is simplicity itself; you just instantiate a pooled_istream or pooled_ostream, which goes through the chain of adaptation code, eventually reserving a buffer, presenting a standard stream interface, etc. The character send/receive functions are used to serialize data across a socket, so they call the overloaded serialize virtual member functions to perform input and output. Remember to call any immediate base classes versions of serialize from within a derived class's serialize functions, and everything should go smoothly.

Since the pooled_istream and pooled_ostream classes provide a conversion operator to std::istream and std::ostream, you can design the serialize functions to use std::istream and std::ostream, while passing a pooled_istream/pooled_ostream object, in which case the conversion allowed by the operator istream&/ostream& will give you the stringstream interface to the multiple i/o socket buffers.

Simple, eh? The serialize member functions are borrowed from MFC's CArchive class, the adaptors from The C++ Programming Language, Special Edition, and the pool data structure from Magmai's post.

EDIT: Changed behavior of pool slightly to contain capacity for streams rather than actual streams, and also changed the pooled_object typedef so that it works.

Edited by - null_pointer on January 15, 2002 9:39:26 AM

Share this post


Link to post
Share on other sites
Void: Not really. I can do it like that on occasion, but it is precisely that which I am trying to avoid.

null_pointer: uhh... I''m confused I should never have mentioned the sockets, as they are pretty much totally irrelevant to what my problem is. They were just mentioned to justify the fact that I need to buffer the data after formatting it. All the formatting takes place as the data is passed to the Character and is completely done before it is even considered for sending to the socket. Note that this is done for all Characters in the game: all the buffers are filled, then all the buffers are emptied.

[ MSVC Fixes | STL | SDL | Game AI | Sockets | C++ Faq Lite | Boost ]

Share this post


Link to post
Share on other sites
I implemented a system of shared buffers with a pool of shared stream interfaces to those buffers, and hid all that code into two function calls.

When you call character::send (), here is what happens. First, a temporary pooled_ostream object is constructed, which retrieves a osocketstream from the global pool of i/o streams, which in turn retrieves exclusive access to a buffer from socketmultibuffer. (The purpose of socketmultibuffer is to manage multiple shared buffers and inserts/extracts them into/out of the socket''s single buffer at the proper time. Each time you get an input buffer, its contents are extracted from the socket. Each time you release an output buffer, its contents are inserted into the socket class''s single output buffer.)

After the temporary pooled_istream object is constructed, the compiler tries to convert it to an ostream& (in order to pass it to character::serialize (ostream&)) and finds the conversion path pooled_ostream->osocketstream&->ostream&, via the chain of conversion operators. So in effect, character::serialize gets a temporary, standard output stream interface to one of a pool of output buffers (to which it has exclusive access), and all through one constructor call. Note how elegant this is; no details of the shared buffer/stream implementation leak into the character classes - for all they know they are all writing into the same buffer.

After the call to character::serialize has been completed, the temporary object is destroyed, destroying the temporary output stream interface and releasing the buffer for use by another instance of a character-derived class. The chicanery of socket::multi/Xsocketstream/Xput_streams/pooled_Xstream class really just adds transparent support for a socket class using multiple buffers, and then abstracts the fact that it does. Unfortunately, due to the way this code was written, unless you are using multiple threads to write to the socket there is no use for it; no calls to character::send will simultaneously, and thus only one stream/buffer pair will ever be used. Massive overkill.

Anyway, it does demonstrate how you can abstract the storage of buffers and streams as well as abstracting the relation between a stream buffer and a stream interface, while still maintaining the standard stream interfaces and all the functionality they bring with them. However, it does not really overcome the problem you wanted to fix...


quote:
Original post by Kylotan

They were just mentioned to justify the fact that I need to buffer the data after formatting it. All the formatting takes place as the data is passed to the Character and is completely done before it is even considered for sending to the socket. Note that this is done for all Characters in the game: all the buffers are filled, then all the buffers are emptied.


Must all buffers really be in use at the same time, or could you find some scheme for sharing buffers among instances of character?

If in your program each instance of character must have its own buffer, then I am afraid that the ideal solution to your problem does not exist. You must: 1) pay the memory overhead of having a stream for each character, or 2) pay the execution time overhead of constructing a stream for each buffer, or 3) some tradeoff between the two. Magmai''s suggestion of pooled stream interfaces does not actually solve anything if a stream must be attached to a buffer for each character; if you can find some way of sharing the buffers then you may be able to reduce this cost.

The only possibility that I can see for a tradeoff would be if you could process the characters in a different order each time; then you could keep a pool of stringstreams that can remain attached to some character buffers between two calls of the loop code. With this method you could have a tradeoff between the memory overhead and speed overhead by choosing the total number of stringstreams. This is not a very good solution, but it is still an option...I''m getting tired.

Share this post


Link to post
Share on other sites
Now I''m getting confused.

Either u want

- Each character to be formatted differently. It means each character must store formatting states.

That means each character stores a ostringstream (which u mentioned is unacceptable) or the state to set (via manipulators or something else). But you say you don''t want to set the state of the stream each time for each character object.

As null_pointer pointed - How is that possible?

- Each character is to be formatted according to the current state of the stream passed.

But you might need multithreading, so the global stream has to be locked/unlocked and not acceptable.


You say it''s for a MUD project, why are you sending text for non players anyway (by streaming in your base class)? It''s seems to be more of a design issue really.

Share this post


Link to post
Share on other sites
quote:
Original post by Void
Either u want

- Each character to be formatted differently. It means each character must store formatting states.

This would be nice. Some players might turn off ANSI colour codes, for example (meaning that the user-defined manipulators for each colour would do nothing). Currently the players can store this state ok. It''s things like setting numerical precision that aren''t handled currently, hence my interest in moving to real streams.
quote:

That means each character stores a ostringstream (which u mentioned is unacceptable) or the state to set (via manipulators or something else). But you say you don''t want to set the state of the stream each time for each character object.

I don''t want to do anything when I output data except output data. I don''t want to have to explicitly acquire a singleton instance, I don''t want to have to explicitly create ostringstreams all over the place, and I don''t want to keep having to clear a global variable. If what I want can''t be done without something like that, then it''s not worth doing.
quote:

As null_pointer pointed - How is that possible?

Sometimes you elite coders find ways to do things that I would never have thought of.

quote:
Each character is to be formatted according to the current state of the stream passed.

Not exactly: mainly yes, they have to be formatted according to the current formatting specifications, but it would be useful to store per-character state as well (such as which currency symbols to use: done with locales in normal iostreams).

quote:
But you might need multithreading, so the global stream has to be locked/unlocked and not acceptable.

The main problem with using a global stream is that I''d have to do some explicit acquiring and releasing, which is a syntactic no-no given how often I''d have to do it.

quote:
You say it''s for a MUD project, why are you sending text for non players anyway (by streaming in your base class)? It''s seems to be more of a design issue really.

Non players see output just as players do, and act upon it. Even if they didn''t, it would be a mess to put in "if (person.IsNPC())" checks before every output operation.

I am not claiming that what I want is possible: just saying what I want and wondering how close I can get to it.

[ MSVC Fixes | STL | SDL | Game AI | Sockets | C++ Faq Lite | Boost ]

Share this post


Link to post
Share on other sites
quote:
Original post by Kylotan
I don''t want to do anything when I output data except output data.



I think it''s unavoidable that you have to set the stream state for each player as they may use different formatting states.

quote:

Non players see output just as players do, and act upon it. Even if they didn''t, it would be a mess to put in "if (person.IsNPC())" checks before every output operation.



I haven''t done a MUD before but I would think sending string messages to each game object would be inefficient (?).

In MUD, the only string messages that needs to get formatted are the ones that gets sent to the client machine( each player ). Shouldn''t the server store everything as game state data instead (as per a normal game)

i.e.
Kylotan[Hit point 100/100] : You cause INTERNAL BLEEDING on monster:
Kylotan[Hit point 80/100] : Monster hits you 20 points
Kylotan[Hit point 100/100] : You mutter a healing spell.

The string is only formatted when sending to the player (after retrieving relevant data from the game objects ).

For differentiating PCs and NPCs, perhaps you need to store two lists of objects in the game server, 1 for PC, 1 for NPC. I can imagine situations that you need perform situations for PCs only like broadcasting a player''s message.

Share this post


Link to post
Share on other sites
quote:
Original post by Void
I haven''t done a MUD before but I would think sending string messages to each game object would be inefficient (?).

Of course, doing anything that you don''t have to is inefficient. But nonplayer characters need to be able to react to what they see in the same way that player characters do. Many NPC actions are scripted with the trigger being an output pattern.
quote:

i.e.
Kylotan[Hit point 100/100] : You cause INTERNAL BLEEDING on monster:
Kylotan[Hit point 80/100] : Monster hits you 20 points
Kylotan[Hit point 100/100] : You mutter a healing spell.

The string is only formatted when sending to the player (after retrieving relevant data from the game objects ).

The problem is that there is often more than one person you''re sending it to:

for (all characters in room)
current_character << attacker << " hits " << defender
" for 20 points" << endl;
end for

In that case, I could instead instantiate/clear a stringstream once per loop iteration, but that gets ugly when repeated a thousand times through the code. Which is why, at the moment, I have operations that make writing to a Character look like writing to a stream, but sadly without many of the useful formatting benefits.

[ MSVC Fixes | STL | SDL | Game AI | Sockets | C++ Faq Lite | Boost ]

Share this post


Link to post
Share on other sites