Jump to content

  • Log In with Google      Sign In   
  • Create Account


#ActualKylotan

Posted 24 February 2013 - 08:13 AM

Part of your problem is that you're talking about a Message in an ambiguous way. Your ReceiveMessage doesn't receive a message. It receives data. That data could be a message. It could also be the start of a message. It could be the end of a previous message. It could be an end followed by a start. It could be both of those things with a complete message in between - or more than one! It can scan the data and make an educated guess as to which of these many scenarios it could be, and it can look at what arrived previously and hasn't been handled yet in order to try and process a new message or messages, but there is no need for this bit of code to perform such a complex decision.

 

Your new code is better-looking but still broken. Imagine your first call to ReceiveMessage() gets 1000 bytes of a 2000 byte message. I presume you read it directly into thisClient.Buffer, then encode it into Contents and empty the buffer. You look for a footer, don't find it yet. Then you leave the ReceiveMessage function and boom, everything in Contents is now destroyed. The next call to ReceiveMessage might get the last half of the message, but you've already lost the first half!

 

The buffer must persist outside of the function that receives data because its purpose is to gather together data that arrives in multiple receive calls. It's not there just for individual messages. The buffer should be a large storage space designed to fit many messages in. It's basically a queue for bytes, with the network pushing them in at one end and your application pulling them off in message-sided chunks at the other end.

 

You're trying to mix network reads with data buffering and message decoding and parsing which is a nightmare. It should be a simple pipeline passing data along, performing one simple step each way. Let me reiterate:

 

The receive callback should do precisely 2 things, no more, no less:

  1. Take data from the network and add it to the end of your incoming data buffer. It doesn't matter whether it's 1 byte, or a million. It doesn't matter whether it's a whole message, part of one, multiples, whatever. This part of your program does not know or care about messages. Just add it to the buffer.
  2. Tell your program that there is more data in the buffer and that it should look for completed messages.

Now you know you have a bunch of bytes in the buffer. Your program has a CollectMessages() function or something similarly named. This function has nothing to do with the network at all, and just exists to pull messages out of bytes. It does this:

  1. Read from the start of the buffer onwards to see if a complete message has been formed yet.
  2. If a complete message is found:
    1. create the relevant message object and hand it to the app for processing (eg. a HandleMessage() function)
    2. remove the data for that message from the buffer (eg. replace the buffer array with a copy of everything from the buffer array except the data for the first message)
    3. Go back to the beginning and look for the next message.
  3. No more messages in the buffer: just stop and leave whatever is in the buffer for next time.

(Note that there are more efficient ways to implement step 2.2 than copying the whole buffer and that experienced people will cringe at this implementation. Note also that for a MUD you are not going to need a more efficient method.)

 

To summarise, it's a 3 - step process:

  1. One function to take bytes from the network and enqueue them in your application. This function doesn't know or care about messages.
  2. One function to examine the front of the byte queue and form messages as soon as enough bytes are there, and to remove those bytes from the queue. This function doesn't know or care about the network.
  3. One function to process a message in whatever way your app needs to. This function doesn't know or care about the network or the buffer of bytes.

 

The reason I've included headers and footers goes back to the above paragraph, and that many of the examples I've seen floating around the interwebs includes both. But...if I understand what you are telling me, I can see how it would only be necessary to have a footer only.

 

You never need delimiters at each end, just like you don't need a period at each end of a sentence. In real life, you know the first word you read or see is the start of a sentence, because by definition the first word is the start. All you need to know is when the sentence ends, so you wait for the delimiter. So it is with networking.

 

By having 2 delimiters, you add in the extra case of having to decide what to do after a footer and before a header - which is most likely an error condition. It's best to eliminate this possibility entirely by removing the redundant delimiter.

 

Whatever you've seen "floating around the interwebs" suggesting that you need headers and footers is not likely to be a reputable source, so proceed with caution there.

: formatting


#3Kylotan

Posted 24 February 2013 - 08:11 AM

Part of your problem is that you're talking about a Message in an ambiguous way. Your ReceiveMessage doesn't receive a message. It receives data. That data could be a message. It could also be the start of a message. It could be the end of a previous message. It could be an end followed by a start. It could be both of those things with a complete message in between - or more than one! It can scan the data and make an educated guess as to which of these many scenarios it could be, and it can look at what arrived previously and hasn't been handled yet in order to try and process a new message or messages, but there is no need for this bit of code to perform such a complex decision.

 

Your new code is better-looking but still broken. Imagine your first call to ReceiveMessage() gets 1000 bytes of a 2000 byte message. I presume you read it directly into thisClient.Buffer, then encode it into Contents and empty the buffer. You look for a footer, don't find it yet. Then you leave the ReceiveMessage function and boom, everything in Contents is now destroyed. The next call to ReceiveMessage might get the last half of the message, but you've already lost the first half!

 

The buffer must persist outside of the function that receives data because its purpose is to gather together data that arrives in multiple receive calls. It's not there just for individual messages. The buffer should be a large storage space designed to fit many messages in. It's basically a queue for bytes, with the network pushing them in at one end and your application pulling them off in message-sided chunks at the other end.

 

You're trying to mix network reads with data buffering and message decoding and parsing which is a nightmare. It should be a simple pipeline passing data along, performing one simple step each way. Let me reiterate:

 

The receive callback should do precisely 2 things, no more, no less:

  1. Take data from the network and add it to the end of your incoming data buffer. It doesn't matter whether it's 1 byte, or a million. It doesn't matter whether it's a whole message, part of one, multiples, whatever. This part of your program does not know or care about messages. Just add it to the buffer.
  2. Tell your program that there is more data in the buffer and that it should look for completed messages.

Now you know you have a bunch of bytes in the buffer. Your program has a CollectMessages() function or something similarly named. This function has nothing to do with the network at all, and just exists to pull messages out of bytes. It does this:

  1. Read from the start of the buffer onwards to see if a complete message has been formed yet.
  2. If a complete message is found:
    1. create the relevant message object and hand it to the app for processing (eg. a HandleMessage() function)
    2. remove the data for that message from the buffer (eg. replace the buffer array with a copy of everything from the buffer array except the data for the first message)
    3. Go back to the beginning and look for the next message.
  3. No more messages in the buffer: just stop and leave whatever is in the buffer for next time.

(Note that there are more efficient ways to implement step 2.2 than copying the whole buffer and that experienced people will cringe at this implementation. Note also that for a MUD you are not going to need a more efficient method.)

 

To summarise, it's a 3 - step process:

  1. One function to take bytes from the network and enqueue them in your application.
  2. One function to examine the front of the byte queue and form messages as soon as enough bytes are there, and to remove those bytes from the queue.
  3. One function to process a message in whatever way your app needs to.

 

The reason I've included headers and footers goes back to the above paragraph, and that many of the examples I've seen floating around the interwebs includes both. But...if I understand what you are telling me, I can see how it would only be necessary to have a footer only.

 

You never need delimiters at each end, just like you don't need a period at each end of a sentence. In real life, you know the first word you read or see is the start of a sentence, because by definition the first word is the start. All you need to know is when the sentence ends, so you wait for the delimiter. So it is with networking.

 

By having 2 delimiters, you add in the extra case of having to decide what to do after a footer and before a header - which is most likely an error condition. It's best to eliminate this possibility entirely by removing the redundant delimiter.

 

Whatever you've seen "floating around the interwebs" suggesting that you need headers and footers is not likely to be a reputable source, so proceed with caution there.

: formatting


#2Kylotan

Posted 24 February 2013 - 08:11 AM

Part of your problem is that you're talking about a Message in an ambiguous way. Your ReceiveMessage doesn't receive a message. It receives data. That data could be a message. It could also be the start of a message. It could be the end of a previous message. It could be an end followed by a start. It could be both of those things with a complete message in between - or more than one! It can scan the data and make an educated guess as to which of these many scenarios it could be, and it can look at what arrived previously and hasn't been handled yet in order to try and process a new message or messages, but there is no need for this bit of code to perform such a complex decision.

 

Your new code is better-looking but still broken. Imagine your first call to ReceiveMessage() gets 1000 bytes of a 2000 byte message. I presume you read it directly into thisClient.Buffer, then encode it into Contents and empty the buffer. You look for a footer, don't find it yet. Then you leave the ReceiveMessage function and boom, everything in Contents is now destroyed. The next call to ReceiveMessage might get the last half of the message, but you've already lost the first half!

 

The buffer must persist outside of the function that receives data because its purpose is to gather together data that arrives in multiple receive calls. It's not there just for individual messages. The buffer should be a large storage space designed to fit many messages in. It's basically a queue for bytes, with the network pushing them in at one end and your application pulling them off in message-sided chunks at the other end.

 

You're trying to mix network reads with data buffering and message decoding and parsing which is a nightmare. It should be a simple pipeline passing data along, performing one simple step each way. Let me reiterate:

 

The receive callback should do precisely 2 things, no more, no less:

  1. Take data from the network and add it to the end of your incoming data buffer. It doesn't matter whether it's 1 byte, or a million. It doesn't matter whether it's a whole message, part of one, multiples, whatever. This part of your program does not know or care about messages. Just add it to the buffer.
  2. Tell your program that there is more data in the buffer and that it should look for completed messages.

Now you know you have a bunch of bytes in the buffer. Your program has a CollectMessages() function or something similarly named. This function has nothing to do with the network at all, and just exists to pull messages out of bytes. It does this:

  1. Read from the start of the buffer onwards to see if a complete message has been formed yet.
  2. If a complete message is found:
    1. create the relevant message object and hand it to the app for processing (eg. a HandleMessage() function)
    2. remove the data for that message from the buffer (eg. replace the buffer array with a copy of everything from the buffer array except the data for the first message)
    3. Go back to step 1 and look for the next message.
  3. No more messages in the buffer: just stop and leave whatever is in the buffer for next time.

(Note that there are more efficient ways to implement step 2.2 than copying the whole buffer and that experienced people will cringe at this implementation. Note also that for a MUD you are not going to need a more efficient method.)

 

To summarise, it's a 3 - step process:

  1. One function to take bytes from the network and enqueue them in your application.
  2. One function to examine the front of the byte queue and form messages as soon as enough bytes are there, and to remove those bytes from the queue.
  3. One function to process a message in whatever way your app needs to.

 

The reason I've included headers and footers goes back to the above paragraph, and that many of the examples I've seen floating around the interwebs includes both. But...if I understand what you are telling me, I can see how it would only be necessary to have a footer only.

 

You never need delimiters at each end, just like you don't need a period at each end of a sentence. In real life, you know the first word you read or see is the start of a sentence, because by definition the first word is the start. All you need to know is when the sentence ends, so you wait for the delimiter. So it is with networking.

 

By having 2 delimiters, you add in the extra case of having to decide what to do after a footer and before a header - which is most likely an error condition. It's best to eliminate this possibility entirely by removing the redundant delimiter.

 

Whatever you've seen "floating around the interwebs" suggesting that you need headers and footers is not likely to be a reputable source, so proceed with caution there.

: formatting


#1Kylotan

Posted 24 February 2013 - 08:09 AM

Part of your problem is that you're talking about a Message in an ambiguous way. Your ReceiveMessage doesn't receive a message. It receives data. That data could be a message. It could also be the start of a message. It could be the end of a previous message. It could be an end followed by a start. It could be both of those things with a complete message in between - or more than one! It can scan the data and make an educated guess as to which of these many scenarios it could be, and it can look at what arrived previously and hasn't been handled yet in order to try and process a new message or messages, but there is no need for this bit of code to perform such a complex decision.

 

Your new code is better-looking but still broken. Imagine your first call to ReceiveMessage() gets 1000 bytes of a 2000 byte message. I presume you read it directly into thisClient.Buffer, then encode it into Contents and empty the buffer. You look for a footer, don't find it yet. Then you leave the ReceiveMessage function and boom, everything in Contents is now destroyed. The next call to ReceiveMessage might get the last half of the message, but you've already lost the first half!

 

The buffer must persist outside of the function that receives data because its purpose is to gather together data that arrives in multiple receive calls. It's not there just for individual messages. The buffer should be a large storage space designed to fit many messages in. It's basically a queue for bytes, with the network pushing them in at one end and your application pulling them off in message-sided chunks at the other end.

 

You're trying to mix network reads with data buffering and message decoding and parsing which is a nightmare. It should be a simple pipeline passing data along, performing one simple step each way. Let me reiterate:

 

The receive callback should do precisely 2 things, no more, no less:

  1. Take data from the network and add it to the end of your incoming data buffer. It doesn't matter whether it's 1 byte, or a million. It doesn't matter whether it's a whole message, part of one, multiples, whatever. This part of your program does not know or care about messages. Just add it to the buffer.
  2. Tell your program that there is more data in the buffer and that it should look for completed messages.
  • Now you know you have a bunch of bytes in the buffer. Your program has a CollectMessages() function or something similarly named. This function has nothing to do with the network at all, and just exists to pull messages out of bytes. It does this:
    1. Read from the start of the buffer onwards to see if a complete message has been formed yet.
    2. If a complete message is found:
      1. create the relevant message object and hand it to the app for processing (eg. a HandleMessage() function)
      2. remove the data for that message from the buffer (eg. replace the buffer array with a copy of everything from the buffer array except the data for the first message)
      3. Go back to step 1 and look for the next message.
    3. No more messages in the buffer: just stop and leave whatever is in the buffer for next time.

(Note that there are more efficient ways to implement step 2.2 than copying the whole buffer and that experienced people will cringe at this implementation. Note also that for a MUD you are not going to need a more efficient method.)

 

To summarise, it's a 3 - step process:

  1. One function to take bytes from the network and enqueue them in your application.
  2. One function to examine the front of the byte queue and form messages as soon as enough bytes are there, and to remove those bytes from the queue.
  3. One function to process a message in whatever way your app needs to.

 

The reason I've included headers and footers goes back to the above paragraph, and that many of the examples I've seen floating around the interwebs includes both. But...if I understand what you are telling me, I can see how it would only be necessary to have a footer only.

 

You never need delimiters at each end, just like you don't need a period at each end of a sentence. In real life, you know the first word you read or see is the start of a sentence, because by definition the first word is the start. All you need to know is when the sentence ends, so you wait for the delimiter. So it is with networking.

 

By having 2 delimiters, you add in the extra case of having to decide what to do after a footer and before a header - which is most likely an error condition. It's best to eliminate this possibility entirely by removing the redundant delimiter.

 

Whatever you've seen "floating around the interwebs" suggesting that you need headers and footers is not likely to be a reputable source, so proceed with caution there.


PARTNERS