Jump to content
  • Advertisement

Archived

This topic is now archived and is closed to further replies.

Krisc

Tokenizer for C#

This topic is 5278 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hey! I am writing a game and I wrote this class based off what I know of the tokenizer in Java...only thing is this is in C#. So enjoy...questions/comments/suggestions/critiques are always welcomed and wanted. If you use this, just give me credit please. Thanks. What it does is basically is split up a string into an array of strings. I use it mainly for configuration file reading. Makes it a bit easier to read. Example: I love monkeys. Would split into: I love monkeys. very simple to use...
iTokenizer t = new iTokenizer("I love monkeys!");
while(t.hasMoreTokens())
{
 System.Console.WriteLine(t.nextToken());
}
 
here it is...sorry if it doesn''t tab, it is tabbed in VS.NET.
/*
 * #################################################################
 * # Class         - iTokenizer                                    #
 * # Filename      - iTokenizer.cs                                 #
 * # Author        - John Sedlak (admin@immane.com)                #
 * # Date Created  - Friday, May 28, 2004                          #
 * # Date Modified - Tuesday, June 1, 2004                         #
 * # Purpose       - iTokenizer takes strings of varying lengths   #
 * #					and splits them based on a delimeter. The  #
 * #					delimeter can be specified or it defaults  #
 * #					to a space. You can navigate through the   #
 * #					tokens by using iTokenizer.nextToken();    #
 * #					The idea is to simplify file reading.      #
 * #################################################################
 */

using System;

namespace Galactic_Wars_2
{
	public class iTokenizer
	{
		private string myString;
		private string[] myTokens;
		private int cToken;

		public iTokenizer(string str)
		{
			// Initialize the class.
			Tokenize(str);
		}
		
		public iTokenizer(string str, char delimiter)
		{
			// Initialize the class with a specified delimeter.
			Tokenize(str, delimiter);
		}

		public void Tokenize(string str)
		{
			char[] c = new char[1];			// This is the delimeter.

			myString = str;					// We setup the class'' string.

			c[0] = '' '';						// Set the delimeter.
			myTokens = myString.Split(c);	// Split the string and put it
											// in myTokens.

			cToken = -1;					// Set the current token to -1.
		}

		public void Tokenize(string str, char delimiter)
		{
			char[] c = new char[1];			// This is the delimeter.

			myString = str;					// Setup the string...

			c[0] = delimiter;				// Set the delimeter (user specified).
			myTokens = myString.Split(c);	// Split the array into myTokens.

			cToken = -1;					// Set the current token to -1.
		}

		public string nextToken()
		{
			/*
			 *	Basically if we still have more tokens, increase the index
			 * and return the next token. Simple really. Otherwise return
			 * an error code.
			 */
			if(hasMoreTokens())
			{
				cToken++;
				return myTokens[cToken];
			}
			else
				return "-1";
		}

		public bool hasMoreTokens()
		{
			// Check to see if the current index is less than
			//		one less than final index.
			return (cToken < myTokens.Length - 1);
		}

		public int numTokens()
		{
			// Return the number of tokens.
			return myTokens.Length;
		}
	}
}
 
<- My Site -> "Discipline is my sword, faith is my shield do not dive into uncertainty, and you may live to reap the rewards" - (Unreal Championship)

Share this post


Link to post
Share on other sites
Advertisement
Thanks! I will certainly look at that! I was wondering about that ever since I saw the webcast of the new C++ that is coming out...

<- My Site ->
"Discipline is my sword, faith is my shield
do not dive into uncertainty, and you may live to reap the rewards" - (Unreal Championship)

Share this post


Link to post
Share on other sites
A good idea is to avoid redundant code in your overloads:


public void Tokenize(string str)
{
Tokenize(str, ' ');
}


public void Tokenize(string str, char delimiter)
{
char[] c = new char[1]; // This is the delimeter.

myString = str; // Setup the string...

c[0] = delimiter; // Set the delimeter (user specified).

myTokens = myString.Split(c); // Split the array into myTokens.

cToken = -1; // Set the current token to -1.

}




[edited by - Dark_Nebula on June 2, 2004 6:47:15 AM]

Share this post


Link to post
Share on other sites

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

We are the game development community.

Whether you are an indie, hobbyist, AAA developer, or just trying to learn, GameDev.net is the place for you to learn, share, and connect with the games industry. Learn more About Us or sign up!

Sign me up!