Sign in to follow this  
DesCr

What is a token?

Recommended Posts

pinacolada    834
In general, I think "token" means "a simple and easy-to-handle thing that represents a more complicated thing"

When you tokenize a string, it means you break the string up into words, so that it's easier to work with. Like, if you tokenize the string "fluffy, bunny" or the string " fluffy       bunny "

then in both cases you end up with the tokens "fluffy" and "bunny".

Tokens aren't always strings, but I think that's the most common usage.

Share this post


Link to post
Share on other sites
Chris Hare    462
There are five kinds of tokens: identifiers (names), keywords (template, int etc), literals (constants), operators (* etc) and other seperators.

Share this post


Link to post
Share on other sites
fractoid    703
I'd say that a token is the largest chunk of whatever is being processed that can be processed in one go by whatever is doing the processing. How's that for a general purpose definition? :)

Chris's definition is what I'd use if I were talking about a programming language (or trying to figure out what a compiler were talking about). I think pinacolada said the same thing as I did but differently. :)

Share this post


Link to post
Share on other sites
bastard2k5    238
Well, tokenizing is usually just something that splits a very large thing, into a container of smaller things, although at least in Java, and Python, there is a split command, that will return a container with the resulting objects from that split. So, a token, is just a string, that you can do anything that you want to do to it, as long as it is legal for your language definition of a string.

Share this post


Link to post
Share on other sites
DesCr    100
I use C++, and I guess the definition I'm looking for is "tokenizing" in terms of writing a scripting language.

I'm ALSO wondering what the C++ token is I see. It comes up now and then, but not much is in my books. I believe, when I see them, they are just a name with an underscore in front of it.

Example:

_name;

That confuses me, because it doesn't seem like a full statement, since there's no specifier. Thanks guys.

Share this post


Link to post
Share on other sites
amedlock    100
In parsing theory a token is the smallest string of characters with meaning which doesn't rely on other tokens.

An example would be a number, roughly "[0-9]+[.0-9]*"
This is also called a lexeme.(or lexical element)

The lexer divides the input stream into these tokens which are fed to a parser which decides how to interpret them.



Share this post


Link to post
Share on other sites
DesCr    100
So let's say I have a script file that I want to read in. Let's say there's lines:

NEWINT gold (200)
NEWINT silver (100)

If I write a program that does this:

While(!EOF) {
-Read until space or newline (to get one word)
-If word == NEWINT
--Read next word
--Allocate new integer / create pointer
--Index the integer by it's name (next word)
--Initialize
}

Very simple psuedo above, so I hope it convey's my way of thinking.

Does it mean that I'm tokenizing the script by reading each word and determinging what it is?

Share this post


Link to post
Share on other sites
amedlock    100
I have to ask,are you doing this as a learning experience or to get things done(with a game I would guess).

If you are trying to get things done, do yourself a favor and get a scripting library. I suggest Lua(my favorite), Angelscript or Small. They have been tested, are free, and will be faster than a solution you could roll yourself.

If you are just learning, buy a good book on the subject compilers and computer languages are both very interesting topics.

Typical human-being parsers are usually what we call recursive descent parsers. They recursively call subroutines looking for particular language features, usually pushing the results of what they find onto a stack of some kind. Look at this:

Here is your line:
NEWINT <varname> <left-parentheses> <number> <right-parentheses>

and here is the pseudo-code

if parse_newint_command() then ....store integer, varname, etc

function parse_newint_command()
if token!='NEWINT' then return false;
next_token(); // move from NEWINT to the next token
if not parse_var() then return false;
next_token();
if token!='(' then return false;
..
end


The trick here is to save the place you are in the token list before you call parse_newint_command() and if it fails, restore to that point.

Sorry if this is unclear.

Share this post


Link to post
Share on other sites
DesCr    100
I'm doing it as a learning experience mostly. I'd like to be able to load a map in my engine, and then attach scripts to it. The scripts would run when a character intersects a certain trigger point. All the scripts would need to do is move some objects around, maybe change money values, and display text on the screen. It's an RPG, and it'd be nice to script IGC's to save on code.

I initially thought of just creating a small parser that would read the script and export it as C++ code.

I liked the direction this tutorial was going on scripts, but the series is unfinished and he never gets around to the actual implementation:

http://www.gamedev.net/reference/articles/article1294.asp

Share this post


Link to post
Share on other sites
Ravyne    14300
If you can afford a 50$ book, "Game Scripting Mastery" by Alex Varanese is an excelent book. It begins at theory, talks about existing solutions, then takes you through the design of a complete scripting solution, including the design and implimentation of a language and an appropriate virtual-machine-code and VM. There are many other books on compiler/script design however most are very accademic and focus more on theory. Alex's book includes a complete implimentation and is easy to follow (relatively speaking.)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this