A token is a single discrete element. Its specific usage depends on context. With respect to compilers and parsers, a token is a single language element such as an identifier (variable or function name), an operator or another form of lexical symbol. With regard to string manipulation, tokens are what are obtained after the string has been separated by delimiters - special symbols such as whitespace (decided by the programmer).
For example, if I decide to use whitespace (space, newline, tab, ...) as separators, then tokenizing the string "Every good boy deserves favor" results in the following tokens:
I wanna work for Microsoft!
For example, if I decide to use whitespace (space, newline, tab, ...) as separators, then tokenizing the string "Every good boy deserves favor" results in the following tokens:
Everygoodboydeservesfavor I wanna work for Microsoft!
quote:Original post by Ignacio Liverotti
And by the way, Who doesn´t want to work in Microsoft??
You''d be surprised at how vehemently opposed a lot of people are to that lovable, cuddly-wuddly firmie-wirmie called Microsoftie-woftie...
*ahem*
I wanna work for Microsoft!
quote:Original post by OluseyiOriginal post by Ignacio Liverotti
And by the way, Who doesn´t want to work in Microsoft??
You''d be surprised at how vehemently opposed a lot of people are to that lovable, cuddly-wuddly firmie-wirmie called Microsoftie-woftie…
*ahem*
I wanna work for Microsoft!
During a co-op placement round, I missed a message that said I had a job interview with MS. But because the interviewer was so impressed by my resume, she was going to give me a phone interview. Problem was that she went on vacation the day of.
Plus they don''t pay that much.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
