#### Archived

This topic is now archived and is closed to further replies.

# Tokenizing Strings

This topic is 6080 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

## Recommended Posts

I need to tokenize strings! I''ve used strtok() but it crashes when I tell it to look for the second token.... This is very annoying. Are there any alternatives, or will I have to write my own function?

##### Share on other sites
If you''re making a script parser, I''d write your own, you''ll have more control over how it is tokenized. eg. operators like +-*/ should be separated even if there isn''t a space between the variable.

If you''re just trying to extract data from a string separated by spaces, then there is some function which is like the reverse of sprintf, but I can''t remember what it''s called. sorry, that''s not very helpful, but what the hell

Frank

##### Share on other sites

Strtok() sucks ass because it modifies/destroys your source string.

Keep that in mind. It screwed me up big time when I wrote a parser for a little synth I tried to make (using text files for input). Anyway, I don''t remember exactly what happened, but I ended up having to make a copy of my source string, and passing that to strtok().

##### Share on other sites
Strtok is very useful, but, as the AP said, you will need a copy of the source string, otherwise it gets mangled. Notice the "char seps[]", in which you can place anything that you want, such as ''+'' or ''-'' or '' '' etc etc etc. The following code is taken from my console. It isn''t the most versatile, being designed specifically for my needs, but it works a treat.

  // make a copy of the input string char stream[LINE_LENGTH]; sprintf ( stream, input ); // establish the strings we will need char seps[] = " ,\t\n"; // the valid word seperators char *command; // the first token ( ie. SET, or GIVE ) char *parm1; // the first argument ( ie. HEALTH or ROCKETLAUNCHER ) char *parm2; // the second argument ( ie. 50, 70 ) char *parm3; // it may be used someday... /* Establish string and get the tokens: */ command = strtok( stream, seps ); parm1 = strtok( NULL, seps ); parm2 = strtok( NULL, seps ); parm3 = strtok( NULL, seps ); // make sure they typed something: if ( command==NULL ) return 0;

I think the formatting may get screwed around with the comments, but I''m not sure...

Anyway, cheers

##### Share on other sites
If you need to write your own func, you could do it fairly easily, especially if you can hard-code the seperator...

something I just made up... not tested
  void strtok2(char* instr, char * outstr, int& start){ instr += start; while(*instr != ' ') { *(outstr++) = *(instr++); start++; } *outstr = 0; start++; //skip over seperator next time }

...instr is the string to tokenize, outstr is a buffer for the token, start is where to start at (init any var to 0 and keep passing the same var w/o modifying it's value)...

edit: btw, if you need more seps, just add " || *instr = ','", etc to the while..

Edited by - TrIaD on June 30, 2001 10:09:03 PM