Archived

This topic is now archived and is closed to further replies.

Tokenizing Strings

This topic is 6008 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I need to tokenize strings! I''ve used strtok() but it crashes when I tell it to look for the second token.... This is very annoying. Are there any alternatives, or will I have to write my own function?

Share this post


Link to post
Share on other sites
If you''re making a script parser, I''d write your own, you''ll have more control over how it is tokenized. eg. operators like +-*/ should be separated even if there isn''t a space between the variable.

If you''re just trying to extract data from a string separated by spaces, then there is some function which is like the reverse of sprintf, but I can''t remember what it''s called. sorry, that''s not very helpful, but what the hell

Frank

Share this post


Link to post
Share on other sites
Guest Anonymous Poster

Strtok() sucks ass because it modifies/destroys your source string.

Keep that in mind. It screwed me up big time when I wrote a parser for a little synth I tried to make (using text files for input). Anyway, I don''t remember exactly what happened, but I ended up having to make a copy of my source string, and passing that to strtok().



Share this post


Link to post
Share on other sites
Strtok is very useful, but, as the AP said, you will need a copy of the source string, otherwise it gets mangled. Notice the "char seps[]", in which you can place anything that you want, such as ''+'' or ''-'' or '' '' etc etc etc. The following code is taken from my console. It isn''t the most versatile, being designed specifically for my needs, but it works a treat.

  
// make a copy of the input string

char stream[LINE_LENGTH];
sprintf ( stream, input );

// establish the strings we will need

char seps[] = " ,\t\n"; // the valid word seperators

char *command; // the first token ( ie. SET, or GIVE )

char *parm1; // the first argument ( ie. HEALTH or ROCKETLAUNCHER )

char *parm2; // the second argument ( ie. 50, 70 )

char *parm3; // it may be used someday...


/* Establish string and get the tokens: */
command = strtok( stream, seps );
parm1 = strtok( NULL, seps );
parm2 = strtok( NULL, seps );
parm3 = strtok( NULL, seps );

// make sure they typed something:

if ( command==NULL ) return 0;



I think the formatting may get screwed around with the comments, but I''m not sure...

Anyway, cheers

Share this post


Link to post
Share on other sites
If you need to write your own func, you could do it fairly easily, especially if you can hard-code the seperator...

something I just made up... not tested
    
void strtok2(char* instr, char * outstr, int& start)
{
instr += start;
while(*instr != ' ')
{
*(outstr++) = *(instr++);
start++;
}
*outstr = 0;
start++; //skip over seperator next time


}


...instr is the string to tokenize, outstr is a buffer for the token, start is where to start at (init any var to 0 and keep passing the same var w/o modifying it's value)...

edit: btw, if you need more seps, just add " || *instr = ','", etc to the while..

--Tr][aD--

Edited by - TrIaD on June 30, 2001 10:09:03 PM

Share this post


Link to post
Share on other sites