Sign in to follow this  

turning a char * into different tokens(c++)

This topic is 3738 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hi, I need a tokeniser of some kind. I have a data string attached to a 3d model, which my modelling app allows me to edit. If I type the following: MATERIAL = grass_1 COLLISION = bullet, character PARAMETERS flammable, destructable I need to use this to allow my game to handle different kinds of model, so the model above would be flammable, and destructable, and would collide with characters and bullets? My texture database would then be instructed to load the files inside data/textures/grass_1/ The objects bit-field for collision would be 00000011 (0x02 for bullet collision, and 0x01 for character collision) (is bit-field the correct term?) How would I go about turning this into a string for the material, and a bit-field for its collision flags and its physical characteristics? Thanks.

Share this post


Link to post
Share on other sites
Quote:
Original post by shotgunnutter
Hi,

I need a tokeniser of some kind. I have a data string attached to a 3d model, which my modelling app allows me to edit. If I type the following:

MATERIAL = grass_1
COLLISION = bullet, character
PARAMETERS flammable, destructable

I need to use this to allow my game to handle different kinds of model, so the model above would be flammable, and destructable, and would collide with characters and bullets?

My texture database would then be instructed to load the files inside data/textures/grass_1/

The objects bit-field for collision would be 00000011 (0x02 for bullet collision, and 0x01 for character collision)
(is bit-field the correct term?)

How would I go about turning this into a string for the material, and a bit-field for its collision flags and its physical characteristics?

Thanks.
There have been a number of threads on parsing text files in this forum over the last few weeks, so you might start by searching the archives for threads on similar topics.

Anyway, the simplest approach will probably be to use sdt::getline() and std::string to read the file a line at a time, and then to write some code to split up each line into tokens (better yet, just use Boost's 'split' function or 'tokenizer' class).

For the bit field, I'd use a std::map that maps strings (such as 'bullet' and 'character') to 'bit' values (e.g. 0x01, 0x02...). Then you would just iterate over the tokens and grab the corresponding values from the map. (Or, you could use std::bitset and access the bits via index instead.)

That's probably enough to get you started, I would think (but post back if have further questions).

Share this post


Link to post
Share on other sites
Basing on the first line:

fscanf("%s" , &type);

if (!strcmp(type, "MATERIAL))
{
fscanf("%s %s", &trashbuffer, &materialtype);
engine.object[i]->texture_filename = materialtype; // supposedly both are strings
}

That will parse your string into single tokens and then you can distribute them among objects. Using pre-made values, you can use strcmp to turn those into bitfields or whatever you feel like it.

The basic idea is:

1. Find the field that interests you
2. Parse the field values stripping garbage "= grass_1"
3. Parse the values ("grass_1") into internal workings of the engine
4. Profit.

Unless I misunderstood what you are trying to achieve here?

Share this post


Link to post
Share on other sites
Quote:
Original post by NestorPL
Basing on the first line:

fscanf("%s" , &type);

if (!strcmp(type, "MATERIAL))
{
fscanf("%s %s", &trashbuffer, &materialtype);
engine.object[i]->texture_filename = materialtype; // supposedly both are strings
}

That will parse your string into single tokens and then you can distribute them among objects. Using pre-made values, you can use strcmp to turn those into bitfields or whatever you feel like it.

The basic idea is:

1. Find the field that interests you
2. Parse the field values stripping garbage "= grass_1"
3. Parse the values ("grass_1") into internal workings of the engine
4. Profit.

Unless I misunderstood what you are trying to achieve here?


I want to do something similar to that, except I already managed to tokenise a file, and now I have a single large char * for the data of the object. I need to break that data down into separate tokens.

[edit: I've found what I need by googling some of the function names you guys used. Thanks]

Share this post


Link to post
Share on other sites
Quote:
Original post by NestorPL
Use sscanf instead of fscanf then. Strip the big array of text into smaller chunks that actually mean something ;-)


Thanks,
I'm going to try doing this using "strtok" to break it down into commands, then again depending on the type of command.
Commands will be ; deliminated, then parameters to each command deliminated in various ways.

so I would write this for a string:
  MATERIAL "grass_1";

This for a bit-field: (using comma deliminated flags)
  COLLISION bullet,character,particle;
  PROPERTIES flammable,destructable;


Anyway, thanks for your help and advice, since I found this stuff by googling what you put in your posts.

Share this post


Link to post
Share on other sites
JohnBolton is right strtok is a dangerous function to use, it has a lot of potential pitfalls that just make it difficult to use. In fact if you are using c++ I would advice against using any c style file reading/parsing (fscanf, sscanf, etc.) and instead use the easier and safer standard c++ iostreams functionality. jyk has given you good advice about using getline and strings to parse the file. You can also use the extraction operator (>>) to grab a token at a time out of the file skipping whitespace and parse the file that way.

Bottom line there are better ways to parse/read files in c++ than using c functionality.

Share this post


Link to post
Share on other sites
Here is a simple implementation for you...it will split the passed in string using the specified delimiter charecters...

I think this is a more c++ oriented approach...


/* Returns a vector that contains all the substrings delimited
by the characters in the passed delims argument */
std::vector<std::string> Split(const std::string& str, const std::string& delims, const unsigned int& maxSplits)
{
std::vector<std::string> ret;
unsigned int numSplits = 0;

unsigned int start = 0;
unsigned int pos = 0;

while (pos != std::string::npos)
{
pos = (unsigned int)str.find_first_of(delims, start);

if (pos == start)
{
start = pos + 1;
}
else if (pos == std::string::npos || (maxSplits && numSplits == maxSplits))
{
ret.push_back(str.substr(start));
break;
}
else
{
ret.push_back(str.substr(start, pos - start));
start = pos + 1;
}

start = (unsigned int)str.find_first_not_of(delims, start);
++numSplits;
}

return ret;
}

Share this post


Link to post
Share on other sites
Quote:
Original post by ShaderDev
Here is a simple implementation for you...it will split the passed in string using the specified delimiter charecters...

I think this is a more c++ oriented approach...


/* Returns a vector that contains all the substrings delimited
by the characters in the passed delims argument */
std::vector<std::string> Split(const std::string& str, const std::string& delims, const unsigned int& maxSplits)
{
std::vector<std::string> ret;
unsigned int numSplits = 0;

unsigned int start = 0;
unsigned int pos = 0;

while (pos != std::string::npos)
{
pos = (unsigned int)str.find_first_of(delims, start);

if (pos == start)
{
start = pos + 1;
}
else if (pos == std::string::npos || (maxSplits && numSplits == maxSplits))
{
ret.push_back(str.substr(start));
break;
}
else
{
ret.push_back(str.substr(start, pos - start));
start = pos + 1;
}

start = (unsigned int)str.find_first_not_of(delims, start);
++numSplits;
}

return ret;
}


Looks like just the right tool for the job, as well as a good lesson in the use of std::vector and string.
I would do it once to create a vector of individual commands, and again depending on the contents of each line to break it into separate tokens.

Share this post


Link to post
Share on other sites

This topic is 3738 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this