• Advertisement
Sign in to follow this  

StreamTokenizer

This topic is 4382 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

(java) I am trying to parse a String and have my method react differently to various characters it encounters throughout the parsing. I implement a StreamTokenizer, which I have very little experience with, to get the job done. My problem is that, when I detect a particular character (say '$'), I want to be able to stop parsing momentarily and iterate the string (both forward and back) while looking for other characters. Here is my original code:
String str = "weji383 jiejd $ dsiwjd iwej d4";
StreamTokenizer st = new StreamTokenizer(new StreamReader(str));
st.parseNumbers();
int token = st.nextToken();

while(token != st.TT_EOF)
{
    if(token == st.TT_NUMBER)
        // Do something
    else if(token == st.TT_WORD)
        // Do something else
    else
    {
        switch(token)
        {
            case '*':
                // Do something
                break;
            case '$':
                // Here is where I want to iterate backwards towards the
                // first character in the string ('w'), and then forwards
                // towards the last character ('4').
                break;
            default:
                break;
        }
    }

    token = st.nextToken();
}

My first thought was to nest an inner-switch inside the main switch. Then, by having a temporary variable, I could manipulate that var's position within the tokenizer, leaving the 'token' variable untouched:
// Local variable
int tempToken = token;

// Inside the main switch
case '$':
    while(tempToken != st.TT_EOF)
    {
        tempToken = st.nextToken(); // Iterate forward towards last character
        switch(tempToken)
        {
            case '4':
                // Do something
                break;
            // etc.
        }
    }
    break;

This attempt creates two problems: (1) StreamTokenizer's don't have a "previousToken()" method, which sort of makes it impossible to iterate backwards. And (2) once I'm in the nested switch (for tempToken), since I don't understand StreamTokenizer very well, I have know idea if I can just check for individual characters, such as '4.' Can I do this, or would I have to treat it as a TT_WORD??

Share this post


Link to post
Share on other sites
Advertisement
Okay, I don't know java, and I've never built a stream tokenizer before (although I've been researching them for a new program I'm working on).

However, I just have a few fundamental questions here that are purely theoretical.

I'm just curious why you couldn't actually just keep a variable such as previous_token that would store, obviously, the previous token and keep it in case you want to look backwards?

Secondly, I once again don't know java or stream tokenizers very well, but to tokenize something, you would need to read it into a character array wouldn't you? Which would allow you to look for single characters of any type, so in my made-up theory, couldn't you have anything as a token?

Sorry this isn't really helping with your problem, but I'm trying to understand these things better myself.

My two cents, something to chew on,

Vopisk

Share this post


Link to post
Share on other sites
Yeah for what you are doing it might be easier to use a character array like Vopisk said

You could loop through it using a for loop and do exactly what you are doing, and still being able to iterate through the chracters inside the loop

Phill

EDIT: since it looks like you are splitting entire words out with $ as a sort of delimiter, maybe you could turn it into an array delimitered buy " " then any part of the array with a length > 1 is a word, if not you can test the character

You will still be able to itterate forward and back.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Advertisement