Sign in to follow this  
Vincent_M

Text Parsing

Recommended Posts

I've written my Shader class that'll load, compile, and link GLSL shaders for OpenGL. What I'd like to do is automatically add a collection of uniforms and attributes to my shader class during load-time. What I'm doing is loading the text file into an allocated char array, and then I started to parse it with strtok(). The problem is that strtok() modifies my argument. If this is the case, I'd have to load my file twice into two char arrays: one for compilation and for parsing.

Any ideas to work around this, or should I just copy the data, and release it when I'm done with it?

Share this post


Link to post
Share on other sites
[quote name='rip-off' timestamp='1318977524' post='4874067']
One solution is to not use strtok(). A stateful implementation could work better, if i understand what you are trying to do.
[/quote]

I thought about making my own version of strtok(), but that'd involve copying buffers every time I call my strtok() implementation.

Could you explain a little more on what you mean by "stateful implementation"?

Share this post


Link to post
Share on other sites
[quote name='ApochPiQ' timestamp='1318978934' post='4874070']
The functionality of strtok is trivial to reimplement in any form you need, provided you're comfortable with C-string manipulation.

Of course, if you're in C++, you can just use stringstreams which will save you a lot of headache.
[/quote]

I was writing my reply to rip-off when you posted this and missed it the first time, lol. I've thought about using C++ stringstreams, but I'm not sure if the data is compatible with my glCompileShader() which takes a pointer to a char buffer.

Share this post


Link to post
Share on other sites
If you don't mind adding some extra stuff to your shader file...

In your shader file (sorry don't really know glsl syntax...)...

[code]
float4x4 a;

//%%CUSTOMIZATION_MARKER%%

float4 vs(..) {
///...
[/code]

and in your c++ code:

[code]
string code = load_shader();
code.replace("//%%CUSTOMIZATION_MARKER%%", "blah blah")

[/code]

or if you know enough about your shader you can even replace [code]float4 vs([/code] with [code]blah blah \n float4 vs([/code] (or something like that)

Of course the above assumes the shader writer knows what she/he's doing.

[quote][color="#1C2837"][size="2"]I've thought about using C++ stringstreams, but I'm not sure if the data is compatible with my glCompileShader() which takes a pointer to a char buffer. [/size][/color][/quote]
[code]string a;
const char* c = a.c_str();[/code]

Share this post


Link to post
Share on other sites
[quote name='Vincent_M' timestamp='1318979019' post='4874071']
[quote name='rip-off' timestamp='1318977524' post='4874067']
One solution is to not use strtok(). A stateful implementation could work better, if i understand what you are trying to do.
[/quote]

I thought about making my own version of strtok(), but that'd involve copying buffers every time I call my strtok() implementation.

Could you explain a little more on what you mean by "stateful implementation"?
[/quote]
I was writing that post on my phone, and I was trying to edit it to improve it but it was tricky and gamedev kept redirecting my to index.php and losing my changes, so I stopped.

When I said "stateful", I actually meant to use explicit state, not like strtok's hidden state. The other thing is that instead of dealing in NUL terminated character arrays (requiring the data to be modified), deal in iterator pairs. You can treat a character pointer as an iterator, but you could also use indices or std::string::const_iterator (if you use the standard library).

Tokenising using iterator pairs allows you to avoid copying until you need to. I assume you're scanning the shader for declarations, and then adding them to a collection. With iterator pairs, you can omit copying the tokens until you know you're parsing a declaration.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this