One solution is to not use strtok(). A stateful implementation could work better, if i understand what you are trying to do.

The functionality of strtok is trivial to reimplement in any form you need, provided you're comfortable with C-string manipulation.

Of course, if you're in C++, you can just use stringstreams which will save you a lot of headache.

If you don't mind adding some extra stuff to your shader file...

In your shader file (sorry don't really know glsl syntax...)...


float4x4 a;

//%%CUSTOMIZATION_MARKER%%

float4 vs(..) {
///...


and in your c++ code:


string code = load_shader();
code.replace("//%%CUSTOMIZATION_MARKER%%", "blah blah")



or if you know enough about your shader you can even replace float4 vs( with blah blah \n float4 vs( (or something like that)

Of course the above assumes the shader writer knows what she/he's doing.

[color="#1C2837"]I've thought about using C++ stringstreams, but I'm not sure if the data is compatible with my glCompileShader() which takes a pointer to a char buffer. [/quote]
string a;
const char* c = a.c_str();

[quote name='rip-off' timestamp='1318977524' post='4874067']
One solution is to not use strtok(). A stateful implementation could work better, if i understand what you are trying to do.

I thought about making my own version of strtok(), but that'd involve copying buffers every time I call my strtok() implementation.

Could you explain a little more on what you mean by "stateful implementation"?
[/quote]
I was writing that post on my phone, and I was trying to edit it to improve it but it was tricky and gamedev kept redirecting my to index.php and losing my changes, so I stopped.

When I said "stateful", I actually meant to use explicit state, not like strtok's hidden state. The other thing is that instead of dealing in NUL terminated character arrays (requiring the data to be modified), deal in iterator pairs. You can treat a character pointer as an iterator, but you could also use indices or std::string::const_iterator (if you use the standard library).

Tokenising using iterator pairs allows you to avoid copying until you need to. I assume you're scanning the shader for declarations, and then adding them to a collection. With iterator pairs, you can omit copying the tokens until you know you're parsing a declaration.