Advertisement Jump to content
Sign in to follow this  

Preprocessor r2

This topic is 5002 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I suppose this is the second 'official' release of the preprocessor. I would appreciate it if the angelscript website was updated. The source can be snagged at Changes include - New method for loading scripts from file. You will need to supply a class derived from Preprocessor::FileSource. The class has the interface
		virtual bool LoadFile(const std::string& filename, std::vector<char>&)
which must load the file 'filename' into the vector. It returns true if sucessful, false if not. - Worse error reporting. Sorry, it is not a priority. Specifically, the error reporting by AngelScript will be flawed because the preprocessor no longer directly calls AngelScript. It is your responsibility to pass the finished code stream to AngelScript, which makes it impossible to extract usefull line and file information. Additionally, this change means you must also pass a functor to receive the code stream. The class Preprocessor::VectorOutStream is already implemented to serve this purpose. - A much more stable recursive descent tokenizer to replace the FSM one. I actually had no idea what a recursive decent model was until I looked into the angelscript compiler code. It is, IMO, a much better design. And it was a lot easier to write, too. - Fixed the hexadecimal constant bug. They are now correctly identified as single tokens, instead of two seperate tokens. - The other reported bug which I have not investigated of leaking file handles is no longer an issue as the preprocessor no longer opens files itself. Things I'm working on - My current project would greatly benefit from the ability to preprocess code already in memory.

Share this post

Link to post
Share on other sites
Thanks, for the new release Deyja. I'm glad to see that you're still improving on this tool.

I'll update the AngelScript site as soon as I can. Hopefully later in the day.

Share this post

Link to post
Share on other sites
Sounds nice. I'm going to have a look at it.

The fix i did for the hexadecimal bug was rather hackish.

One way you can fix the error codes (i had a giant trouble with it) problem with line numbers being off is storing the starting point of the script and just counting the number of CRLF's you run into.

I then modified the as error code output function to place the error line# and column into a vector<pair<int,int> > and the actual error code into a vector<string> (i suppose i could have gotten more creative and done a vector< pair<pair<int,int>, string > >) (this made it so that I did not have to parse the actual error string for the line#/column but is not required )and then do some calculations with the error line and combine them back to make a correct judgement of where the error is. With a little more work it would be quite reasonable to be able to say "Error line 57, 5, file 'myfile.txt': cannot convert int to MyStruct" by keeping track of beginnning and ending line #'s of a script that was included into the file.

The #defines on the other hand would pose a different situation should they expand to more than 1 line, but it would be reasonable to also account for that.

Share this post

Link to post
Share on other sites
The simplest solution to line numbers in errors is of course the way I originally did it. Just pass each file to the script engine as a seperate script section. This unfortunatly has it's own drawbacks. Mainly, it means that include guards are no longer adequate to prevent multiple inclusion. The file might expand to nothing the second time, but then it replaces the script section that is already registered. Additionally, I need to parse the filename into a true absolute path. Currently, if you use the ../ directory name, it stays in the filename as in. You could end up with something like scripts/level1/../general/scriptname.asc. I didn't feel like tackling the problem of actually parsing filenames so I didn't bother. Also keep in mind that I don't want to couple this that tightly with angelscript. As it stands it can preprocess any language.

What I propose instead is to simply save the preprocessed script and examine it instead of the original scripts. This is easily accomplished in the current system by writing a Preprocessor::OutStream that saves to disc, or by just grabbing the contents of the Preprocessor::VectorOutStream you will probably be using otherwise.

Another thing that came up is that it does not support the syntax 0.000f, with the trailing f to signify a float constant instead of a double constant. Fixing this will neccesitate changing how the numbers are tokenized so the problem of and such being a valid number token will also be fixed.

Here is a quick example of how to work with the new interface. I'm working on some docs for the preprocessor, and am going to add it to my webpage at Be warned that I am one of the worse html writers ever and that that simple webpage represents the pinnacle of my design adventures. Any help is appreciated.

class VFSFileSource : public Preprocessor::FileSource
virtual bool LoadFile(const std::string& filename, std::vector<char>& into)
/* load the file... */
return true;

Preprocessor::VectorOutStream VOS;
VFSFileSource VFS;
std::string script = "filename_of_script";

Preprocessor::preprocess(script,VFS,VOS); //preprocess's final parameter is optional


The static cast is only neccessary to silence a compiler warning. VectorOutStream returns it's size as a size_t, but AddScriptSection wants an int. size_t as of vc2003 (IIRC) is a 64bit type.
You will of course want to add proper error checking.

Share this post

Link to post
Share on other sites
Apparently I am stupid. I neglected to recall that hex constants can also include the characters ABCDEF. I've fixed it. Also; it now properly handles the #.##f format.

Share this post

Link to post
Share on other sites
Sign in to follow this  

  • Advertisement

Important Information

By using, you agree to our community Guidelines, Terms of Use, and Privacy Policy. is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!