Syntax hilighting

Published January 02, 2008
Advertisement
So, something I've been working on a fair amount recently is a new syntax hilighter for GDNet. The existing hilighter supports something like four languages, runs in VBScript, and is basically just a small number of regexp passes over the code.

What I'm looking at doing is replacing it with a component that I've built around the colorer library. Colorer supports over 100 languages, has somewhat more powerful parsing capabilities, and is written in native C++ such that the component I've built around it can process a 3K file of Python source in ~500ms, 1K of C++ in ~90ms, and 95K of C++ in ~3800ms. There's still more optimization to be done.

Here's a couple of examples of the kind of output I'm producing: 1, 2. Take a look at the HTML sources to those. There's a lot of detail in there.

Some of the languages that colorer supports are missing potential extensions - Oluseyi suggested hilighting standard library functions in Python, for example - but it's open-source, and the way you specify a language for the parser is a published standard, so it shouldn't be too hard for people to do the bulk of my work for me on those things. We'll also benefit from any new languages that other people using Colorer contribute, or optimizations they make to the parsing engine, while also keeping our ability to develop it further ourselves if we want to.

This is how OSS is supposed to work - it just frequently doesn't. (I'm still to discover exactly how I contribute the changes I've made to the code back into the codebase on sourceforge).
Previous Entry Upcoming downtime
Next Entry Avatars fixed
0 likes 3 comments

Comments

johnhattan
Cool. I didn't even know that an improved syntax highlighter was on the radar for enhancements. Is there a to-do list of gamedev improvements and bugfixes somewhere?
January 02, 2008 10:52 AM
superpig
Well, you are able to view things in my JIRA project - but truth be told, this is really a V5 component. Every part of the site that currently happens in ASP (which is pretty much all of it) has to get pushed out to .NET or COM or some other replacement that can be used from ASP.NET. The syntax hilighter just happens to be one of the parts that is neatly self-contained such that changing V4 of the site to use the V5 component isn't a massive undertaking.

So, consider it like this: Every part of the site is on the radar for replacement, and if it's sensible to actually upgrade a part while it's being replaced, then we'll do it.
January 02, 2008 12:16 PM
jollyjeffers
Good work - is it the 3K Python file that is now your 30k HTML file? [lol]

Any chance of getting HLSL/GLSL/Cg/FX syntax highlighting? Even Visual Studio can't hack that!

Jack

PS - you raise an interesting point about OSS and how hard it is to contribute. I've hit that before.
January 03, 2008 09:43 AM
You must log in to join the conversation.
Don't have a GameDev.net account? Sign up!
Profile
Author
Advertisement
Advertisement