Jump to content
  • Advertisement
Sign in to follow this  
Eliad Moshe

lexertl encapsulation

This topic is 2670 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I made a modest lexertl "wrapper":

#ifndef LEXER
#define LEXER

class Lexer {


lexertl::rules rules_;
lexertl::state_machine state_machine_;
lexertl::match_results * results_ ;


void SetBuf( char * str );
Lexer & operator () (char * regexp , int Id); // Add Rule
void Build();
int Lookup( char * & Token ) ;



& :

#include "Lexer.h"

Lexer :: Lexer(){}

Lexer & Lexer :: operator () (char * regexp , int Id){
rules_.add (regexp, Id);
return *this ;

void Lexer :: Build(){ lexertl::generator::build (rules_, state_machine_); }

void Lexer :: SetBuf( char * str ){

//if(results_ != 0) {delete results_ ; results_ = 0 ;}

std::string input_ (str);
std::string::const_iterator iter_ = input_.begin ();
std::string::const_iterator end_ = input_.end ();

results_ = new lexertl::match_results (iter_, end_) ;


int Lexer :: Lookup( char * & Token ) {

lexertl::lookup ( state_machine_, *results_ );

std::string Tok ((results_)->start, (results_)->end) ;
Token = (char *) Tok.c_str();
std :: cout << Tok ;

return (results_)->id ;


It compiles fine , however i receives :

"Debug assertion failed...
string iterators incompatible..."

While() using Breakpoints i saw that, there is some problem with the

lexertl::match_results * results_ pointer. (points to an invalid match_results struct after it's dynamic allocation).

I'm not sure why.

Some useful snippets:

typedef basic_match_results<std::string::const_iterator, std::size_t>

namespace lexertl
template<typename iter, typename id_type = std::size_t>
struct basic_match_results
typedef iter iter_type;
typedef typename std::iterator_traits<iter_type>::value_type char_type;
typedef typename basic_char_traits<char_type>::index_type index_type;

id_type id;
id_type unique_id;
iter_type start;
iter_type end;
iter_type eoi;
bool bol;
id_type state;

basic_match_results (iter_type &start_, iter_type &end_) :
id (0),
unique_id (npos ()),
start (start_),
end (start_),
eoi (end_),
bol (true),
state (0)

virtual ~basic_match_results ()

static id_type npos ()
return static_cast<id_type>(~0);

Share this post

Link to post
Share on other sites

I am Ben Hanson the author of lexertl. If you want to wrap lexertl in that way I would make the match_results an object instead of a pointer. You are probably better off using lexertl::basic_match_results<const char *, std::size_t> and then you can construct the results with two zeros. When you come to set the match results with input, just assign to the start and end members.

Note that in your example you have input on the stack, set match results with iterators to that input and then when you exit that function the input string is destroyed! This explains your crash. You could have an input string in your class along with the lexertl objects to avoid that.




Share this post

Link to post
Share on other sites
I have just uploaded some improvements:

match_results now has a default constructor and reset() and clear() functions.

That should make things a lot easier! Note that the manual method I mentioned earlier was missing eoi. Better to use the new reset function...

Thanks for your feedback - I would have done this earlier but it didn't work under VC++6! The minimum compiler I support now is VC++ 7.1 (2003), which thankfully still works fine.



Share this post

Link to post
Share on other sites
Sign in to follow this  

  • Advertisement

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

GameDev.net is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!