What made the fast-lexer branch a priority? Several things. It started when @dockimbel looked into ticket #3606, which was impossible to fix currently, and we didn’t want to give up on the auto-syncing between /text and /data facets. So he had to consider bigger options, including how to make the lexer instrumentable. It was not easy, because the current lexer is not re-entrant, so having the lexer emit events to a callback function could have caused serious problems.
Over the last few weeks the Red Lang core team drilled down to make some truly great progress on Red’s fast-lexer branch–while we also gained valuable support from the contributions of Red doers and makers as they consolidate a world of useful information and resources. Fast-Lexer Benchmarks In the fast-lexer branch of Red, you can see lots of new work from Red creator @dockimbel (Nenad Rakocevic) and core teammate @qxtie.