Review/std.d.lexer

From D Wiki
Revision as of 15:03, 28 September 2013 by Dicebot (talk | contribs) (Created page with "== Description == [https://github.com/Hackerpilot/phobos/tree/master/std/d std.d.lexer] is standard module for lexing D code, written by Brian Schott === Related links === ...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Description

std.d.lexer is standard module for lexing D code, written by Brian Schott

Related links

Official lexing specification Documentation Example project that uses std.d.lexer

Current state

Being polished before putting on vote.

Review 1

Result

There were no critical concerns raised (by review manager opinion) but Brian has decided to address some of smaller ones before putting it on vote.

Description

Probably longest part of discussion was about requiring lookahead in lexer and related complexity/performance implications. Outcome of discussion wasn't really clear.

Other comments/proposals:

  1. using _ suffix for keywords (listed separately as raised some discussion)
  2. various other naming issues
  3. general lack of documentation
  4. TokenType definition (performance concerns)
  5. providing some benchmark for comparing with other lexerout of the box
  6. splitting it and converting into package
  7. tests need more coverage / comments
  8. sharing identifier pool between multiple lexers


Changes so far

https://github.com/Hackerpilot/phobos/commit/9bdb7f97bb8021f3b0d0291896b8fe21a6fead23#std/d/lexer.d :

  • There are a few more unit tests now
  • bitAnd renamed to amp
  • slice rename to dotdot
  • Much more cross-referencing in the doc comments
  • Start line and column can be specified in the lexer config

https://github.com/Hackerpilot/phobos/compare/D-Programming-Language:df38839...master

Clarifications about concerns

TODO: