Lexer: avoid predicates in varsym rules
The varsym
lexing rules are defined as follows:
<0> {
@varsym / { precededByClosingToken `alexAndPred` followedByOpeningToken } { varsym_tight_infix }
@varsym / { followedByOpeningToken } { varsym_prefix }
@varsym / { precededByClosingToken } { varsym_suffix }
@varsym { varsym_loose_infix }
}
Unfortunately, this means that the predicates precededByClosingToken
and followedByOpeningToken
will be recomputed several times before we figure out the whitespace context. Rather than check for closing/opening tokens using lexer predicates, I'd rather do it in the lexer action:
<0> {
varsym { varsym }
}
varsym :: Action
varsym span buf len = ... -- check for opening/closing tokens here
But this isn't currently possible because the lexer action doesn't give access to the buffer state after the token. Only the predicates have this information. This is fixable by passing around an additional parameter to every lexer action:
- type Action = PsSpan -> StringBuffer -> Int -> P (PsLocated Token)
+ type Action = PsSpan -> StringBuffer -> Int -> StringBuffer -> P (PsLocated Token)
Hopefully, this will improve the lexer performance ever so slightly. But even if not, it should make the handling of operator whitespace a bit cleaner.