Split up PState
The PState
record is huge (25 fields) and will be updated on every token lexed. That's about 25*8=200 bytes allocation for every byte lexed, for PState
update alone! I see that we already store a ParserOpts
field in PState
that will probably be mostly constant throughout. But there are also a lot of other fields which probably don't change that often, which we could push into a secondary state parameter. E.g. maybe we can manage to have a HotPState
(buffer
, last_tk
, *loc
, ...) that is small and changes for every AlexToken
emitted and a SlowPState
(warnings
, errors
, tab_fist
, ...) that changes only sometimes. Maybe split up that one too into a State record for things that are modified together. We could thread them as separate parameters in the P
monad or as a tuple/strict PState
record.