• Jedai's avatar
    RichTokenStream support · 36104d7a
    Jedai authored
    This patch adds support for raw token streams, that contain more
    information than normal token streams (they contains comments at
    least). The "lexTokenStream" function brings this support to the
    Lexer module. In addition to that, functions have been added to
    the GHC module to make easier to recover of the token stream of 
    a module ("getTokenStream").
    Building on that, I added what could be called "rich token
    stream": token stream to which have been added the source string
    corresponding to each token, the function addSourceToToken takes
    a StringBuffer and a starting SrcLoc and a token stream and build
    this rich token stream. getRichTokenStream is a convenience
    function to get a module rich token stream. "showRichTokenStream"
    use the SrcLoc information in such a token stream to get a string
    similar to the original source (except unsignificant
    whitespaces). Thus "putStrLn . showRichTokenStream =<<
    getRichTokenStream s mod" should print a valid module source, the
    interesting part being to modify the token stream between the get
    and the show of course.
DynFlags.hs 85.7 KB