This patch adds support for raw token streams, that contain more information than normal token streams (they contains comments at least). The "lexTokenStream" function brings this support to the Lexer module. In addition to that, functions have been added to the GHC module to make easier to recover of the token stream of a module ("getTokenStream"). Building on that, I added what could be called "rich token stream": token stream to which have been added the source string corresponding to each token, the function addSourceToToken takes a StringBuffer and a starting SrcLoc and a token stream and build this rich token stream. getRichTokenStream is a convenience function to get a module rich token stream. "showRichTokenStream" use the SrcLoc information in such a token stream to get a string similar to the original source (except unsignificant whitespaces). Thus "putStrLn . showRichTokenStream =<< getRichTokenStream s mod" should print a valid module source, the interesting part being to modify the token stream between the get and the show of course.
Showing with 127 additions and 23 deletions