1. 09 Jan, 2006 2 commits
  2. 06 Jan, 2006 1 commit
    • simonmar's avatar
      [project @ 2006-01-06 16:30:17 by simonmar] · 9d7da331
      simonmar authored
      Add support for UTF-8 source files
      
      GHC finally has support for full Unicode in source files.  Source
      files are now assumed to be UTF-8 encoded, and the full range of
      Unicode characters can be used, with classifications recognised using
      the implementation from Data.Char.  This incedentally means that only
      the stage2 compiler will recognise Unicode in source files, because I
      was too lazy to port the unicode classifier code into libcompat.
      
      Additionally, the following synonyms for keywords are now recognised:
      
        forall symbol 	(U+2200)	forall
        right arrow   	(U+2192)	->
        left arrow   		(U+2190)	<-
        horizontal ellipsis 	(U+22EF)	..
      
      there are probably more things we could add here.
      
      This will break some source files if Latin-1 characters are being used.
      In most cases this should result in a UTF-8 decoding error.  Later on
      if we want to support more encodings (perhaps with a pragma to specify
      the encoding), I plan to do it by recoding into UTF-8 before parsing.
      
      Internally, there were some pretty big changes:
      
        - FastStrings are now stored in UTF-8
      
        - Z-encoding has been moved right to the back end.  Previously we
          used to Z-encode every identifier on the way in for simplicity,
          and only decode when we needed to show something to the user.
          Instead, we now keep every string in its UTF-8 encoding, and
          Z-encode right before printing it out.  To avoid Z-encoding the
          same string multiple times, the Z-encoding is cached inside the
          FastString the first time it is requested.
      
          This speeds up the compiler - I've measured some definite
          improvement in parsing at least, and I expect compilations overall
          to be faster too.  It also cleans up a lot of cruft from the
          OccName interface.  Z-encoding is nicely hidden inside the
          Outputable instance for Names & OccNames now.
      
        - StringBuffers are UTF-8 too, and are now represented as
          ForeignPtrs.
      
        - I've put together some test cases, not by any means exhaustive,
          but there are some interesting UTF-8 decoding error cases that
          aren't obvious.  Also, take a look at unicode001.hs for a demo.
      9d7da331