This project is mirrored from Pull mirroring updated .
  1. 06 Oct, 2016 17 commits
    • Edward Z. Yang's avatar
      Add the bulk of the new Backpack code · f2840cca
      Edward Z. Yang authored
      This will shortly be used in the package configuration step.
          A copy of the Progress monad from cabal-install solver, which we use
          to report failure and logging from otherwise pure code.
          Defines a type class ModSubst for semantic objects that can have
          module substitutions defined over them.  Helps us define ModSubst
          A module scope represents the set of required and provided modules
          which are "in scope", i.e., available for import and mix-in linking.
          This is not a simple mapping from module name to module, as we're
          permitted to have conflicting definitions for a module name as long
          as we don't actually use it.  There's a comment which explains this
          more clearly in the file.  These are based off of 'IndefModule'
          because not all modules in scope are necessarily fully instantiated.
          A module shape describes the provisions and requirements of a
          library.  It's similar to a module scope, except that every
          export must be unambiguous; it too is based off of 'IndefModule'.
          An 'IndefUnitId' is not guaranteed to record a module substitution
          (it could be opaquely represented as a hash); a 'FullUnitId',
          however, IS guaranteed to do so.  Given, for example, an installed
          package database, we can map opaque 'UnitId' into their expanded
          representation.  This can be important to handle obscure unification
          problems when mix-in linking.
          The unification monad, unifiable variants of Module/UnitId (with
          conversions to and from), and low-level unification operations on
          them.  Uses 'UnionFind' heavily.
          There is some commented out support for handling recursive
          unification.  At present there is no surface syntax for representing
          such situations.  (We would also need DFA minimization to
          canonicalize these regular trees.)
          The actual implementation of mix-in linking, on top of the
          unification monad 'UnifyM'.  The key function is 'mixLink',
          which takes two module scopes and links them together.
          The progress monad, specialized with LogMsg traces and Doc
          errors.  We provide a function to run such computations,
          outputting traces according to their 'Verbosity' and 'die'ing
          if there is an error.
          A small helper function for handling the source-level graph
          of components (so, just the relationship between stanzas
          in a Cabal file.)  This components graph will eventually get
          elaborated into a more complex graph with instantiations, etc.
          A helper module which now contains the functions for computing
          component identifiers and compatibility package keys.  This
          functionality used to live in Distribution.Simple.Configure
          but I split it out.  There are also adjustments to handle
          the new Backpack functionality.
          A configured component is one for which we've resolved all
          source level dependencies (e.g., every entry in build-depends,
          we know the specific transitive versions of each thing
          we're going to use.)  That means we have a 'ComponentId' for
          this component.  This module also contains functions for
          creating a 'ConfiguredComponent' from a source 'Component'.
          A linked component is one which we have done mix-in linking
          for, so we know its 'IndefUnitId' and its 'IndefUnitId'
          This module calls out to mix-in linking to actually do linking.
          The workhorse, in a sense!
          This module implements the instantiation process, where we
          zip through all of the fully instantiated components, and
          recursively instantiate their dependencies, so that we
          get a separate linked component per thing we need to
          compile, and also finishes off any top-level indefinite
          components.  This gives us the final 'UnitId' for our
          This functionality is reimplemented in a different way in
          cabal-install; the assumptions are slightly different (in
          particular, in the library we can't assume we have access to all
          packages to build them; but in cabal-install we can assume it) so I
          didn't try to abstract over both implementations.
          This is a "interoperability" data type which identifies
          precisely the information from a 'LinkedComponent' that
          can be derived from an 'InstalledPackageInfo'.
    • Edward Z. Yang's avatar
      Extend ComponentLocalBuildInfo with backpack info · be1a184c
      Edward Z. Yang authored
      (1) add 'componentInstantiatedWith' to record how a component was
      instantiated (analogous to @instantiated-with@) and
      (2) fix 'componentComponentId' for the new constructors in 'Module'.
    • Edward Z. Yang's avatar
      Replace the module renaming/thinning system · 1017f710
      Edward Z. Yang authored
      We had an old implementation of 'ModuleRenaming', with the
      assumption that it would be used directly in build-depends; since
      we have dropped this assumption, we can refactor 'ModuleRenaming'
      and we do so.  The main idea is to make the data type more directly
      reflect the syntax you can specify in a Cabal file; so the default
      renaming and an explicit thinning renaming are now different
      constructors.  It's no longer possible to use the "with" syntax, but
      it's not necessary either, since we have a special backpack-includes
      field to specify renamings, so we don't need them to be Monoidal.
      There is also a new syntax for 'hiding', which just lets you hide
      some modules when including a package. Handy!
      Previously, we recorded 'ModuleRenaming' in @build-depends@, but
      separated it out when we stored in 'BuildInfo'.  We now go even
      further, by changing it from a 'Map' (the only thing @build-depends@
      could support) to a list (so that a package name can be specified
      multiple times.)  This is good because now a user can instantiate
      something several times, which is useful in Backpack.
      Also add the new field @backpack-includes@ which can be used to exert
      fine-grained control over what modules a package brings into scope,
      include it multiple times, etc.
      In the .cabal checks, replace 'depsUsingThinningRenamingSyntax' with a
      more direct check to see if @backpack-includes@ was used.
      Dropped the legacy 'lookupRenaming' export from ModuleRenaming and
      PackageDescription; we will shortly not use it anymore. As an
      intermediate hack we have a local definition in Configure, but this
      will go away shortly.
    • Edward Z. Yang's avatar
      Rename .cabal required-signatures field to signatures · 8d31f43b
      Edward Z. Yang authored
      Change of .cabal file syntax: rename @required-signatures@ field to
      just @signatures@. Update the parser and error messages that mention
      the field.
      Also rename the corresponding field in the Library type.
    • Edward Z. Yang's avatar
      Backpack InstalledPackageInfo representation changes · 3de0e4c4
      Edward Z. Yang authored
      New field, @instantiated-with@, which records the full
      module substitution (it is dropped when we do 'improveUnitId').
      For flexibility in the case of indefinite packages, some
      occurences of Module are relaxed to IndefModule (exposedReexport).
      This is just for convenience; in the case of a definite package
      these reexports and instantiations are guaranteed to be 'Module's.
      This patch also includes the minimal changes in other modules
      needed due to the representation change.
    • Edward Z. Yang's avatar
      Support for GHC flags -instantiated-with and -fno-code · 305935d9
      Edward Z. Yang authored
      New GhcOptions fields, 'ghcOptInstantiatedWith' for
      @-instantiated-with@ and 'ghcOptNoCode' for @-fno-code@.
    • Edward Z. Yang's avatar
      Update GhcOptions to use IndefUnitId for -package-id · 5b378e48
      Edward Z. Yang authored
      We will use this when building indefinite libraries.
    • Edward Z. Yang's avatar
      Add D.Backpack module with IndefUnitId and IndefModule types · fdf30f80
      Edward Z. Yang authored
      This module defines the key data types 'IndefUnitId' and
      'IndefModule', which represent components which are partially
      instantiated with holes in them.  The intent is that they
      can be substituted over (Distribution.Backpack.ModSubst),
      and once all the holes are filled they can be improved into
      proper 'UnitId's, which can then be installed.
      Also add Distribution.Util.Base62 module containing an
      implementation of base-62 encoding, which we use when
      computing hashes for fully instantiated unit ids.
    • Edward Z. Yang's avatar
      Introduce the new representation of UnitId · d7bd9078
      Edward Z. Yang authored
      'SimpleUnitId' constructor renamed to 'UnitId', and augmented
      with a new field 'Maybe String' recording a hash that uniquely
      identifies an instantiated unit of the library 'ComponentId'.
      'UnitId' can't be used to represent partially instantiated
      unit identifiers; see Distribution.Backpack for how we handle that.
      Previous uses of 'SimpleUnitId' should now use 'newSimpleUnitId'.
      'unitIdComponentId' folded into a record selector for 'ComponentId'.
    • Edward Z. Yang's avatar
      Add a parser for 'key' field of InstalledPackageInfo · cbc1a1de
      Edward Z. Yang authored
      Rather than reusing the one for ComponentId.
    • Edward Z. Yang's avatar
      A minor improvement to convenience libraries · 7637d067
      Edward Z. Yang authored
      We force convenience libraries to NOT be exposed (they're private,
      after all.)  This will help us on some of our test cases.
    • Edward Z. Yang's avatar
    • Edward Z. Yang's avatar
      Update 'getHaskellObjects' to take 'ComponentLocalBuildInfo' · 9e8fea3e
      Edward Z. Yang authored
      Eliminating the use of deprecated 'getComponentLocalBuildInfo'
    • Edward Z. Yang's avatar
    • Edward Z. Yang's avatar
    • Edward Z. Yang's avatar
      Add more reexported module tests · 6059c6b3
      Edward Z. Yang authored
      Signed-off-by: default avatarEdward Z. Yang <>
    • Edward Z. Yang's avatar
  2. 04 Oct, 2016 4 commits
    • Duncan Coutts's avatar
      Optimise Distribution.Version.mkVersion · 883b3c30
      Duncan Coutts authored
    • Herbert Valerio Riedel's avatar
      Fix thinko in `decodeStringUtf8` · ed329074
      Herbert Valerio Riedel authored
      This resulted in some two-bytes utf8 encodings to be decoded
      into U+FFFD unintentionally (such as e.g. U+0142).
      With this fix, the property
          [ c | c <- [minBound..maxBound]
              , c < '\xD800' || c >= '\xE000' -- surrogate pair codes
              , (decodeStringUtf8 . encodeStringUtf8) [c] /= [c]
              ] == ['\xfffe','\xffff']
      holds. It's not clear to me why U+FFFE and U+FFFF ought to be singled
      out. Needs more investigation.
      TODO: testsuite coverage
    • Herbert Valerio Riedel's avatar
      Fix embarassing thinko in `encodeStringUtf8` · a87fcd10
      Herbert Valerio Riedel authored
      TODO: we need more unit-tests to catch this; also I noticed
      that `decodeStringUtf8` (which was previously named `fromUTF8BSImpl`)
      appears to have been broken for quite some time.
    • Herbert Valerio Riedel's avatar
      Add Data.Word/Int to Compat.Prelude · e14da30b
      Herbert Valerio Riedel authored
      For one this compensates for `Word` being exported from `Prelude` only with
      base >= 4.8, and moreover this brings the well-known {Int,Word}{8,16,32,64}
      names into scope by default when using the Compat.Prelude.
  3. 03 Oct, 2016 1 commit
    • Herbert Valerio Riedel's avatar
      Optimise internal representation of `ModuleName` (#3927) · b0e958a3
      Herbert Valerio Riedel authored
      This changes the representation of module names from `[[Char]]`
      (e.g. `Control.Monad.Fail` is decomposed into `["Control","Monad","Fail"]`)
      which results in many small heap objects, to a strict list of `ShortText`s.
      `ModuleName` was already an opaque type, so there is no visible change
      to the exposed API.
  4. 02 Oct, 2016 5 commits
    • Herbert Valerio Riedel's avatar
      Make `AbiHash` opaque & `ShortText`-backed (#3921) · 7378066d
      Herbert Valerio Riedel authored
      This doesn't have any noteworthy performance impact since
      there aren't that many `AbiHash` values to keep track of and is
      rather for consistency with the other opaque types.
    • Herbert Valerio Riedel's avatar
    • Herbert Valerio Riedel's avatar
      Optimise `Ord Version` instance · aabee7c3
      Herbert Valerio Riedel authored
      Profiling showed that there are quite a few 'Ord' calls (~70k calls)
      which result in 'versionNumbers' being evaluated because the need to
      compare with Versions not fitting in a Word64 becomes necessary, mostly
      when inserting into a `Map`. So I've optimised this a bit more.  After
      some experimentation, the new `compare` implemetation results in faster
      comparisions (`cmpOpt` is the new optimised impl, `compare` is the
      previous naive one):
      comparing two PV1 constructors:
          benchmarking compare
          time                 42.33 ns   (42.17 ns .. 42.47 ns)
                               1.000 R²   (1.000 R² .. 1.000 R²)
          mean                 42.17 ns   (42.13 ns .. 42.28 ns)
          std dev              205.1 ps   (128.7 ps .. 325.7 ps)
          benchmarking cmpOpt
          time                 33.31 ns   (33.23 ns .. 33.40 ns)
                               1.000 R²   (1.000 R² .. 1.000 R²)
          mean                 33.37 ns   (33.29 ns .. 33.46 ns)
          std dev              288.6 ps   (242.9 ps .. 315.8 ps)
          benchmarking compare [] []
          time                 31.92 ns   (31.89 ns .. 31.94 ns)
                               1.000 R²   (1.000 R² .. 1.000 R²)
          mean                 31.89 ns   (31.88 ns .. 31.91 ns)
          std dev              37.09 ps   (24.38 ps .. 54.82 ps)
      comparing a PV0 constructor with a PV1 constructor when the first digit
      decides the outcome:
          benchmarking compare 2.3.4
          time                 24.96 ns   (24.78 ns .. 25.25 ns)
                               0.996 R²   (0.989 R² .. 1.000 R²)
          mean                 25.50 ns   (24.95 ns .. 26.95 ns)
          std dev              2.802 ns   (1.275 ns .. 5.163 ns)
          variance introduced by outliers: 93% (severely inflated)
          benchmarking cmpOpt  2.3.4
          time                 11.29 ns   (11.27 ns .. 11.30 ns)
                               1.000 R²   (1.000 R² .. 1.000 R²)
          mean                 11.28 ns   (11.27 ns .. 11.29 ns)
          std dev              24.69 ps   (12.02 ps .. 46.86 ps)
          benchmarking compare [2.3.4] []
          time                 11.05 ns   (11.04 ns .. 11.06 ns)
                               1.000 R²   (1.000 R² .. 1.000 R²)
          mean                 11.05 ns   (11.04 ns .. 11.06 ns)
          std dev              32.82 ps   (15.89 ps .. 61.64 ps)
      These timings are now very close to the 'Ord [Int]' timings.
      With this optimisation, the total number of 'versionNumbers' calls
      reported by `+RTS -p` for `cabal new-build --dry`'ing haskell-ide-engine
      went down from originally 73501 calls to 6789 calls.
    • Herbert Valerio Riedel's avatar
      Optimise `Version` representation · cbddd8cb
      Herbert Valerio Riedel authored
      This optimises the `[Int]` representation to a 16-byte heap object for
      ~99% of version numbers (up to 4 components, each within a [0..0xfffe]
      value range) occuring on Hackage.
      One noteworthy improvement of this optimisation is a significant reduction
      of the size of the 01-index.cache file from previously 6299700 bytes
      (before #3905) down to 3408864 bytes, i.e. down to ~54%
      Also, this reduces the memory footprint and GC overhead a bit for
      e.g. `cabal info zzz` (which reads in the index cache) from
          cabal.0: There is no package named 'zzz'.
          You may need to run 'cabal update' to get the latest list of available
               859,337,368 bytes allocated in the heap
               447,261,128 bytes copied during GC
                37,385,208 bytes maximum residency (19 sample(s))
                 1,311,136 bytes maximum slop
                       103 MB total memory in use (0 MB lost due to fragmentation)
                                               Tot time (elapsed)  Avg pause  Max pause
            Gen  0      1613 colls,     0 par    0.268s   0.268s     0.0002s    0.0012s
            Gen  1        19 colls,     0 par    0.227s   0.227s     0.0119s    0.0506s
            TASKS: 4 (1 bound, 3 peak workers (3 total), using -N1)
            SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)
            INIT    time    0.001s  (  0.001s elapsed)
            MUT     time    0.431s  (  0.758s elapsed)
            GC      time    0.495s  (  0.495s elapsed)
            EXIT    time    0.006s  (  0.005s elapsed)
            Total   time    0.934s  (  1.259s elapsed)
            Alloc rate    1,991,870,623 bytes per MUT second
            Productivity  46.9% of total user, 34.8% of total elapsed
          cabal.1: There is no package named 'zzz'.
          You may need to run 'cabal update' to get the latest list of available
               834,314,392 bytes allocated in the heap
               440,791,176 bytes copied during GC
                36,663,112 bytes maximum residency (19 sample(s))
                 2,225,040 bytes maximum slop
                        96 MB total memory in use (0 MB lost due to fragmentation)
                                               Tot time (elapsed)  Avg pause  Max pause
            Gen  0      1574 colls,     0 par    0.254s   0.254s     0.0002s    0.0007s
            Gen  1        19 colls,     0 par    0.223s   0.223s     0.0118s    0.0474s
            TASKS: 4 (1 bound, 3 peak workers (3 total), using -N1)
            SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)
            INIT    time    0.001s  (  0.001s elapsed)
            MUT     time    0.383s  (  0.699s elapsed)
            GC      time    0.477s  (  0.477s elapsed)
            EXIT    time    0.005s  (  0.005s elapsed)
            Total   time    0.869s  (  1.182s elapsed)
            Alloc rate    2,175,866,164 bytes per MUT second
            Productivity  44.9% of total user, 33.0% of total elapsed
    • Edward Z. Yang's avatar
      GHC 8.1 compatibility fixes. · effb4ea6
      Edward Z. Yang authored
      It seems that GHC HEAD is now unable to infer the correct
      type for "f . g" when g has a higher-rank type.  So eta-expand
      all such occurrences to keep the build working.
      Signed-off-by: default avatarEdward Z. Yang <>
  5. 01 Oct, 2016 3 commits
  6. 30 Sep, 2016 2 commits
  7. 29 Sep, 2016 3 commits
    • Herbert Valerio Riedel's avatar
    • Herbert Valerio Riedel's avatar
      Move ShortText to Distribution.Utils.ShortText to avoid cycles · e204a346
      Herbert Valerio Riedel authored
      This moves `String`/UTF8 conversion helpers to Distribution.Utils.String
    • Duncan Coutts's avatar
      Adjust behaviour of reportBuildFailures · 4d308c04
      Duncan Coutts authored
      Make sure it consistently uses stderr, rather than a mixture of stdout
      and stderr. Also rename it to dieOnBuildFailure to make it clear that
      it is fatal in the case of build failures.
      In general we report build failures in two steps: header plus build log
      for each failing package, then a final die exception with summary.
      Previously the build log step was reported to stdout whereas the die
      exception message gets reported to stderr. So we switch that to use
      new dieMsg and dieMsgNoWrap utils so that it all goes to stderr. Also,
      like die, these are reported irrespective of the verbosity.
      This is more or less just a workaround for the fact that we do not yet
      have a nice structured/formatted error mechanism that would allow us to
      throw just the one error in this case.
  8. 28 Sep, 2016 2 commits
    • Herbert Valerio Riedel's avatar
      Add `Distribution.Simple.Utils.ShortText` type (#3898) · 993d20a2
      Herbert Valerio Riedel authored
      This implements a type with a compact representation of `[Char]`.
      The data is stored internally as UTF8 in an 'Data.ByteString.Short.ShortByteString'
      when compiled against `bytestring >= 0.10.4`, and otherwise in a
      plain old `[Char]`.
      `ShortByteString` is available only from `bytestring` 0.10.4 on, and GHC
      7.8.4 was the first GHC to bundle `binary-0.10.4`. So this fallback
      affects mostly only GHC 7.6 and older.
      Note: Originally a strict `ByteString` was used as fallback for this patch. However, the 
      `[Char]` fallback avoids pinned memory and may be more preferable when dealing with
      many small `ShortText`s
    • Herbert Valerio Riedel's avatar
      Make `Version` type opaque (#3905) · bb2026c4
      Herbert Valerio Riedel authored
      Similiar to dabd9d98 which made
      `PackageName` opaque, this makes `Distribution.Version.Version` opaque.
      The most common version numbers occuring on Hackage are 3- and
      4-part versions. This results in significant Heap overhead due to
      `Data.Version`'s inefficient internal representation.
      So like the `PackageName` commit, this commit is a preparatory commit to
      pave the way for replacing `Version`'s internal representation by a
      representation with a memory footprint which can be an order of
      magnitude smaller.
      Finally, this also adds a new functor-like convenience function
          alterVersion :: ([Int] -> [Int]) -> Version -> Version
      for modifying the version number components.
  9. 27 Sep, 2016 1 commit
    • Herbert Valerio Riedel's avatar
      Make `PackageName` type opaque (#3896) · dabd9d98
      Herbert Valerio Riedel authored
      When looking at heap-profiles of `cabal-install`, the `(:)` constructor
      stands out as the most-allocated constructor on the heap.
      Having to handle 10k+ package names contributes to the allocation
      numbers, especially on 64bit archs where ASCII `String`s have a 24 byte
      per character footprint.
      This commit is a preparatory commit to pave the way for changing
      `PackageName`'s internal representation to something like
      `ShortByteString` (which is essentially a thin wrapper around primitive
      `ByteArray#`s which themselves have have an overhead of 2 words + one
      byte per ASCII character rounded up to nearest word) which would allow
      to reduce the memory footprint by a full order of magnitude, as well as
      reduce pointer chasing and GC overhead.
  10. 26 Sep, 2016 2 commits
    • Herbert Valerio Riedel's avatar
      Introduce `Distribution.Compat.Prelude.Internal` + WARNING · 4c730f58
      Herbert Valerio Riedel authored
      This unexposes `Distribution.Compat.Prelude` again, and adds
      an exposed `.Internal` module which reexports the `.Prelude` module with
      an attached module-level `WARNING` pragma.
      This way users are discouraged to use this in `Setup.hs` files as they'd
      see compile warning like:
          Foo.hs:19:1-55: warning: [-Wdeprecations]
            Module ‘Distribution.Compat.Prelude.Internal’:
              This modules' API is not stable. Use at your own risk, or better yet, use @base-compat@!
      In `cabal-install` we import the `.Internal` module exactly once, and
      use -Wno-deprecations to suppress that the warning of that single
    • Herbert Valerio Riedel's avatar
      Introduce `Distribution.Client.Compat.Prelude` · 82a22706
      Herbert Valerio Riedel authored
      This is supposed to become more or less a superset of Cabal's
      As a side-effect,t his exposes `Distribution.Compat.Prelude` from the
      Cabal library (which may be actually a good thing, as it may be useful
      module to Setup.hs writers).