This project is mirrored from https://github.com/haskell/Cabal. Pull mirroring updated .
  1. 25 Sep, 2016 1 commit
    • Herbert Valerio Riedel's avatar
      Use hash-consing to optimise index cache (#3897) · e0dd63cc
      Herbert Valerio Riedel authored
      Without this optimisation, `cabal info somethingnonexisting` results in
      
           960,397,120 bytes allocated in the heap
           739,652,560 bytes copied during GC
            67,757,128 bytes maximum residency (24 sample(s))
             2,234,096 bytes maximum slop
                   147 MB total memory in use (0 MB lost due to fragmentation)
      
      with this optimisation:
      
         1,000,825,744 bytes allocated in the heap
           656,112,432 bytes copied during GC
            44,476,616 bytes maximum residency (24 sample(s))
             2,302,864 bytes maximum slop
                   109 MB total memory in use (0 MB lost due to fragmentation)
      
      So the total memory in use is significantly lower. The total runtime is
      also slightly reduced, from
      
        INIT    time    0.001s  (  0.001s elapsed)
        MUT     time    0.683s  (  1.050s elapsed)
        GC      time    0.946s  (  0.946s elapsed)
        EXIT    time    0.005s  (  0.005s elapsed)
        Total   time    1.637s  (  2.002s elapsed)
      
      to
      
        INIT    time    0.001s  (  0.001s elapsed)
        MUT     time    0.664s  (  0.988s elapsed)
        GC      time    0.797s  (  0.797s elapsed)
        EXIT    time    0.004s  (  0.004s elapsed)
        Total   time    1.467s  (  1.789s elapsed)
      
      
      Note that there's currently ~80k cache entries, but only ~10k unique package names
      and ~6k unique versions. So hash-consing helps reduce the amount of heap objects 
      for both value types by one order of magnitude, which among other benefits also
      reduces GC overhead.
      e0dd63cc
  2. 24 Sep, 2016 4 commits
  3. 22 Sep, 2016 2 commits
  4. 21 Sep, 2016 2 commits
    • Edward Z. Yang's avatar
      Don't solve for executables in legacy code path. · 9e99b3f4
      Edward Z. Yang authored
      
      
      There is a bug in `cabal configure`'s invocation of the solver in
      Distribution/Client/Configure.hs:
      
          standardInstallPolicy
              installedPkgIndex
              (SourcePackageDb mempty packagePrefs)
              [SpecificSourcePackage localPkg]
      
      We can see that the solver is given an EMPTY source package database.
      This is because we assume that everything you need from cabal configure
      is taken from the installed package index.
      
      But this is NOT true for executables, which never have an entry in the
      installed package index. So we SHOULD NOT solve for
      executables in the legacy codepath, since there isn't anything useful we
      can do with the info anyway.  This gets toggled using a new solver
      parameter SolveExecutables.
      
      I didn't bother with a test because this will be obsoleted by
      nix-local-build.
      
      Fixes #3875
      Signed-off-by: default avatarEdward Z. Yang <ezyang@cs.stanford.edu>
      9e99b3f4
    • Herbert Valerio Riedel's avatar
      Refactor & optimise construction of index cache · db1ef505
      Herbert Valerio Riedel authored
      This commit was motivated by @dcoutts' code-review comment:
      
      > Originally with using the `Sec.directoryEntries` that gave us only the
      > final version of each file, ie not all intermediate revisions. And
      > previously our strategy was to go through the final versions of each
      > file, in file order, and lookup just the ones we're interested in (which
      > in practice is 99% of them).
      >
      > Now for the new cache we want to go through all revisions, which means
      > all entries in file order. So instead of using `Sec.directoryEntries`
      > which reads from the tar index, we go straight for `Sec.directoryFirst`
      > which is block 0 and iterate through, using `lazyUnfold`.
      >
      > But we can now significantly simplify this and do it more
      > efficiently. Note that `indexLookupEntry` and `indexLookupFileEntry` are
      > expensive operations that seek in the tar file and read the tar entry at
      > that point. So lets do it exactly once per entry. The current code does
      > it once in the `lazyUnfold indexLookupEntry` and then again in `mk`. But
      > the old `mk` only did that because it had not previously looked up the
      > entry.
      db1ef505
  5. 20 Sep, 2016 4 commits
    • Herbert Valerio Riedel's avatar
      Try to regenerate a corrupted 01-index.cache · 92c51628
      Herbert Valerio Riedel authored
      With this commit, if a corrupted index cache is detected the
      `readIndexCache` function now regenerates the index cache and then
      reattempt to read the index once (and 'die's if it fails again).
      92c51628
    • Mikhail Glushenkov's avatar
      3ab7fc8e
    • Herbert Valerio Riedel's avatar
      Extend 01-index.cache & use 'Binary' encoding · f91e65a5
      Herbert Valerio Riedel authored
      This commit extends the index cache entries relevant for 01-index to
      include block numbers and timestamps, and makes them strict so recent
      GHCs unpack the fields:
      
          data IndexCacheEntry
              = CachePackageId PackageId BlockNo
              | CachePreference Dependency
              | CacheBuildTreeRef BuildTreeRefType BlockNo
      
      to
      
         data IndexCacheEntry
             = CachePackageId PackageId !BlockNo !Timestamp
             | CachePreference Dependency !BlockNo !Timestamp
             | CacheBuildTreeRef !BuildTreeRefType !BlockNo
      
      For the legacy `00-index.tar`s, the 'Timestamp' field is set to (-1),
      and the original 00-index.cache format is retained.
      
      For (secure) `01-index.tar`s, all of `IndexCacheEntry`s data is stored
      in the `01-index.cache` file.
      
      Moreover, to avoid having to write out and parse new two integers per
      cache entry, this patch switches to using `Binary` instances for
      encoding the `01-index.cache` file (while `00-index.cache` remains
      plain-text).
      f91e65a5
    • Edward Z. Yang's avatar
      Revert "Merge pull request #3863 from haskell/installplan-installed-state" · 026dbf06
      Edward Z. Yang authored
      This reverts commit 7b67f37b, reversing
      changes made to c12290f8.
      026dbf06
  6. 19 Sep, 2016 11 commits
    • Edward Z. Yang's avatar
      Never use --enable-profiling when invoking Setup. · bf3d3e68
      Edward Z. Yang authored
      
      
      In Cabal 1.22.5.0, the semantics of
      --disable-profiling/--enable-profiling depend on ordering (because there
      is a hack that operates by looking at the current flag assignment and
      doing something). In particular, if I specify --enable-library-profiling
      --disable-profiling, I end up with library profiling DISABLED.
      
      The fix is that we NEVER pass --enable-profiling or --disable-profiling
      to Cabal. At the moment, and according to my historical analysis, Cabal
      ONLY uses configProf to affect the effective library/executable
      profiling, which means that anything we do with --enable-profiling, we
      can do using the library/executable profiling individually. Since these
      are always flags for the versions of Cabal library we support, we will
      get order invariance. Historical versions have varied on whether or not
      setting executable profiling implies library profiling, but if we set
      both explicitly this change in behavior doesn't matter.
      
      This patch is difficult to test because the bad profiling flags
      can't be induced on an inplace build.  I tested by hand by building
      a package that depended on 'distributive' by hand.
      
      Fixes #3790.
      Signed-off-by: default avatarEdward Z. Yang <ezyang@cs.stanford.edu>
      bf3d3e68
    • Duncan Coutts's avatar
      Refactor implementation of InstallPlan.installed · 66ed37a2
      Duncan Coutts authored
      All the use sites (currently only two but soon to be three) use
      InstallPlan.installed to do a bulk change of states, differing only in
      the filter condition. So it simplifies things and shares more code if
      we make the main one be the bulk version. The InstallPlan.remove already
      works similarly.
      66ed37a2
    • Duncan Coutts's avatar
      Simplify plan improvement, avoid reading store ghc-pkg db · 5584569c
      Duncan Coutts authored
      Install plan improvement is the process of replacing configured source
      packages with installed instances from the store. Originally we did this
      by reading the ghc-pkg db to get the InstalledPackageInfo for all the
      packages in the store. We had to do that because when we replaced
      configured source packages by installed instances we used the
      PreExisting constructor which requires an InstalledPackageInfo, which we
      get from the installed package db. But now that we no longer use
      PreExisting for packages from the store we also no longer need the
      InstalledPackageInfo. All we need is a set of UnitIds. Also, once
      support for depending on executables was added then we needed a way to
      do plan improvement for executable packages/components. We did this by
      the simple approach of grabbing the dir listing for the store and using
      that as a set of UnitIds.
      
      So this patch extends the approach we use for executables and uses it
      for all packages. This means we no longer load the package db for the
      store. Note that we still have to create the package db in the store.
      
      This also relates to the locking protocol in the store. The goal for the
      store is to be able to access and update it concurrently. The locking
      protocol will include checking membership by checking if the directory
      entry for the package is present. So this patch gets us to the right
      point for the reading side, leaving the writing side to do.
      5584569c
    • Edward Z. Yang's avatar
      Print that we are building all due to Custom setup. · 512648fd
      Edward Z. Yang authored
      
      
      Previously the output was:
      
          Building profunctors-5.2 lib...
          Building semigroupoids-5.1...
      
      Now it is:
      
          Building profunctors-5.2 (lib)...
          Building semigroupoids-5.1 (all, due to Custom setup)...
      
      Fixes #3808.
      Signed-off-by: default avatarEdward Z. Yang <ezyang@cs.stanford.edu>
      512648fd
    • Edward Z. Yang's avatar
    • Duncan Coutts's avatar
      Distinguish PreExisting vs Installed in BuildStatus · f3fc6504
      Duncan Coutts authored
      Not a big deal, but should be useful later for more precise status
      reporting. For now just means the rebuild reasons can be more precise.
      f3fc6504
    • Duncan Coutts's avatar
      Remove now-unused InstallPlan.preexisting · f07db3ed
      Duncan Coutts authored
      We only ever switch Configured to Installed. The PreExisting state only
      comes from the original solver plan, which only uses installed packages
      from the global db.
      f07db3ed
    • Duncan Coutts's avatar
      Start using new InstallPlan.Installed state · 8f870aa0
      Duncan Coutts authored
      Change improvement and --dry-run phases to use Installed state rather
      than the PreExisting state. This means that PreExisting is now only used
      for installed packages from the global db, and never for installed
      packages from the store.
      8f870aa0
    • Duncan Coutts's avatar
      Update encodePlanAsJson for Installed package state · 60ce97d2
      Duncan Coutts authored
      The change in how we use the PreExisting vs Installed states means that
      we'll now have full details for all packages, rather than installed
      ones having only the subset of info available from the
      InstalledPackageInfo. So the 'type' field now can take the values
      "pre-existing", "configured" or "installed".
      
      Also do a little bit of tidying up.
      60ce97d2
    • Duncan Coutts's avatar
      Add an Installed state to InstallPlan packages · 492db5b5
      Duncan Coutts authored
      This patch just adds the state without yet using it. That'll follow in
      subsequent patches.
      
      So why add an Installed state? Didn't we just remove the Installed,
      Processing and Failed states? Those states were used when we followed
      the approach of updating the InstallPlan as a build progressed (whereas
      we now do traversals without altering the InstallPlan).
      
      The idea of adding an Installed state now is that we can more usefully
      represent the state of the plan when we "improve" the plan with packages
      from the store or when we update the plan having checked if inplace
      packages are up to date. Currently in these two places we replace
      Configured source packages with PreExisting packages. There's a couple
      problems with this. Firstly the PreExisting state only contains an
      InstalledPackageInfo which means we loose information compared to all
      the detail in the Configured source package. This is relevant for things
      like plan.json output or other features that want to know the status of
      a project. Secondly we have to fake things for executables since they
      are not properly represented by InstalledPackageInfo.
      492db5b5
    • Herbert Valerio Riedel's avatar
      Store secure repo index data as 01-index.* (#3862) · dc889b17
      Herbert Valerio Riedel authored
      "Secure" cabal repositories use a new extended & incremental
      `01-index.tar`. In order to avoid issues resulting from clobbering
      new/old-style index data, we save them locally to different names.
      
      With this patch, secure repos generate/update the files below on `cabal update`
      
      - `01-index.cache`
      - `01-index.tar`
      - `01-index.tar.gz`
      - `01-index.tar.idx`
      - `mirrors.json`
      - `root.json`
      - `snapshot.json`
      - `timestamp.json`
      
      ...while the legacy codepaths for non-secure repos operate on the files
      
      - `00-index.cache`
      - `00-index.tar`
      - `00-index.tar.gz`
      - `00-index.tar.gz.etag`
      
      This way the old/new codepaths don't interfere with each other anymore.
      
      Note: The format of `01-index.cache` file will be extended by the upcoming `--index-state` feature
      
      This trivially fixes #3854
      dc889b17
  7. 18 Sep, 2016 5 commits
    • Herbert Valerio Riedel's avatar
      Revert accidental renaming of `packageKey` · 0986e11b
      Herbert Valerio Riedel authored
      This was introduced accidentally via
      b083151f
      
      /cc @ezyang
      0986e11b
    • Brendan Hay's avatar
      Track project configuration provenance · c2ebd714
      Brendan Hay authored
      - Excludes provenance from all roundtrip tests
      - Inserting explicit provenance when read from file
      - Special cases for bad package errors arising from
        implicit project configuration
      c2ebd714
    • Edward Z. Yang's avatar
      Renumber flags version numbers in 'filterConfigureFlags'. · 2fa18577
      Edward Z. Yang authored
      
      
      Previously, the code was inconsistent on whether or not
      flags_x_y_z indicated that these flags could be used
      up to version x.y.z, or should be used prior to x.y.z.
      This commit picks the LATTER and renames everything
      consistently this way.  The bonus is that now the names
      match up with the conditionals. Yay.
      Signed-off-by: default avatarEdward Z. Yang <ezyang@cs.stanford.edu>
      2fa18577
    • Edward Z. Yang's avatar
      Refactor ComponentEnabledSpec into ComponentRequestedSpec. · d03fe594
      Edward Z. Yang authored
      
      
      In the previous documentation for 'ComponentEnabledSpec', I claimed
      that enabled components were buildable, as well as requested
      by the user.  In the course of working on #3847, however,
      I realized that I hadn't actually *checked* that the component
      was buildable anywhere.  In particular, the 'ComponentDisabled'
      reason was *never used*.  This mostly didn't cause problems,
      however, because when we 'flattenPD' all non-buildable components
      get deleted, so you basically never actually have a non-buildable
      'Component'.
      
      But it still seemed a bit silly, so I fixed it by doing this:
      
      1) I introduce a new concept of a component being requested,
      which captures the use of --enable-tests and friends.  This
      does NOT imply buildability.  Renamed ComponentEnabledSpec
      to ComponentRequestedSpec
      
      2) A component is enabled if it is requested and buildable.
      If you give me a Component and a ComponentRequestedSpec I
      can tell you if it's enabled.  However, if you give me a
      ComponentName I can't, because I have no idea if it's buildable.
      
      3) Stopped reexporting ComponentRequestedSpec from
      Distribution.Simple.LocalBuildInfo
      
      4) Added a test for attempting to specify a non-buildable
      component as a target.  The test is accepting suboptimal
      output at the moment, see #3858.
      Signed-off-by: default avatarEdward Z. Yang <ezyang@cs.stanford.edu>
      d03fe594
    • Edward Z. Yang's avatar
      Print selected configure flags in install plan. · 52e8bfd0
      Edward Z. Yang authored
      
      
      When debugging #3827, I wished I could see what the profiling
      status of all packages in our install plan was.  This patch
      outputs this information, by printing the *configure flag*
      we would have passed to configure.
      
      In an ideal world, we'd print exactly the configure flags
      we actually pass in practice.  But this is not so great because
      the flag lists tend to be really long.  So we pare it down
      in the following ways:
      
      1. Not all flags are output: only a selected few.  In this
         patch, it's just enabling/disabling profiling flags, but
         we could add more.
      
      2. If we have Flag False for profiling, we will actually
         explicitly pass --disable-profiling to the configure
         script.  But it's not so good to output that to the
         user.  So 'nubFlag' lets us drop the flag, if omitting
         it would have the same effect as specifying it.
         Figuring that out is a little tricky though.
      Signed-off-by: default avatarEdward Z. Yang <ezyang@cs.stanford.edu>
      52e8bfd0
  8. 17 Sep, 2016 1 commit
  9. 16 Sep, 2016 1 commit
  10. 15 Sep, 2016 1 commit
  11. 14 Sep, 2016 1 commit
  12. 13 Sep, 2016 2 commits
  13. 10 Sep, 2016 4 commits
  14. 09 Sep, 2016 1 commit