This project is mirrored from Pull mirroring updated .
  1. 26 Sep, 2016 5 commits
  2. 25 Sep, 2016 2 commits
    • Duncan Coutts's avatar
      Add comment on MonitorStateFileSet being a bag [skip ci] · 2fb4de6c
      Duncan Coutts authored
      Explain why it's not represented by a Set.
    • Herbert Valerio Riedel's avatar
      Use hash-consing to optimise index cache (#3897) · e0dd63cc
      Herbert Valerio Riedel authored
      Without this optimisation, `cabal info somethingnonexisting` results in
           960,397,120 bytes allocated in the heap
           739,652,560 bytes copied during GC
            67,757,128 bytes maximum residency (24 sample(s))
             2,234,096 bytes maximum slop
                   147 MB total memory in use (0 MB lost due to fragmentation)
      with this optimisation:
         1,000,825,744 bytes allocated in the heap
           656,112,432 bytes copied during GC
            44,476,616 bytes maximum residency (24 sample(s))
             2,302,864 bytes maximum slop
                   109 MB total memory in use (0 MB lost due to fragmentation)
      So the total memory in use is significantly lower. The total runtime is
      also slightly reduced, from
        INIT    time    0.001s  (  0.001s elapsed)
        MUT     time    0.683s  (  1.050s elapsed)
        GC      time    0.946s  (  0.946s elapsed)
        EXIT    time    0.005s  (  0.005s elapsed)
        Total   time    1.637s  (  2.002s elapsed)
        INIT    time    0.001s  (  0.001s elapsed)
        MUT     time    0.664s  (  0.988s elapsed)
        GC      time    0.797s  (  0.797s elapsed)
        EXIT    time    0.004s  (  0.004s elapsed)
        Total   time    1.467s  (  1.789s elapsed)
      Note that there's currently ~80k cache entries, but only ~10k unique package names
      and ~6k unique versions. So hash-consing helps reduce the amount of heap objects 
      for both value types by one order of magnitude, which among other benefits also
      reduces GC overhead.
  3. 24 Sep, 2016 15 commits
    • Mikhail Glushenkov's avatar
      Merge pull request #3894 from hvr/pr/rtsopts · ecd2cb1f
      Mikhail Glushenkov authored
      Enable `-rtsopts` for `cabal` executable
    • Herbert Valerio Riedel's avatar
      Merge PR #3893 (Implement `--index-state` aka index freezing) · 1535bd3e
      Herbert Valerio Riedel authored
      This PR implements a new flag `--index-state` (and its `cabal.project`/config-file
      equivalent `index-state: ...`) which allows to change the source package index
      state the solver uses to compute install-plans. This is particularly useful in
      combination with freeze-files in order to also freeze the state the package
      index was in at the time the install-plan was frozen.
      This provides the core functionality, on which future enhancements can build
      upon. See also description of PR #3604 for some possible enhancements.
    • Herbert Valerio Riedel's avatar
    • Herbert Valerio Riedel's avatar
      Enable `-rtsopts` for `cabal` executable · 5edaa0c4
      Herbert Valerio Riedel authored
      Fwiw, the GHC executables are compiled with full `-rtsopts` as well.
      The benefit is that one can use flags such as `-M1G` to limit the heap
      space, or profile memory usage via `-hT`.
    • Herbert Valerio Riedel's avatar
    • Herbert Valerio Riedel's avatar
    • Herbert Valerio Riedel's avatar
    • Herbert Valerio Riedel's avatar
      Turn 'Timestamp' into a proper type (#3892) · 060b9061
      Herbert Valerio Riedel authored
      Most notably, this allows us to provide a custom instance for `Text`
    • Duncan Coutts's avatar
      Refactor implementation of InstallPlan.installed · d5288df0
      Duncan Coutts authored
      All the use sites (currently only two but soon to be three) use
      InstallPlan.installed to do a bulk change of states, differing only in
      the filter condition. So it simplifies things and shares more code if
      we make the main one be the bulk version. The InstallPlan.remove already
      works similarly.
    • Duncan Coutts's avatar
      Simplify plan improvement, avoid reading store ghc-pkg db · bf0b5dfe
      Duncan Coutts authored
      Install plan improvement is the process of replacing configured source
      packages with installed instances from the store. Originally we did this
      by reading the ghc-pkg db to get the InstalledPackageInfo for all the
      packages in the store. We had to do that because when we replaced
      configured source packages by installed instances we used the
      PreExisting constructor which requires an InstalledPackageInfo, which we
      get from the installed package db. But now that we no longer use
      PreExisting for packages from the store we also no longer need the
      InstalledPackageInfo. All we need is a set of UnitIds. Also, once
      support for depending on executables was added then we needed a way to
      do plan improvement for executable packages/components. We did this by
      the simple approach of grabbing the dir listing for the store and using
      that as a set of UnitIds.
      So this patch extends the approach we use for executables and uses it
      for all packages. This means we no longer load the package db for the
      Note that still need to create the package db in the store. Previously
      we would create it when getting the package db contents, but we don't
      do that any more, but we still need to make sure the db exists.
      This also relates to the locking protocol in the store. The goal for the
      store is to be able to access and update it concurrently. The locking
      protocol will include checking membership by checking if the directory
      entry for the package is present. So this patch gets us to the right
      point for the reading side, leaving the writing side to do.
    • Duncan Coutts's avatar
      Distinguish PreExisting vs Installed in BuildStatus · 81691044
      Duncan Coutts authored
      Not a big deal, but should be useful later for more precise status
      reporting. For now just means the rebuild reasons can be more precise.
    • Duncan Coutts's avatar
      Remove now-unused InstallPlan.preexisting · f6feb199
      Duncan Coutts authored
      We only ever switch Configured to Installed. The PreExisting state only
      comes from the original solver plan, which only uses installed packages
      from the global db.
    • Duncan Coutts's avatar
      Start using new InstallPlan.Installed state · 0de4177c
      Duncan Coutts authored
      Change improvement and --dry-run phases to use Installed state rather
      than the PreExisting state. This means that PreExisting is now only used
      for installed packages from the global db, and never for installed
      packages from the store.
    • Duncan Coutts's avatar
      Update encodePlanAsJson for Installed package state · 9d6205e6
      Duncan Coutts authored
      The change in how we use the PreExisting vs Installed states means that
      we'll now have full details for all packages, rather than installed
      ones having only the subset of info available from the
      InstalledPackageInfo. So the 'type' field now can take the values
      "pre-existing", "configured" or "installed".
      Also do a little bit of tidying up.
    • Duncan Coutts's avatar
      Add an Installed state to InstallPlan packages · 435725ef
      Duncan Coutts authored
      This patch just adds the state without yet using it. That'll follow in
      subsequent patches.
      So why add an Installed state? Didn't we just remove the Installed,
      Processing and Failed states? Those states were used when we followed
      the approach of updating the InstallPlan as a build progressed (whereas
      we now do traversals without altering the InstallPlan).
      The idea of adding an Installed state now is that we can more usefully
      represent the state of the plan when we "improve" the plan with packages
      from the store or when we update the plan having checked if inplace
      packages are up to date. Currently in these two places we replace
      Configured source packages with PreExisting packages. There's a couple
      problems with this. Firstly the PreExisting state only contains an
      InstalledPackageInfo which means we loose information compared to all
      the detail in the Configured source package. This is relevant for things
      like plan.json output or other features that want to know the status of
      a project. Secondly we have to fake things for executables since they
      are not properly represented by InstalledPackageInfo.
  4. 23 Sep, 2016 3 commits
    • Duncan Coutts's avatar
      New Rebuild monad utils and use in ProjectPlanning · c7d55c3c
      Duncan Coutts authored
      Monitored variants of createDirectory and getDirectoryContents.
      Define thee wrappers in the RebuildMonad module so we have fewer
      open-coded tricky monitorFiles calls.
      In particular replace a glob monitor on the content of the store with a
      monitor on the store directory itself. This is valid based on the
      behaviour of directory mtimes, which is specified by posix and we have a
      sanity check for it in the unit tests.
    • Duncan Coutts's avatar
      Add a sanity unit test for dir mtime behaviour · 5a880c07
      Duncan Coutts authored
      We rely on posix & ntfs directory mtime semantics, which is that a dir
      mtime changes if the set of names in the dir changes.
      We rely on this both for the meaning of monitorDirectory but also we
      rely on it as an optimisation within the glob checking code.
      So this test should always pass on posix-compliant file systems.
      Apparently it may not hold on FAT file systems.
    • Duncan Coutts's avatar
      Fix the FileMonitor to keep multiple monitors for the same file · 1f9c6674
      Duncan Coutts authored
      Rather than doing any fancy merging of monitor kinds, we just do the
      simple thing and keep each monitor spec separately, so each one will be
      checked when the file system is probed. Internally, rather than keeping
      a Map indexed by FilePath we just keep a list.
      Add a regression test for this.
  5. 22 Sep, 2016 5 commits
  6. 21 Sep, 2016 6 commits
    • Mikhail Glushenkov's avatar
      Merge pull request #3879 from ezyang/pr/no-legacy-exe-solving · c18974b9
      Mikhail Glushenkov authored
      Don't solve for executables in legacy code path.
    • Mikhail Glushenkov's avatar
      Merge pull request #3874 from 23Skidoo/user-config-init-remote-repo · 99c7c221
      Mikhail Glushenkov authored
      'user-config init': Print the default 'remote-repo' correctly.
    • Edward Z. Yang's avatar
      Don't solve for executables in legacy code path. · 9e99b3f4
      Edward Z. Yang authored
      There is a bug in `cabal configure`'s invocation of the solver in
              (SourcePackageDb mempty packagePrefs)
              [SpecificSourcePackage localPkg]
      We can see that the solver is given an EMPTY source package database.
      This is because we assume that everything you need from cabal configure
      is taken from the installed package index.
      But this is NOT true for executables, which never have an entry in the
      installed package index. So we SHOULD NOT solve for
      executables in the legacy codepath, since there isn't anything useful we
      can do with the info anyway.  This gets toggled using a new solver
      parameter SolveExecutables.
      I didn't bother with a test because this will be obsoleted by
      Fixes #3875
      Signed-off-by: default avatarEdward Z. Yang <>
    • Edward Z. Yang's avatar
      Merge pull request #3873 from ezyang/pr/nix-local-build-prof · da71f1c8
      Edward Z. Yang authored
      Never use --enable-profiling when invoking Setup.
    • Herbert Valerio Riedel's avatar
      Merge pull request #3878 from hvr/pr/extended-01-index-cache · b1dd85f0
      Herbert Valerio Riedel authored
      - Extend `01-index.cache` with timestamps in preparation of `--index-state`
      - Use 'Binary' encoding for `01-index.cache`
    • Herbert Valerio Riedel's avatar
      Refactor & optimise construction of index cache · db1ef505
      Herbert Valerio Riedel authored
      This commit was motivated by @dcoutts' code-review comment:
      > Originally with using the `Sec.directoryEntries` that gave us only the
      > final version of each file, ie not all intermediate revisions. And
      > previously our strategy was to go through the final versions of each
      > file, in file order, and lookup just the ones we're interested in (which
      > in practice is 99% of them).
      > Now for the new cache we want to go through all revisions, which means
      > all entries in file order. So instead of using `Sec.directoryEntries`
      > which reads from the tar index, we go straight for `Sec.directoryFirst`
      > which is block 0 and iterate through, using `lazyUnfold`.
      > But we can now significantly simplify this and do it more
      > efficiently. Note that `indexLookupEntry` and `indexLookupFileEntry` are
      > expensive operations that seek in the tar file and read the tar entry at
      > that point. So lets do it exactly once per entry. The current code does
      > it once in the `lazyUnfold indexLookupEntry` and then again in `mk`. But
      > the old `mk` only did that because it had not previously looked up the
      > entry.
  7. 20 Sep, 2016 4 commits