GitLab

sep allows for horizontal separation, which is absolutely
not what we want.  Use hcat instead.  Thanks @stepcut
for reporting.
Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

Use 'ShortText' as internal representation of `PackageName`

This moves `String`/UTF8 conversion helpers to Distribution.Utils.String

Project orchestration preparatory work

We're going to need ProjectPlanOutput to import the ProjectBuilding
types, but this would cause a cycle with ProjectPlanning. So do the same
thing we did with ProjectPlanning and move the types out.

Instead rebuildTargetsDryRun will just return the BuildStatusMap and
runProjectPreBuildPhase in ProjectOrchestration will compose things by
calling improveInstallPlanWithUpToDatePackages.

This is just a slight shifting of functionality from here to there, but
better reflects responsibilities. This is also slightly with a future
status command in mind which likely only needs the BuildStatusMap.

Also adjust the tests after changing type of rebuildTargetsDryRun.

Make sure it consistently uses stderr, rather than a mixture of stdout
and stderr. Also rename it to dieOnBuildFailure to make it clear that
it is fatal in the case of build failures.

In general we report build failures in two steps: header plus build log
for each failing package, then a final die exception with summary.
Previously the build log step was reported to stdout whereas the die
exception message gets reported to stderr. So we switch that to use
new dieMsg and dieMsgNoWrap utils so that it all goes to stderr. Also,
like die, these are reported irrespective of the verbosity.

This is more or less just a workaround for the fact that we do not yet
have a nice structured/formatted error mechanism that would allow us to
throw just the one error in this case.

Stick it into the ProjectBuildContext. It's not used yet but it will be
useful, in particular for updating the .ghc.environment file. Arguably
it ought to just live in the DistDirLayout.

Previously the post-build phase was just reportBuildFailures and was
called directly from Cmd{Build,Repl}. So initially this just rearranges
things without changing behaviour, but it gives us a place to add other
post-build actions.

This also simplifies the code in Cmd{Configure,Build,Repl}.

A few examples are mentioned in a comment, including updating:
  .ghc.environment
  bin symlinks/wrappers
  haddock/hoogle/ctags indexes
  delete stale lib registrations
  delete stale package dirs

In particular it will not reflect the status of the build, ie it will
list all the source packages and their configuration but not say if
they're already installed in the store.

There's a slightly vauge principle at work here, that we should
distinguish between build outputs and build ststus: build outputs
should be (morally) pure functions of the environment, where as build
status is about the degree to which the current cached outputs reflect
the state of the inputs (ie are build outputs up to date or not).

So in this case that means the plan.json is considered an output and
thus it should not depend on the state of the store. We can certainly
provide machine readable status info, but this should be a separate
thing.

This was causing serious performance regression in the solver
so I'm commenting thise one out; see #3911 for details.

[skip ci]

This implements a type with a compact representation of `[Char]`.

The data is stored internally as UTF8 in an 'Data.ByteString.Short.ShortByteString'
when compiled against `bytestring >= 0.10.4`, and otherwise in a
plain old `[Char]`.

`ShortByteString` is available only from `bytestring` 0.10.4 on, and GHC
7.8.4 was the first GHC to bundle `binary-0.10.4`. So this fallback
affects mostly only GHC 7.6 and older.

Note: Originally a strict `ByteString` was used as fallback for this patch. However, the 
`[Char]` fallback avoids pinned memory and may be more preferable when dealing with
many small `ShortText`s

Similiar to dabd9d98 which made
`PackageName` opaque, this makes `Distribution.Version.Version` opaque.

The most common version numbers occuring on Hackage are 3- and
4-part versions. This results in significant Heap overhead due to
`Data.Version`'s inefficient internal representation.

So like the `PackageName` commit, this commit is a preparatory commit to
pave the way for replacing `Version`'s internal representation by a
representation with a memory footprint which can be an order of
magnitude smaller.

Finally, this also adds a new functor-like convenience function

    alterVersion :: ([Int] -> [Int]) -> Version -> Version

for modifying the version number components.

Make interactive setup delegate CTRL+C

Fixes #3899

When cabal new-repl spawns ghci, it spawns this as a subprocess.
In UNIX like systems, if a CTRL+C is sent to this child process
it also bubbles up to the parent process, causing it to terminate.
Also see https://hackage.haskell.org/package/process-1.4.2.0/docs/System-Process.html#g:4.

However, this is not what we want for interactive subprocesses.
Interactive processes usually define their own handlers for CTRL+C,
for example GHCi uses CTRL+C to reset the input buffer. So instead
of terminating on CTRL+C we want to delegate CTRL+C to GHCi
and let it do its thing.

Luckily, we can enable CTRL+C delegation such that the parent
process ignores the CTRL+C and instead delegates it to the
child process.

When looking at heap-profiles of `cabal-install`, the `(:)` constructor
stands out as the most-allocated constructor on the heap.

Having to handle 10k+ package names contributes to the allocation
numbers, especially on 64bit archs where ASCII `String`s have a 24 byte
per character footprint.

This commit is a preparatory commit to pave the way for changing
`PackageName`'s internal representation to something like
`ShortByteString` (which is essentially a thin wrapper around primitive
`ByteArray#`s which themselves have have an overhead of 2 words + one
byte per ASCII character rounded up to nearest word) which would allow
to reduce the memory footprint by a full order of magnitude, as well as
reduce pointer chasing and GC overhead.

InstallPlan fixes and misc housekeeping

This unexposes `Distribution.Compat.Prelude` again, and adds
an exposed `.Internal` module which reexports the `.Prelude` module with
an attached module-level `WARNING` pragma.

This way users are discouraged to use this in `Setup.hs` files as they'd
see compile warning like:

    Foo.hs:19:1-55: warning: [-Wdeprecations]
      Module ‘Distribution.Compat.Prelude.Internal’:
        This modules' API is not stable. Use at your own risk, or better yet, use @base-compat@!

In `cabal-install` we import the `.Internal` module exactly once, and
use -Wno-deprecations to suppress that the warning of that single
import.

This is supposed to become more or less a superset of Cabal's
`Distribution.Compat.Prelude`.

As a side-effect,t his exposes `Distribution.Compat.Prelude` from the
Cabal library (which may be actually a good thing, as it may be useful
module to Setup.hs writers).

At least keysSet will be useful shortly. Others are just for good
measure (though there's probably some cases where we could avoid
converting to and from lists if we used toMap or toGraph).

This better matches the feeling of the thing now, and also better
matches the rest of the naming.

Same idea as a patch proposed by ezyang but with somewhat less
infrastructure needed. We just use individual asserts for each part of
the invariant, so the assert source location tells us which part was
violated. Hopefully call stacks can also tell us in which function the
invariant was checked.

What they mean informally, to augment the formal definitions.

Add two additional properties that we believe are true:

 - All immediate reverse deps of packges that are currently processing
   are not currently being processed (ie not in the processing set).
 - The failed set is upwards closed, i.e. equal to its rev dep closure

Remove the wrong assertion:
  all (`Set.notMember` failedSet)  (tail newlyFailedIds)

This is wrong for much the same reasons as the part of the invariant
(in the previous patch) was wrong.

Suppose Q depends on P1 and P2 and the processing set intially is
{P1, P2}. Now suppose that P1 fails, so the failure set becomes {P1,Q}
and the new processing set is {P2}. Now suppose P2 also fails. Then the
newlyFailedIds is [P2, Q] and so the tail is [Q]. Of couse Q is in the
failure set already, in violation of the above assertion. But this is a
perfectly reasonable scenario. It's just the assertion that is wrong.

FileMonitor fix

Previously it required:
  noIntersection processingClosure failedSet

Lets see what it's saying and why it's wrong. Suppose we have P1 and P2
in the processing set. We have Q that depends on P1 and P2. So the
processingClosure is {P1,P2,Q}. Now suppose building P1 fails. The
failedSet is now {P1,Q1}, and the processingSet is {P2}. The
processingClosure however is {P2,Q}. Thus we have a non-empty
intersection between the processingClosure and failedSet, namely {Q}.

So clearly it's bogus, but also it's not clear if I was thinking of
something else that is correct. So we just drop this part of the
invariant.

This should fix issue #3889.

Add InstallPlan.Installed state

Explain why it's not represented by a Set.

Without this optimisation, `cabal info somethingnonexisting` results in

     960,397,120 bytes allocated in the heap
     739,652,560 bytes copied during GC
      67,757,128 bytes maximum residency (24 sample(s))
       2,234,096 bytes maximum slop
             147 MB total memory in use (0 MB lost due to fragmentation)

with this optimisation:

   1,000,825,744 bytes allocated in the heap
     656,112,432 bytes copied during GC
      44,476,616 bytes maximum residency (24 sample(s))
       2,302,864 bytes maximum slop
             109 MB total memory in use (0 MB lost due to fragmentation)

So the total memory in use is significantly lower. The total runtime is
also slightly reduced, from

  INIT    time    0.001s  (  0.001s elapsed)
  MUT     time    0.683s  (  1.050s elapsed)
  GC      time    0.946s  (  0.946s elapsed)
  EXIT    time    0.005s  (  0.005s elapsed)
  Total   time    1.637s  (  2.002s elapsed)

to

  INIT    time    0.001s  (  0.001s elapsed)
  MUT     time    0.664s  (  0.988s elapsed)
  GC      time    0.797s  (  0.797s elapsed)
  EXIT    time    0.004s  (  0.004s elapsed)
  Total   time    1.467s  (  1.789s elapsed)


Note that there's currently ~80k cache entries, but only ~10k unique package names
and ~6k unique versions. So hash-consing helps reduce the amount of heap objects 
for both value types by one order of magnitude, which among other benefits also
reduces GC overhead.

Enable `-rtsopts` for `cabal` executable

This PR implements a new flag `--index-state` (and its `cabal.project`/config-file
equivalent `index-state: ...`) which allows to change the source package index
state the solver uses to compute install-plans. This is particularly useful in
combination with freeze-files in order to also freeze the state the package
index was in at the time the install-plan was frozen.

This provides the core functionality, on which future enhancements can build
upon. See also description of PR #3604 for some possible enhancements.

[skip ci]

Fwiw, the GHC executables are compiled with full `-rtsopts` as well.
The benefit is that one can use flags such as `-M1G` to limit the heap
space, or profile memory usage via `-hT`.