This commit was motivated by @dcoutts' code-review comment:

> Originally with using the `Sec.directoryEntries` that gave us only the
> final version of each file, ie not all intermediate revisions. And
> previously our strategy was to go through the final versions of each
> file, in file order, and lookup just the ones we're interested in (which
> in practice is 99% of them).
>
> Now for the new cache we want to go through all revisions, which means
> all entries in file order. So instead of using `Sec.directoryEntries`
> which reads from the tar index, we go straight for `Sec.directoryFirst`
> which is block 0 and iterate through, using `lazyUnfold`.
>
> But we can now significantly simplify this and do it more
> efficiently. Note that `indexLookupEntry` and `indexLookupFileEntry` are
> expensive operations that seek in the tar file and read the tar entry at
> that point. So lets do it exactly once per entry. The current code does
> it once in the `lazyUnfold indexLookupEntry` and then again in `mk`. But
> the old `mk` only did that because it had not previously looked up the
> entry.

With this commit, if a corrupted index cache is detected the
`readIndexCache` function now regenerates the index cache and then
reattempt to read the index once (and 'die's if it fails again).

This commit extends the index cache entries relevant for 01-index to
include block numbers and timestamps, and makes them strict so recent
GHCs unpack the fields:

    data IndexCacheEntry
        = CachePackageId PackageId BlockNo
        | CachePreference Dependency
        | CacheBuildTreeRef BuildTreeRefType BlockNo

to

   data IndexCacheEntry
       = CachePackageId PackageId !BlockNo !Timestamp
       | CachePreference Dependency !BlockNo !Timestamp
       | CacheBuildTreeRef !BuildTreeRefType !BlockNo

For the legacy `00-index.tar`s, the 'Timestamp' field is set to (-1),
and the original 00-index.cache format is retained.

For (secure) `01-index.tar`s, all of `IndexCacheEntry`s data is stored
in the `01-index.cache` file.

Moreover, to avoid having to write out and parse new two integers per
cache entry, this patch switches to using `Binary` instances for
encoding the `01-index.cache` file (while `00-index.cache` remains
plain-text).

Add helper for Generic NFData instances

This reverts commit 7b67f37b, reversing
changes made to c12290f8.

Installplan installed state

This is needed in order to support `deepseq < 1.4` which didn't
have a `Generic`-based default `rnf` implementation yet.

In order to define instances, use e.g.

> instance NFData MyType where rnf = genericRnf

The implementation has been taken from `deepseq-1.4.2`'s default
`rnf` implementation.

Print that we are building all due to Custom setup.

Rename configProf/configCoverage local variables to not shadow.

Clarify the differences with the GetOpt from base

All the use sites (currently only two but soon to be three) use
InstallPlan.installed to do a bulk change of states, differing only in
the filter condition. So it simplifies things and shares more code if
we make the main one be the bulk version. The InstallPlan.remove already
works similarly.

Install plan improvement is the process of replacing configured source
packages with installed instances from the store. Originally we did this
by reading the ghc-pkg db to get the InstalledPackageInfo for all the
packages in the store. We had to do that because when we replaced
configured source packages by installed instances we used the
PreExisting constructor which requires an InstalledPackageInfo, which we
get from the installed package db. But now that we no longer use
PreExisting for packages from the store we also no longer need the
InstalledPackageInfo. All we need is a set of UnitIds. Also, once
support for depending on executables was added then we needed a way to
do plan improvement for executable packages/components. We did this by
the simple approach of grabbing the dir listing for the store and using
that as a set of UnitIds.

So this patch extends the approach we use for executables and uses it
for all packages. This means we no longer load the package db for the
store. Note that we still have to create the package db in the store.

This also relates to the locking protocol in the store. The goal for the
store is to be able to access and update it concurrently. The locking
protocol will include checking membership by checking if the directory
entry for the package is present. So this patch gets us to the right
point for the reading side, leaving the writing side to do.

And invite anyone to have a go at merging. Also re-export the types from
the GetOpt from base rather than re-defining locally. In particular this
means we gain a few instances.

The changes wrt base GetOpt are:

One change to allow us to treat "cabal --flag command" as
"cabal command --flag" e.g. "cabal -v configure" to mean
"cabal configure -v" for flags that are not recognised as global flags,
pass them on to the sub-command. See the difference in 'shortOpt'.

Line wrapping in the 'usageInfo' output, plus a more compact rendering
of short options, and slightly less padding.

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

Print profiling status when displaying install plan.

Use enabledBuildInfos rather than allBuildInfo

Previously the output was:

    Building profunctors-5.2 lib...
    Building semigroupoids-5.1...

Now it is:

    Building profunctors-5.2 (lib)...
    Building semigroupoids-5.1 (all, due to Custom setup)...

Fixes #3808.

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

CC @hvr



Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

Add a new field to solver Done nodes, and simplify Explore.exploreLog.

Not a big deal, but should be useful later for more precise status
reporting. For now just means the rebuild reasons can be more precise.

We only ever switch Configured to Installed. The PreExisting state only
comes from the original solver plan, which only uses installed packages
from the global db.

Change improvement and --dry-run phases to use Installed state rather
than the PreExisting state. This means that PreExisting is now only used
for installed packages from the global db, and never for installed
packages from the store.

The change in how we use the PreExisting vs Installed states means that
we'll now have full details for all packages, rather than installed
ones having only the subset of info available from the
InstalledPackageInfo. So the 'type' field now can take the values
"pre-existing", "configured" or "installed".

Also do a little bit of tidying up.

This patch just adds the state without yet using it. That'll follow in
subsequent patches.

So why add an Installed state? Didn't we just remove the Installed,
Processing and Failed states? Those states were used when we followed
the approach of updating the InstallPlan as a build progressed (whereas
we now do traversals without altering the InstallPlan).

The idea of adding an Installed state now is that we can more usefully
represent the state of the plan when we "improve" the plan with packages
from the store or when we update the plan having checked if inplace
packages are up to date. Currently in these two places we replace
Configured source packages with PreExisting packages. There's a couple
problems with this. Firstly the PreExisting state only contains an
InstalledPackageInfo which means we loose information compared to all
the detail in the Configured source package. This is relevant for things
like plan.json output or other features that want to know the status of
a project. Secondly we have to fake things for executables since they
are not properly represented by InstalledPackageInfo.

Give more informative error when ghc-pkg dump fails.

Track cabal.project provenance for error reporting

Renumber flags version numbers in 'filterConfigureFlags'.

"Secure" cabal repositories use a new extended & incremental
`01-index.tar`. In order to avoid issues resulting from clobbering
new/old-style index data, we save them locally to different names.

With this patch, secure repos generate/update the files below on `cabal update`

- `01-index.cache`
- `01-index.tar`
- `01-index.tar.gz`
- `01-index.tar.idx`
- `mirrors.json`
- `root.json`
- `snapshot.json`
- `timestamp.json`

...while the legacy codepaths for non-secure repos operate on the files

- `00-index.cache`
- `00-index.tar`
- `00-index.tar.gz`
- `00-index.tar.gz.etag`

This way the old/new codepaths don't interfere with each other anymore.

Note: The format of `01-index.cache` file will be extended by the upcoming `--index-state` feature

This trivially fixes #3854

doc: "Local versus external packages" intro paragraph bad contrast

[skip ci]

I believe it's "per-project" vs "cached across *projects*" not "...across *packages*"

This was introduced accidentally via
b083151f

/cc @ezyang

- Excludes provenance from all roundtrip tests
- Inserting explicit provenance when read from file
- Special cases for bad package errors arising from
  implicit project configuration

Previously, the code was inconsistent on whether or not
flags_x_y_z indicated that these flags could be used
up to version x.y.z, or should be used prior to x.y.z.
This commit picks the LATTER and renames everything
consistently this way.  The bonus is that now the names
match up with the conditionals. Yay.

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

In many places, we incorrectly used allBuildInfo, which
returns all BuildInfos that are buildable, and not
necessarily the ones we are actually going to *build*.
This used to "mostly do the right thing" because we
literally edited the PackageDescription to nub out things,
so you wouldn't see non-enabled components anyway.  However
when I added support for per-component configure, I stopped
editing the PackageDescription, which meant all of these
uses were wrong.

So, I updated them to do the right thing. Note that there
are still uses of allBuildInfo in Check, but that probably
requires a closer look.

Fixes #3847.

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>