GitLab

The PlanningFailed case does not happen. The ReplFailed, HaddocksFailed
cases were previously covered by BuildFailed but they're worth
disinguishing.

In both cases it's optional and only added when the log file is used.
This will be used later in reporting the build outcomes.

We're going to alter and extend the BuildResult and BuildFailure types
for the new-build code, so we cannot share those types with the old
install code.

This patch doesn't change the structure, just redefines them locally and
switches uses to record style so we can add new fields easily.

I think these names are now a bit more induitive / less confusing and a
bit more consistent.

We now have BuildOutcome(s) to mean either failure or a build result,
and BuildResult to mean the result of a non-failing build.

[ci skip]

Download errors are now put into the residual install plan, like other
build errors.

Fixes issue #3387

Split things up a little so the generic async fetch can live with the
other fetch utils. This also makes it easier to test.

Change the exception handling so that any exception in fetching is
propagated when collecting the fetch result.

As a result of the previous InstallPlan refactoring, we can now use the
non-serialisable BuildFailure type from D.C.Types which uses
SomeException, where previously we had to use a copy of that type that
used String for the errors.

So now there's no longer any need to have a separate set of types for
BuildResult, BuildResults, BuildSuccess or BuildFailure. There was a
minor difference in the structure of the BuildSuccess, where in the new
build code we need to be able to produce the InstalledPackageInfo at a
different point from the rest of the info in the BuildSuccess. This can
be kept local to the ProjecBuilding module, but accounts for the
somewhat larger number of changes in that module.

These were used previously for the Installed and Failed package states,
but these states are now gone.

Importantly this now means that we can have a serialisable InstallPlan
without the failure types having to be serialisable. This means we can
use things like SomeException which is not serialisable. Since the
traversal is done separately, the result of the traversal contains the
failure values, but this result set does not have to be serialised.

We now just have the initial states, PreExisting and Configured. The
Processing, Installed and Failed states are gone. We now do traversals
separately from the InstallPlan, so we no longer need these states.

Eliminate the local executeInstallPlan. The main change is that the new
execute returns BuildResults instead of an upated InstallPlan. This has
a few knock-on conseuqnces for the code that looks at the result of the
build execution.

Currently we don't actually do that much with the results in the
new-build code path (though we should) so there's less disruption than
one might imagine. The biggest change is in the integration tests which
do inspect the execution result to check things worked or didn't work as
expected.

The equivalent change for the old build code path will be more
disruptive since it does a lot of stuff with the execution results.

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

Instead initialise all the ones we'll need in advance of building any
packages. The alternative would have been to use a lock, but there's no
particular advantage in trying to delay initialisation and avoiding
locks keeps things simpler.

This should fix #3460

Another similar issue was fixed by 1acc00f8 which serialised the
registration of packages in the local inplace package db, avoiding
lost updates (which could previously be observed in test case T3460).

Allowing parallel installation of files, just serialising the actual
registration. Also previously only store package installs were
serialised but we also need to serialise registration of packages in the
local inplace package db.

The parallel JobControl now has a built-in maximum job limit, so the
separate build JobLimit is unnecessary. The separate fetch JobLimit
is still useful.

For both install and new-build.

Add a keepGoing :: Bool arg to control whether to keep going after a
build failure or whether to try and stop as soon as possible.

This makes use of the new JobControl interface to try cancelling
outstanding jobs and also for checking if there are jobs in progress.

For now the call sites are hard coded to use keepGoing = False.

The motivation here is to support an install mode where we stop as soon
as possible after one package fails, rather than going on as far as
possible with other packages.

To properly support this we need to be able to cancel jobs that have
been submitted but that have not yet been started. This is easy for the
serial JobControl but requires re-implementing the paralle JobControl.

In the previous impl we simply fork a thread immediately for each job
(and use a JobLimit externally to limit parallelim). The problem with
that is that all submitted jobs are actually executing, so there's no
way to cancel jobs that have not started yet (there are none). Combined
with the executeInstallPlan behaviour of exposing the maximum possible
paralleism this means that there can be a lot of jobs in progress
(though most waiting on the JobLimit), and none of them can be sanely
cancelled.

So in this impl we have a built-in maximum parallelism, so we're
executing at most that many jobs at once, while all others are waiting
to be executed. The ones that are waiting can be cancelled.

Because it's unspecified how many jobs can be cancelled, users can no
longer rely on counting the number of outstanding jobs so we add a new
method to check if there are any jobs remaining.

Subsequent patches will add tests and make use of these new features to
stop the install process as soon as possible after a failure, with a
new --keep-going flag to get the existing behaviour.

Check that we do get the registration info we expect in a couple places,
and add detail to the error message originally reported in #3428.

Also build the integration tests with assertions on, which might have
caught this error earlier (via the invariant for the install plan).

When building packages we update the install plan with the completed
packages and for libraries we include the InstalledPackageInfo. We
now support packages that register multiple libraries but only one of
them is the representative public library, and only that one is stashed
in the install plan. Out of the list of library registraions we select
the primary one by looking for the one with the expected unit it. As
part of collecting the registration info we do some processing (to
account for older Cabal and ghc versions) and the mistake was that
while did that post-processing ok and registered the right libraries
we ended up returing the un-processed registraion info and so then
failed to find the primary lib, and thus failed to stash any library
info in the install plan, causing internal errors later on.

So this patch fixes it to return the same post-processed registration
info as actually gets registered. Subsequent patches improve the
internal error checking for this issue.

When executing the install plan, the various failures are supposed to be
caught and put into the residual install plan. This adds the main cases
for that, including the configure and build cases. Download is still not
covered.

This is a step towards better reporting of what failed when there are
failures in deps etc.

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

As a refactoring, this moves allPackageSourceFiles from
Distribution.Client.Sandbox.Timestamp to
Distribution.Client.SrcDist, makes it thread safe, and has
it return files relative to the source directory.
Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

The ProgramDb in the ElaboratedSharedConfig actually only ever contains
the programs configured by the compiler, and this is the way it will
remain, so rename to clarify that usage.

Config for other non-compiler programs will be handled differently. This
makes sense because the compiler (and programs) have to be shared
between all packages, while for non-compiler programs it's possible in
principle to use different versions for different packages within a
project. Thus it's not possible to have a single shared ProgramDb for
all programs for all packages (other than for the compiler).

Previously, ReadyPackage was a ConfiguredPackage elaborated with
a dependencies data structure which had InstalledPackageInfo
rather than ConfiguredId. Well, it turned out that we only
used the data from ConfiguredId! So that extra info is useless.

Instead, ReadyPackage is now purely a newtype wrapper for type
safety; a reminder that not all ConfiguredPackages can be built,
only the ready ones.
Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

Duncan Coutts authored Mar 25, 2016 and Mikhail Glushenkov committed Mar 27, 2016

File globs can now be absolute, e.g. starting with / or c:\
Also allow homedir relative, ie ~/

Globs can also have a trailing slash, in which case they only match
directories, not files.

Previously whether globs could match dirs was not totally consistent.
The matchFileGlob would match dirs, but the file monitor globs would
not. The file monitor globs can now match dirs (or with a trailing
slash, only match dirs). File monitors now also detect changes in the
file type, ie switching from file to dir or the other way around.

The file monitor are now pretty consistent between single file monitors
and globs monitors. They now have equivalent capabilities and share
code. For a single file or for a glob we can now control what we
monitor if the path is a file or a dir. In both cases we can monitor
mere existence, non-existence or modification time. For files we can
additionally monitor content hash.

File monitors now also detect changes in the file type, ie switching
from file to dir or the other way around.

New tests cover all these new file monitor cases. There are also new
tests for glob syntax, covering printing/parsing round trips.

(cherry picked from commit f6c1e71c)

Duncan Coutts authored Mar 14, 2016 and Mikhail Glushenkov committed Mar 27, 2016

This stage takes the ElaboratedInstallPlan and executes it. It does it
in two passes.

The first pass is the "dry run" pass where we work out which packages
and components within packages we actually need to build. If we are in
fact in --dry-run mode then of course this is the only pass we run, we
have enough info after this to accurately report on what we would do.

The first pass does the file monitor change stuff, which checks if
package config has changed and probes for package file changes. Based on
this we make a note of which phase of the build we would start at, up to
and including skipping building that package entirely. In the latter
case we replace the Configured package by a corresponding Installed
package.

The first pass also prunes the ElaboratedInstallPlan based on the
targets that the user actually wants to build. For example we solve and
plan on the basis that the user might want to run the test suites or
benchmarks, but that doesn't mean we want to build them all the time
(they'll usually pull in additional deps). So there's a moderately
complex pass where we keep just the packages and components that are
needed to build the target components.

The first pass is moderately complicated but it makes no state changes
and just returns the updated ElaboratedInstallPlan, so it should make
things easier to debug.

The second phase has very little logic it just does what its told based
on all the earlier passes. For each package the dry run pass has worked
out which build step to start with, e.g. downloading, configuring or
building.

The major cases the build phase has to deal with are packages that will
be built and installed to the store, vs packages that are local and so
are built inplace. It is only the latter that need all the complex
rebuild checking. In both cases all the "setup $cmd" flags are
calculated by the planning phase and we just do as we're told.

(cherry picked from commit 300d9bad)