Add support for HEAD builds

@mpickering this is a bit tricky to get right with ghcup's current static ghc version <-> artifact url mapping design. I.e we'd need some way to infer the version number to distinguish which release of GHC HEAD we have, as well as the ability to request a specific version of GHC HEAD (which isn't even a well-defined one, as e.g. the ones which are generated by my PPA might have the same numeric version but will most likely not match the commits the gitlab CI builder picked for obvious reasons).

If it's always the same tarball name, then we could skip the use of our url mapping file entirely. It won't be very pretty code-wise, but then it would be doable.

@hasufell but how do we detect whether we already have the requested version before starting a multi-MiB download, if the single url we download from keeps mutating? I'd like a reasonably robust scheme and so far this looks like a very unclean hack.

but how do we detect whether we already have the requested version

we don't

@hasufell that sounds terrible :-(

HEAD is rolling, why would you try to be smart about it? That has little benefit.

There is a nightly job which gets run from master which could provide a cleaner version to artefact mapping. It's a bit unclear to me how to access this information from the API though and it looks you need to be authenticated to.

Perhaps some combination of

https://gitlab.haskell.org/api/v4/projects/ghc%2Fghc/pipelines/?scope=finished&ref=master

and

https://gitlab.haskell.org/api/v4/projects/1/jobs/1437

@hvr If you type a command which means "download the latest unstable version of the compiler" you should probably expect it to download a large file.

However, one way would be to inspect the headers of the URL before commencing the download and seeing if you have already downloaded the artefact.

curl -I -L https://gitlab.haskell.org/ghc/ghc/-/jobs/artifacts/master/download?job=validate-x86_64-linux-deb8
HTTP/2 302 
server: nginx
date: Wed, 23 Jan 2019 13:37:14 GMT
content-type: text/html; charset=utf-8
location: https://gitlab.haskell.org/ghc/ghc/-/jobs/15451/artifacts/download
cache-control: no-cache
x-content-type-options: nosniff
x-frame-options: DENY
x-request-id: 5QH9EixLSs4
x-runtime: 0.020671
x-ua-compatible: IE=edge
x-xss-protection: 1; mode=block

HTTP/2 302 
server: nginx
date: Wed, 23 Jan 2019 13:37:14 GMT
content-type: text/html; charset=utf-8
location: https://ghc-gitlab.s3.amazonaws.com/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b/2019_01_22/15451/21447/artifacts.zip?response-content-disposition=attachment%3Bfilename%3D%22artifacts.zip%22&response-content-type=application/zip&X-Amz-Expires=600&X-Amz-Date=20190123T133714Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAI3NPPHETCY4XXTOA/20190123/us-east-1/s3/aws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=5f45ef2dfe06906fe77e026afc0e38982438162515a93ac016153564d0704e12
cache-control: no-cache
x-content-type-options: nosniff
x-frame-options: DENY
x-request-id: ytFzhdKUry3
x-runtime: 0.017884
x-ua-compatible: IE=edge
x-xss-protection: 1; mode=block

HTTP/1.1 403 Forbidden
x-amz-request-id: 57ACD6F8E66CC417
x-amz-id-2: h6M/Mge7y45J6ppB6XkMSrqUe+aXyjn+6B+zq/xPC+x2gME7ZpSN/6AGT+kJ38qAIowy359hVO0=
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Wed, 23 Jan 2019 13:37:14 GMT
Server: AmazonS3

the purpose of installing GHC HEAD is for testing; and there you need the ability to select a specific ghc version for figuring out where a regression was introduced or similar (and/or for reproducing bug reports against a specific ghc snapshot); the huge benefit of the gitlab-ci generated artifacts is that the history of binary artifacts is preserved -- one thing I cannot do easily w/ launchpad's apt repos which garbage collect very keenly and basically only support one version per package; also for CI usage you'd typically have local caches and run "install head" frequently, and then you'd end up wasting tons of network bandwidth as well as disk I/O if we don't detect that we already have installed whatever the symbolic "HEAD" indirection points to...

the purpose of installing GHC HEAD is for testing; and there you need the ability to select a specific ghc version for figuring out where a regression was introduced or similar

Not sure that is the primary use case. This sounds like GHC development and I would rather expect people to have the sources checked out and be able to compile and git bisect at will.

have the sources checked out and be able to compile and git bisect at will.

that's an extremely slow process; that's why we've been waiting to have some more automated means of bindist generation of GHC HEAD, so that we could speed up the bisecting process significantly by two or more orders of magnitude

I don't really see why the two are exclusive. You can ask for an artefact specifically or not. They are different use cases and both are possible to implement.

that's an extremely slow process; that's why we've been waiting to have some more automated means of bindist generation of GHC HEAD, so that we could speed up the bisecting process significantly by two or more orders of magnitude

I think that's a different and rather complicated use case to support properly. I was only thinking of a simple use case, which is "just install whatever HEAD points to", which is also basically what autogenerated github archives do.

I think someone has to come up with a PR. I don't have a clear picture of this feature or its requirements.

Add support for HEAD builds

Child items 0

Activity