Skip to content

Changes

Page history
Add some troubleshooting tips for submodule errors authored by Torsten Schmits's avatar Torsten Schmits
Hide whitespace changes
Inline Side-by-side
working-conventions/git/submodules.md
View page @ 7d07d100
......@@ -8,17 +8,15 @@ General information about Git's submodule support:
- [Pro Git "6.6 Git Tools - Submodules" chapter](http://git-scm.com/book/en/Git-Tools-Submodules)
- [Submodule Tutorial](http://www.vogella.com/tutorials/Git/article.html#submodules)
## Cloning a fresh GHC source tree
Initial cloning of GHC HEAD (into the folder `./ghc`) is a simple as:
```
```plaintext
git clone --recursive https://gitlab.haskell.org/ghc/ghc
```
(Obviously, the clone URL can be replaced by any of the supported `ghc.git` URLs as listed on [http://git.haskell.org/ghc.git](http://git.haskell.org/ghc.git))
See [getting the sources](building/getting-the-sources) for more ways of getting the sources (e.g., from the GitHub mirror), and specifically this [note on using forks](building/getting-the-sources#using-a-fork-of-ghc).
(Obviously, the clone URL can be replaced by any of the supported `ghc.git` URLs as listed on http://git.haskell.org/ghc.git) See [getting the sources](building/getting-the-sources) for more ways of getting the sources (e.g., from the GitHub mirror), and specifically this [note on using forks](building/getting-the-sources#using-a-fork-of-ghc).
## Updating an existing GHC source tree clone
......@@ -26,7 +24,7 @@ Sometimes when you pull in new commits, the authors updated a submodule. After p
At the top-level of `ghc.git` working copy:
```
```plaintext
git pull --rebase
git submodule update --init
```
......@@ -50,72 +48,80 @@ git submodule update --init
Here we will describe how to modify make changes in the submodules of the GHC repository. In the discussion that follows we will assume that the user is working on a GHC branch named `wip/$BRANCH_NAME`. It's very important to keep in mind that Git submodules track commits, not branches. Therefore, `git submodule update` will result in submodules having checked out a so-called [detached HEAD](http://alblue.bandlem.com/2011/08/git-tip-of-week-detached-heads.html).
To make a change to a submodule (here we use `Cabal` for concreteness), start by ensuring that your tree's submodules are up-to-date:
```bash
git submodule update --init
```
Next, create a branch for your change, ensuring that the name contains the `wip/` prefix (it is common to name the submodule branches similarly to the GHC branch with which they are associated),
Next, create a branch for your change, ensuring that the name contains the `wip/` prefix (it is common to name the submodule branches similarly to the GHC branch with which they are associated),
```bash
git -C libraries/Cabal checkout -b wip/$BRANCH_NAME
```
We can now proceed with making the desired changes in the submodule and commit them as usual:
```
```plaintext
# perform modifications and as many `git {add,rm}`s as you deem necessary
$EDITOR libraries/Cabal/src/somefile.hs
# commit
git -C libraries/Cabal commit
```
Finally, we commit the submodule change in the `ghc` repository.
```bash
git add libraries/Cabal
git commit
```
In the commit message be sure to mention any submodule changes made; a [linter](https://gitlab.haskell.org/ghc/git-haskell-org-hooks/-/blob/master/src/validate-submod-refs.hs) in GHC's CI process checks that any commits containing submodule changes mention the word "submodule" to prevent unintentional submodule changes from accidentally being merged.
To push the changes, we first push the submodule changes to the GHC mirror repository (e.g. ghc/packages/Cabal>). GHC developers have permission to push branches to these mirrors under the `wip/` branch namespace (if you encounter a permission denied error, ask someone in `#ghc` to grant you Developer rights in the `ghc` group).
```bash
git -C libraries/Cabal push git@gitlab.haskell.org:ghc/packages/Cabal wip/$BRANCH_NAME
```
Now create a draft merge request against the upstream projeect (e.g. <https://github.com/haskell/Cabal>) proposing to merge your branch into the branch being tracked by GHC (typically `master`).
Now create a draft merge request against the upstream projeect (e.g. https://github.com/haskell/Cabal) proposing to merge your branch into the branch being tracked by GHC (typically `master`).
Finally, we can push the GHC branch as well:
```bash
git push origin wip/$BRANCH_NAME
```
We can now open a GHC merge request [as usual](https://gitlab.haskell.org/ghc/ghc/-/wikis/Contributing-a-Patch#merge-request-workflow). When doing so, include a list of the associated upstream merge requests for any submodule changes made in the MR.
### Merging an MR containing submodule changes
When an MR containing submodule changes has passed review, it is important that any upstream submodule MRs have been merged *before* the MR is added to the merge queue. While the submodule linting job will catch any MRs which refer to submodule commits not reachable via a persistent branch (e.g. not a `wip/` branch), this will cause @marge-bot jobs to fail, stalling the merge process.
When an MR containing submodule changes has passed review, it is important that any upstream submodule MRs have been merged _before_ the MR is added to the merge queue. While the submodule linting job will catch any MRs which refer to submodule commits not reachable via a persistent branch (e.g. not a `wip/` branch), this will cause @marge-bot jobs to fail, stalling the merge process.
## Upstream repositories
Check out the [Repositories](repositories) page for a full breakdown of all the repositories GHC uses.
## TODO
- Sort out what to do about `transformers`
- Describe status of `pretty` which is one-off at the moment and doesn't exactly track upstream.
## Mirror configuration
GHC maintains mirrors of its core libraries in the GHC/Packages> namespace. There are a few reasons for these mirrors:
GHC maintains mirrors of its core libraries in the GHC/Packages\> namespace. There are a few reasons for these mirrors:
* to provide GHC contributors with a place other than upstream to push their changes for testing in CI
* to provide a level of safety against forced pushes by upstream
* to provide a single trusted source for GHC and its dependencies
* to provide GHC contributors with a place other than upstream to push their changes for testing in CI
* to provide a level of safety against forced pushes by upstream
* to provide a single trusted source for GHC and its dependencies
To accomplish this we configure the submodule projects as follows:
* The submodule project is configured with pull mirroring from the upstream repository with the "Only mirror protected branches" option enabled
* All of the "interesting" branches we want to mirror from upstream are added as "Protected branches" (e.g. `master`, and `1.24`)
* A push rule is added to restrict pushes to the interesting branches and `wip/.*`
* The submodule project is configured with pull mirroring from the upstream repository with the "Only mirror protected branches" option enabled
* All of the "interesting" branches we want to mirror from upstream are added as "Protected branches" (e.g. `master`, and `1.24`)
* A push rule is added to restrict pushes to the interesting branches and `wip/.*`
This ensures that commits can only be pushed to `wip/.*`, which the submodule check linter does not consider as roots.
## Submodule-specific policies
### Haddock
......@@ -132,4 +138,27 @@ To land a GHC MR that requires changes to Haddock, you will need to coordinate y
* Get your MR in a mergeable state: approved and passing CI. The `lint-submods` test should give a warning at this point, but it will not cause the CI to fail.
* Open an MR against the `ghc-head` branch of Haddock **on GitLab**, and get it merged.
* If necessary, update the GHC MR to point to the correct ghc-head haddock commit. If the commit hash didn't change, then you can skip this step.
* At this point, the GHC MR should be passing CI, **including** the `lint-submods` test. It can then be assigned to `marge-bot` and merged.
\ No newline at end of file
* At this point, the GHC MR should be passing CI, **including** the `lint-submods` test. It can then be assigned to `marge-bot` and merged.HH
# Troubleshooting
Git sometimes produces a rather cryptic error messages when pushing the GHC repo with unpushed submodule changes:
```plaintext
fatal: remote 'myremote' not configured
fatal: process for submodule 'utils/haddock' failed
fatal: the remote end hung up unexpectedly
remote:
remote: ========================================================================
remote:
remote: rpc error: code = Canceled desc = user canceled the push
remote:
remote: ========================================================================
remote:
```
This means that the commit of the Haddock submodule that's recorded in the latest commit could not be found in the appropriate repository (here `git@gitlab.haskell.org:ghc/haddock`).
Make sure to follow the instructions above and push the branch with a `wip/` prefix to that repository.
This error may sometimes be accompanied by the inability to force-push a rebased submodule – in that case, make sure you fetched all remotes of the main repo and submodule and ran `git submodule update --recursive`.
\ No newline at end of file