Concurrent executions of ghc-pkg can cause inconstant package.cache files
I am doing 24 way parallel builds of system images, including all packages on a system. This includes ghc and multiple ghc packages.
I am seeing intermittent dependency failure from the ghc packaging system. From examining Main.hs in ghc-pkg, I see the function withFileAtomic write to a temporary file in package.conf.d and then atomically rename on top of a target file, package.cache in the case. With parallel execution the last rename would win, leading to lost entries in package.cache.
In my case, the following things happened: ("Building" indicates a start, "Built" indicates completion, "Installing" is setup in a separate chroot'd environment and is isolated)
The FreeBSD ports system is used to drive the Haskell build system.
The process works single threaded and fails intermittently when done in parallel.
Building: devel/hs-data-default-instances-base Building: devel/hs-data-default-instances-containers Building: devel/hs-data-default-instances-old-locale Built: devel/hs-dlist Building: devel/hs-data-default-instances-dlist Built: devel/hs-temporary Built: jail-image-full Installing: system-image__jail-image-full Built: devel/hs-base64-bytestring Built: archivers/hs-zlib Building: security/hs-digest Built: devel/hs-syb Building: textproc/hs-hs-bibutils Building: textproc/hs-pandoc-types Built: devel/hs-utf8-string Built: devel/hs-data-default-instances-old-locale Built: devel/hs-data-default-instances-containers Built: devel/hs-data-default-instances-base Built: devel/hs-data-default-instances-dlist Building: devel/hs-data-default Built: devel/hs-random Installed: system-image__lang/ghc Installing: system-image__archivers/hs-zlib Installing: system-image__devel/hs-utf8-string Installing: system-image__devel/hs-syb Installing: system-image__devel/hs-base64-bytestring Installing: system-image__devel/hs-data-default-class Installing: system-image__devel/hs-dlist Installing: system-image__devel/hs-random Installing: system-image__devel/hs-temporary Installing: system-image__devel/hs-extensible-exceptions Built: devel/hs-data-default FAILED
The error from the Haskell data-default build was:
setup: At least the following dependencies are missing: data-default-instances-base -any
Looking in the in the package.conf.d directory shows that the data-default-instances-base-0.0.1-7bdf8678f0d8637e096e397e7910f82a.conf file was present, but running "ghc-pkg list" did not show data-default-instances-base
Running /usr/local/lib/cabal/ghc-7.6.3/data-default-instances-base-0.0.1/register.sh (which was also present) caused ghc-pkg to now show data-default-instances-base.
To me this looks like a race condition between multiple instances of ghc-pkg causing the cache to become inconsistent.
Trac metadata
| Trac field | Value |
|---|---|
| Version | 7.6.3 |
| Type | Bug |
| TypeOfFailure | OtherFailure |
| Priority | normal |
| Resolution | Unresolved |
| Component | Package system |
| Test case | |
| Differential revisions | |
| BlockedBy | |
| Related | |
| Blocking | |
| CC | |
| Operating system | FreeBSD |
| Architecture |