Object code produced by GHC is non-deterministic
This is a continuation of #4012 (closed), which was resolved in 8.0.2 by making interface files deterministic. Now someone needs to make object code deterministic.
Is blocked by
- #252749.12.1
- #252879.12.1
Relates to
Activity
mentioned in issue #4012 (closed)
mentioned in issue #7231 (closed)
mentioned in issue #10037
mentioned in issue #12262 (closed)
added Tbug Trac import labels
mentioned in issue #14769
Attached file
objdump-headers-1.txt(download).Attached file
objdump-headers-1.2.txt(download).Attached file
objdump-headers-2.txt(download).I have attached some files,
objdump-headers-1.txtandobjdump-headers-1.txtwhich are the outputs ofobjdump --all-headers Example.oas produced byghc --make -j1of a project of mine.Have a look at them in a diff viewer.
At the top you will find probably what was talked about before: Uniques making it into symbol names, like
cZoQ_infovscQJU_info.But further down you see
0000000000002f10 R_X86_64_PC32 .data.rel.ro+0x00000000000000e8vs
0000000000002f10 R_X86_64_PC32 .data.rel.ro+0x00000000000000f0which is the result of some values in the
RELOCATION RECORDS FOR [.data.rel.ro]having their positions swapped:0000000000000100 R_X86_64_64 monozmtraversablezm1zi0zi2zi1zm4aNtRX1lKHJF1rNRp4LmtV_DataziMonoTraversable_zdfMonoFoldableZMZN_closure 0000000000000108 R_X86_64_64 monozmtraversablezm1zi0zi2zi1zm4aNtRX1lKHJF1rNRp4LmtV_DataziMonoTraversableziUnprefixed_any_closurevs
0000000000000100 R_X86_64_64 monozmtraversablezm1zi0zi2zi1zm4aNtRX1lKHJF1rNRp4LmtV_DataziMonoTraversableziUnprefixed_any_closure 0000000000000108 R_X86_64_64 monozmtraversablezm1zi0zi2zi1zm4aNtRX1lKHJF1rNRp4LmtV_DataziMonoTraversable_zdfMonoFoldableZMZN_closureThese are the same things, they have exactly the same symbol names, just their order swapped.
How might this happen?
It's not an entirely trivial task. We use uniques quite often to name symbols in code produced by the native codegen. Fixing this without compromising compiler performance is rather tricky.
@bgamari Question:
Why cannot the uniques be predetermined by a seed that's dependent on the module name?
My understanding so far is this: We need an infinite stream of unused strings to generate symbols. Parallel compilation with
-jand "partial" compilation of the projects (resumes, incremental compilation), means some number or pseudorandom generator has a different initial state when we come by the file next time.So couldn't we initialise the seed of that pseudorandom generator as
hash(module name), so that each time a module is compiled, it always generates the same uniques in the same order (as within a module there is no parallel compilation)? And when a module name changes, we have to recompile anyway and it's probably OK if the object code changes when that happens.(Edit: Looks like this is the same question asked but unanswered here: #4012 (closed)##12935)
Trac metadata
Trac field Value BlockedBy - → #4012 (closed) Related - → #12262 (closed) 
So couldn't we initialise the seed of that pseudorandom generator as hash(module name), so that each time a module is compiled, it always generates the same uniques in the same order
Yes, I believe we could. The real issue is that (I suspect) this will be a rather major surgery of the backend. We produce symbol names and labels in many places and often we only have access to a unique supply. Threading the necessary state around will likely be a non-trivial exercise.
Trac metadata
Trac field Value BlockedBy #4012 (closed) → - Related #12262 (closed) → - @nh: What about collisions? Uniques are 64 bits on 64 bit systems. 8 bits are taken for the tag, that leaves us with 56 bits. You can expect collisions after
2^26uniques are allocated. That's67108864. It's a pretty small number. We've definitely seen cases of running out of 24 bits for larger projects.@niteria: First, I think there's a misunderstanding here. Hashes would be used only for hashing the module name, to determine another N-bit prefix similiar as for the tag (for example, you could have 8 bits for the tag, 28 bits for the hash of the module name, and 28 bits for the actual uniques). Within this sub-namespace (per-module) uniques would continue to be allocated in a (+1) fashion, counting up from 0 to
2^26-1. So there would never be the risk of a "uniques collision" as you describe, only the risk of a "module name hash colision".Second, to avoid module name hashes colliding: You just check them for uniqueness. This is easy, because you can do it ahead of time, before the build, where you already know all modules in the project. If they collide, you just use the next bigger integer. The point of that module name hashing (under the assumption that we have only 64 bits) is not to guarantee separate namespaces in all cases , just to make it very likely while also being stable against adding a new module to the project (you could also just enumerate the modules from 1 to N, but then if you add one in the middle, all modules get new subnamespaces assigned, and with name hashing that doesn't happen this way). If there's a collision, that's not the end of the world or a critical error, it just means you'll get different uniques next time (relatively unlikely with
2^28).And you still have the property that given the same project with same modules and same compiler, you're guaranteed to be fully deterministic. Just not between changes that add a module.
If we want to guarantee to be fully deterministic for individual modules even across additions of modules, all uniques have to be prefixed with the full module name (which might also be a good option).
I'm looking into doing this. First a remark:
Instead of hashing for extra stability, we could also just use the increasing
[1 of 3] Compiling Anumbers for unique namespacing. But I noticed that the order of those is currently not stable:In a project with modules
A, B, C, Main,Aimports nothing,BandCboth importA,MainimportsCandD, sometimes you will get a different output of which of these modules is[3 of 4]. Example:% touch B.hs && /raid/src/ghc/git-8.2.1/inplace/bin/ghc-stage2 --make Main.hs [3 of 4] Compiling B ...% touch C.hs && /raid/src/ghc/git-8.2.1/inplace/bin/ghc-stage2 --make Main.hs [3 of 4] Compiling C ...That is because in
GhcMake.upsweep'themod_indexis just counting up+1, but oversccs :: [SCC ModSummary]the order of which is somehow influenced by the file mtime (I haven't found yet how the mtime makes it to be a more significant field than the module name).OK, the reason for the ordering being dependent on mtime is #7231 (closed):
mg = stable_mg ++ unsable_mgwas implemented to fixGHCi erroneously unloads modules after a failed :reload.Trac metadata
Trac field Value Differential revisions - → D4388 mentioned in commit d8e47a2e
As I mentioned over on #14769, I don't think relying on the compilation order of modules for determinism is the right way to do this. The rationale is that the compilation order can change for other reasons: the user may invoke GHC on a different set of modules, without changing anything about the source code or dependencies of any particular module.
The determinism property we want is that the output depends only on
- The source code of a module,
- Its dependencies,
- compiler flags that affect the generated code
mentioned in issue #15058 (closed)
added determinism label
added Pnormal label
marked this issue as related to #4012 (closed)
mentioned in merge request !5616 (closed)
mentioned in merge request !5661 (merged)
Hello!
I've looked into this for reproducible Arch Linux, the packages I've checked so far are deterministic besides something in the debug data (causing the build id to be off, the debug section itself is stripped).
I've built haskell-tar twice and explicitly disabled stripping, then got this diff:
--- artifacts/a/haskell-tar-0.5.1.1-68-x86_64.pkg.tar.zst +++ artifacts/b/haskell-tar-0.5.1.1-68-x86_64.pkg.tar.zst ├── haskell-tar-0.5.1.1-68-x86_64.pkg.tar │ ├── file list │ │ @@ -1,9 +1,9 @@ │ │ -rw-r--r-- 0 root (0) root (0) 5200 2021-07-13 16:42:47.000000 .BUILDINFO │ │ --rw-r--r-- 0 root (0) root (0) 1598 2021-07-13 16:42:47.000000 .MTREE │ │ +-rw-r--r-- 0 root (0) root (0) 1600 2021-07-13 16:42:47.000000 .MTREE │ │ -rw-r--r-- 0 root (0) root (0) 532 2021-07-13 16:42:47.000000 .PKGINFO │ │ drwxr-xr-x 0 root (0) root (0) 0 2021-07-13 16:42:47.000000 usr/ │ │ drwxr-xr-x 0 root (0) root (0) 0 2021-07-13 16:42:47.000000 usr/lib/ │ │ drwxr-xr-x 0 root (0) root (0) 0 2021-07-13 16:42:47.000000 usr/lib/ghc-8.10.5/ │ │ drwxr-xr-x 0 root (0) root (0) 0 2021-07-13 16:42:47.000000 usr/lib/ghc-8.10.5/site-local/ │ │ drwxr-xr-x 0 root (0) root (0) 0 2021-07-13 16:42:47.000000 usr/lib/ghc-8.10.5/site-local/tar-0.5.1.1/ │ │ drwxr-xr-x 0 root (0) root (0) 0 2021-07-13 16:42:47.000000 usr/lib/ghc-8.10.5/site-local/tar-0.5.1.1/Codec/ │ ├── .MTREE │ │ ├── .MTREE-content │ │ │ @@ -1,15 +1,15 @@ │ │ │ #mtree │ │ │ /set type=file uid=0 gid=0 mode=644 │ │ │ ./.BUILDINFO time=1626194567.0 size=5200 md5digest=7c31f44b3ff69f07c51641ccf0439b33 sha256digest=f29b297048e291c2a86678d961b415bf499d5931a27af6fd086b274ed62c8a3b │ │ │ ./.PKGINFO time=1626194567.0 size=532 md5digest=cffa39cf85bac889e6c4fffb098c3f0d sha256digest=7635c2010a1e42f8513bc6c47e3b58925f7b56ad100958ddacf3e7959127ae44 │ │ │ /set mode=755 │ │ │ ./usr time=1626194567.0 type=dir │ │ │ ./usr/lib time=1626194567.0 type=dir │ │ │ -./usr/lib/libHStar-0.5.1.1-3BHZnOVLCAEhdQMv0Q6MU-ghc8.10.5.so time=1626194567.0 size=591024 md5digest=38c33725ed584e03ca55d00fd76d880d sha256digest=732520e562570f5e64180afe7af63040198213c17af0521f5068a7cb495c9b3a │ │ │ +./usr/lib/libHStar-0.5.1.1-3BHZnOVLCAEhdQMv0Q6MU-ghc8.10.5.so time=1626194567.0 size=591024 md5digest=3f6792c67e74a51a18f778fc1e758843 sha256digest=5d9a9f348426040bd7065d3914fbf11f42a5aed9ec7d8b82987e0ece2dd58b8b │ │ │ ./usr/lib/ghc-8.10.5 time=1626194567.0 type=dir │ │ │ ./usr/lib/ghc-8.10.5/site-local time=1626194567.0 type=dir │ │ │ ./usr/lib/ghc-8.10.5/site-local/tar-0.5.1.1 time=1626194567.0 type=dir │ │ │ ./usr/lib/ghc-8.10.5/site-local/tar-0.5.1.1/Codec time=1626194567.0 type=dir │ │ │ ./usr/lib/ghc-8.10.5/site-local/tar-0.5.1.1/Codec/Archive time=1626194567.0 type=dir │ │ │ ./usr/lib/ghc-8.10.5/site-local/tar-0.5.1.1/Codec/Archive/Tar.dyn_hi time=1626194567.0 mode=644 size=7052 md5digest=27725be08c04602dc29b1ab76c0b4721 sha256digest=2df518991cae76b3642aaab462ad31c3e113bcde06ec8c890c2b84d02f37f980 │ │ │ /set mode=644 │ ├── usr/lib/libHStar-0.5.1.1-3BHZnOVLCAEhdQMv0Q6MU-ghc8.10.5.so │ │ ├── readelf --wide --symbols {} │ │ │ @@ -350,17 +350,17 @@ │ │ │ 346: 0000000000000000 0 FUNC GLOBAL DEFAULT UND base_GHCziEnum_toEnum_info │ │ │ 347: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND stg_sel_0_upd_info │ │ │ 348: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND stg_sel_1_upd_info │ │ │ 349: 0000000000000000 0 OBJECT GLOBAL DEFAULT UND base_ControlziExceptionziBase_patError_closure │ │ │ 350: 0000000000000000 0 FUNC GLOBAL DEFAULT UND base_ControlziExceptionziBase_patError_info │ │ │ 351: 0000000000000000 0 OBJECT GLOBAL DEFAULT UND bytestringzm0zi10zi12zi0_DataziByteStringziInternal_zdfShowByteString_closure │ │ │ 352: 0000000000000000 0 FUNC GLOBAL DEFAULT UND containerszm0zi6zi4zi1_DataziMapziInternal_zdwzdcshowsPrec_info │ │ │ - 353: 0000000000000000 0 OBJECT GLOBAL DEFAULT UND base_GHCziInt_zdfIxInt32_closure │ │ │ - 354: 0000000000000000 0 OBJECT GLOBAL DEFAULT UND arrayzm0zi5zi4zi0_DataziArrayziBase_zdfIArrayUArrayInt32_closure │ │ │ - 355: 0000000000000000 0 OBJECT GLOBAL DEFAULT UND arrayzm0zi5zi4zi0_DataziArrayziBase_zdwzdsshowsIArray_closure │ │ │ + 353: 0000000000000000 0 OBJECT GLOBAL DEFAULT UND arrayzm0zi5zi4zi0_DataziArrayziBase_zdfIArrayUArrayInt32_closure │ │ │ + 354: 0000000000000000 0 OBJECT GLOBAL DEFAULT UND arrayzm0zi5zi4zi0_DataziArrayziBase_zdwzdsshowsIArray_closure │ │ │ + 355: 0000000000000000 0 OBJECT GLOBAL DEFAULT UND base_GHCziInt_zdfIxInt32_closure │ │ │ 356: 0000000000000000 0 OBJECT GLOBAL DEFAULT UND arrayzm0zi5zi4zi0_DataziArrayziBase_zdfIArrayUArrayWord32_closure │ │ │ 357: 0000000000000000 0 OBJECT GLOBAL DEFAULT UND base_GHCziWord_zdfShowWord32_closure │ │ │ 358: 0000000000000000 0 OBJECT GLOBAL DEFAULT UND base_GHCziInt_zdfShowInt32_closure │ │ │ 359: 0000000000000000 0 FUNC GLOBAL DEFAULT UND arrayzm0zi5zi4zi0_DataziArrayziBase_zdwzdsshowsIArray_info │ │ │ 360: 0000000000000000 0 OBJECT GLOBAL DEFAULT UND bytestringzm0zi10zi12zi0_DataziByteString_empty_closure │ │ │ 361: 0000000000000000 0 FUNC GLOBAL DEFAULT UND base_GHCziEnum_efdtIntUp_info │ │ │ 362: 0000000000000000 0 FUNC GLOBAL DEFAULT UND base_GHCziEnum_efdtIntDn_info │ │ ├── readelf --wide --relocs {} │ │ │ @@ -2706,19 +2706,19 @@ │ │ │ 000000000008e178 000000f700000001 R_X86_64_64 0000000000000000 base_GHCziShow_intToDigit_closure + 0 │ │ │ 0000000000088d10 000000f800000006 R_X86_64_GLOB_DAT 0000000000000000 base_GHCziShow_zdfShowInt_closure + 0 │ │ │ 0000000000088d18 000000f900000006 R_X86_64_GLOB_DAT 0000000000000000 base_GHCziReal_zdfIntegralInt_closure + 0 │ │ │ 000000000008e140 000000f900000001 R_X86_64_64 0000000000000000 base_GHCziReal_zdfIntegralInt_closure + 0 │ │ │ 0000000000088d20 000000fa00000006 R_X86_64_GLOB_DAT 0000000000000000 base_Numeric_showIntAtBase_info + 0 │ │ │ 0000000000088d28 000000fb00000006 R_X86_64_GLOB_DAT 0000000000000000 base_GHCziList_zdwunsafeTake_info + 0 │ │ │ 000000000008e150 000000fc00000001 R_X86_64_64 0000000000000000 base_Numeric_showIntAtBase_closure + 0 │ │ │ -000000000008e188 000000fc00000001 R_X86_64_64 0000000000000000 base_Numeric_showIntAtBase_closure + 0 │ │ │ +000000000008e180 000000fc00000001 R_X86_64_64 0000000000000000 base_Numeric_showIntAtBase_closure + 0 │ │ │ 0000000000088d30 000000fd00000006 R_X86_64_GLOB_DAT 0000000000000000 base_GHCziInt_zdfShowInt64_closure + 0 │ │ │ 0000000000088d38 000000fe00000006 R_X86_64_GLOB_DAT 0000000000000000 base_GHCziInt_zdfIntegralInt64_closure + 0 │ │ │ -000000000008e180 000000fe00000001 R_X86_64_64 0000000000000000 base_GHCziInt_zdfIntegralInt64_closure + 0 │ │ │ -000000000008e668 000000fe00000001 R_X86_64_64 0000000000000000 base_GHCziInt_zdfIntegralInt64_closure + 0 │ │ │ +000000000008e188 000000fe00000001 R_X86_64_64 0000000000000000 base_GHCziInt_zdfIntegralInt64_closure + 0 │ │ │ +000000000008e670 000000fe00000001 R_X86_64_64 0000000000000000 base_GHCziInt_zdfIntegralInt64_closure + 0 │ │ │ 000000000008e1c8 000000ff00000001 R_X86_64_64 0000000000000000 base_GHCziWord_W8zh_con_info + 0 │ │ │ 0000000000088d40 0000010000000006 R_X86_64_GLOB_DAT 0000000000000000 bytestringzm0zi10zi12zi0_DataziByteStringziInternal_zdfMonoidByteString_closure + 0 │ │ │ 000000000008e1e0 0000010000000001 R_X86_64_64 0000000000000000 bytestringzm0zi10zi12zi0_DataziByteStringziInternal_zdfMonoidByteString_closure + 0 │ │ │ 0000000000088d48 0000010100000006 R_X86_64_GLOB_DAT 0000000000000000 base_GHCziBase_mempty_info + 0 │ │ │ 000000000008e3a8 0000010200000001 R_X86_64_64 0000000000000000 bytestringzm0zi10zi12zi0_DataziByteStringziLazzy_zdwreplicate_closure + 0 │ │ │ 0000000000088d50 0000010300000006 R_X86_64_GLOB_DAT 0000000000000000 bytestringzm0zi10zi12zi0_DataziByteStringziLazzy_zdwreplicate_info + 0 │ │ │ 000000000008e460 0000010400000001 R_X86_64_64 0000000000000000 bytestringzm0zi10zi12zi0_DataziByteStringziLazzyziInternal_zdwpackChunks1_closure + 0 │ │ │ @@ -2726,15 +2726,15 @@ │ │ │ 000000000008e870 0000010500000001 R_X86_64_64 0000000000000000 stg_SRT_5_info + 0 │ │ │ 0000000000088d68 0000010600000006 R_X86_64_GLOB_DAT 0000000000000000 base_SystemziPosixziTypes_zdfIntegralCMode_closure + 0 │ │ │ 000000000008e5b0 0000010600000001 R_X86_64_64 0000000000000000 base_SystemziPosixziTypes_zdfIntegralCMode_closure + 0 │ │ │ 0000000000088d60 0000010700000006 R_X86_64_GLOB_DAT 0000000000000000 base_SystemziPosixziTypes_zdfShowCMode_closure + 0 │ │ │ 0000000000088d70 0000010800000006 R_X86_64_GLOB_DAT 0000000000000000 bytestringzm0zi10zi12zi0_DataziByteStringziLazzyziInternal_zdwpackChunks1_info + 0 │ │ │ 0000000000088d78 0000010900000006 R_X86_64_GLOB_DAT 0000000000000000 stg_noDuplicatezh + 0 │ │ │ 0000000000088d80 0000010a00000006 R_X86_64_GLOB_DAT 0000000000000000 base_GHCziIOziUnsafe_unsafeDupableInterleaveIO_info + 0 │ │ │ -000000000008e670 0000010b00000001 R_X86_64_64 0000000000000000 base_GHCziReal_zdwzdszdcfloor_closure + 0 │ │ │ +000000000008e668 0000010b00000001 R_X86_64_64 0000000000000000 base_GHCziReal_zdwzdszdcfloor_closure + 0 │ │ │ 000000000008e688 0000010c00000001 R_X86_64_64 0000000000000000 base_GHCziReal_zdwzdszdczs_closure + 0 │ │ │ 000000000008e6a8 0000010d00000001 R_X86_64_64 0000000000000000 timezm1zi9zi3_DataziTimeziClockziPOSIX_zdwutcTimeToPOSIXSeconds_closure + 0 │ │ │ 0000000000088d88 0000010e00000006 R_X86_64_GLOB_DAT 0000000000000000 timezm1zi9zi3_DataziTimeziClockziPOSIX_zdwutcTimeToPOSIXSeconds_info + 0 │ │ │ 0000000000088d90 0000010f00000006 R_X86_64_GLOB_DAT 0000000000000000 base_GHCziReal_zdfEnumRatio1_closure + 0 │ │ │ 0000000000088d98 0000011000000006 R_X86_64_GLOB_DAT 0000000000000000 base_DataziFixed_zdfHasResolutionTYPEE5_closure + 0 │ │ │ 0000000000088da0 0000011100000006 R_X86_64_GLOB_DAT 0000000000000000 base_GHCziReal_zdwzdszdczs_info + 0 │ │ │ 0000000000088da8 0000011200000006 R_X86_64_GLOB_DAT 0000000000000000 base_GHCziReal_zdwzdszdcfloor_info + 0 │ │ │ @@ -2856,24 +2856,24 @@ │ │ │ 0000000000088f20 0000015b00000006 R_X86_64_GLOB_DAT 0000000000000000 stg_sel_0_upd_info + 0 │ │ │ 0000000000088f28 0000015c00000006 R_X86_64_GLOB_DAT 0000000000000000 stg_sel_1_upd_info + 0 │ │ │ 000000000008f670 0000015d00000001 R_X86_64_64 0000000000000000 base_ControlziExceptionziBase_patError_closure + 0 │ │ │ 0000000000088f30 0000015e00000006 R_X86_64_GLOB_DAT 0000000000000000 base_ControlziExceptionziBase_patError_info + 0 │ │ │ 0000000000088f38 0000015f00000006 R_X86_64_GLOB_DAT 0000000000000000 bytestringzm0zi10zi12zi0_DataziByteStringziInternal_zdfShowByteString_closure + 0 │ │ │ 000000000008f820 0000015f00000001 R_X86_64_64 0000000000000000 bytestringzm0zi10zi12zi0_DataziByteStringziInternal_zdfShowByteString_closure + 0 │ │ │ 0000000000088f40 0000016000000006 R_X86_64_GLOB_DAT 0000000000000000 containerszm0zi6zi4zi1_DataziMapziInternal_zdwzdcshowsPrec_info + 0 │ │ │ -0000000000088f58 0000016100000006 R_X86_64_GLOB_DAT 0000000000000000 base_GHCziInt_zdfIxInt32_closure + 0 │ │ │ -000000000008f8d8 0000016100000001 R_X86_64_64 0000000000000000 base_GHCziInt_zdfIxInt32_closure + 0 │ │ │ -000000000008f900 0000016100000001 R_X86_64_64 0000000000000000 base_GHCziInt_zdfIxInt32_closure + 0 │ │ │ -0000000000088f70 0000016200000006 R_X86_64_GLOB_DAT 0000000000000000 arrayzm0zi5zi4zi0_DataziArrayziBase_zdfIArrayUArrayInt32_closure + 0 │ │ │ -000000000008f8e0 0000016200000001 R_X86_64_64 0000000000000000 arrayzm0zi5zi4zi0_DataziArrayziBase_zdfIArrayUArrayInt32_closure + 0 │ │ │ -000000000008f8e8 0000016300000001 R_X86_64_64 0000000000000000 arrayzm0zi5zi4zi0_DataziArrayziBase_zdwzdsshowsIArray_closure + 0 │ │ │ -000000000008f910 0000016300000001 R_X86_64_64 0000000000000000 arrayzm0zi5zi4zi0_DataziArrayziBase_zdwzdsshowsIArray_closure + 0 │ │ │ -0000000000090a58 0000016300000001 R_X86_64_64 0000000000000000 arrayzm0zi5zi4zi0_DataziArrayziBase_zdwzdsshowsIArray_closure + 0 │ │ │ +0000000000088f70 0000016100000006 R_X86_64_GLOB_DAT 0000000000000000 arrayzm0zi5zi4zi0_DataziArrayziBase_zdfIArrayUArrayInt32_closure + 0 │ │ │ +000000000008f8d8 0000016100000001 R_X86_64_64 0000000000000000 arrayzm0zi5zi4zi0_DataziArrayziBase_zdfIArrayUArrayInt32_closure + 0 │ │ │ +000000000008f8e0 0000016200000001 R_X86_64_64 0000000000000000 arrayzm0zi5zi4zi0_DataziArrayziBase_zdwzdsshowsIArray_closure + 0 │ │ │ +000000000008f908 0000016200000001 R_X86_64_64 0000000000000000 arrayzm0zi5zi4zi0_DataziArrayziBase_zdwzdsshowsIArray_closure + 0 │ │ │ +0000000000090a58 0000016200000001 R_X86_64_64 0000000000000000 arrayzm0zi5zi4zi0_DataziArrayziBase_zdwzdsshowsIArray_closure + 0 │ │ │ +0000000000088f58 0000016300000006 R_X86_64_GLOB_DAT 0000000000000000 base_GHCziInt_zdfIxInt32_closure + 0 │ │ │ +000000000008f8e8 0000016300000001 R_X86_64_64 0000000000000000 base_GHCziInt_zdfIxInt32_closure + 0 │ │ │ +000000000008f910 0000016300000001 R_X86_64_64 0000000000000000 base_GHCziInt_zdfIxInt32_closure + 0 │ │ │ 0000000000088f60 0000016400000006 R_X86_64_GLOB_DAT 0000000000000000 arrayzm0zi5zi4zi0_DataziArrayziBase_zdfIArrayUArrayWord32_closure + 0 │ │ │ -000000000008f908 0000016400000001 R_X86_64_64 0000000000000000 arrayzm0zi5zi4zi0_DataziArrayziBase_zdfIArrayUArrayWord32_closure + 0 │ │ │ +000000000008f900 0000016400000001 R_X86_64_64 0000000000000000 arrayzm0zi5zi4zi0_DataziArrayziBase_zdfIArrayUArrayWord32_closure + 0 │ │ │ 0000000000090a50 0000016400000001 R_X86_64_64 0000000000000000 arrayzm0zi5zi4zi0_DataziArrayziBase_zdfIArrayUArrayWord32_closure + 0 │ │ │ 0000000000088f48 0000016500000006 R_X86_64_GLOB_DAT 0000000000000000 base_GHCziWord_zdfShowWord32_closure + 0 │ │ │ 0000000000088f50 0000016600000006 R_X86_64_GLOB_DAT 0000000000000000 base_GHCziInt_zdfShowInt32_closure + 0 │ │ │ 0000000000088f68 0000016700000006 R_X86_64_GLOB_DAT 0000000000000000 arrayzm0zi5zi4zi0_DataziArrayziBase_zdwzdsshowsIArray_info + 0 │ │ │ 0000000000088f78 0000016800000006 R_X86_64_GLOB_DAT 0000000000000000 bytestringzm0zi10zi12zi0_DataziByteString_empty_closure + 0 │ │ │ 000000000008fa10 0000016800000001 R_X86_64_64 0000000000000000 bytestringzm0zi10zi12zi0_DataziByteString_empty_closure + 0 │ │ │ 000000000008fac8 0000016800000001 R_X86_64_64 0000000000000000 bytestringzm0zi10zi12zi0_DataziByteString_empty_closure + 0 │ │ ├── readelf --wide --notes {} │ │ │ @@ -1,12 +1,12 @@ │ │ │ │ │ │ Displaying notes found in: .note.gnu.property │ │ │ Owner Data size Description │ │ │ GNU 0x00000010 NT_GNU_PROPERTY_TYPE_0 Properties: x86 feature used: x86 │ │ │ │ │ │ Displaying notes found in: .note.gnu.build-id │ │ │ Owner Data size Description │ │ │ - GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring) Build ID: f9022081d58bb3926e57f1b8112c445df7e65489 │ │ │ + GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring) Build ID: 6b5b81bd3e58280e6c44ce46febb1b9f2fb29038 │ │ │ │ │ │ Displaying notes found in: .note.gnu.gold-version │ │ │ Owner Data size Description │ │ │ GNU 0x00000009 NT_GNU_GOLD_VERSION (gold version) Version: gold 1.16 │ │ ├── strings --all --bytes=8 {} │ │ │ @@ -1556,17 +1556,17 @@ │ │ │ tarzm0zi5zi1zi1zm3BHZZnOVLCAEhdQMv0Q6MU_CodecziArchiveziTarziIndexziStringTable_zdfShowStringTableBuilderzuzdcshow_closure │ │ │ tarzm0zi5zi1zi1zm3BHZZnOVLCAEhdQMv0Q6MU_CodecziArchiveziTarziIndexziStringTable_zdfShowStringTable2_closure │ │ │ tarzm0zi5zi1zi1zm3BHZZnOVLCAEhdQMv0Q6MU_CodecziArchiveziTarziIndexziStringTable_zdfShowStringTableBuilderzuzdcshowList_info │ │ │ tarzm0zi5zi1zi1zm3BHZZnOVLCAEhdQMv0Q6MU_CodecziArchiveziTarziIndexziStringTable_zdfShowStringTableBuilderzuzdcshowList_closure │ │ │ tarzm0zi5zi1zi1zm3BHZZnOVLCAEhdQMv0Q6MU_CodecziArchiveziTarziIndexziStringTable_zdfShowStringTableBuilder_info │ │ │ tarzm0zi5zi1zi1zm3BHZZnOVLCAEhdQMv0Q6MU_CodecziArchiveziTarziIndexziStringTable_zdfShowStringTableBuilder_closure │ │ │ tarzm0zi5zi1zi1zm3BHZZnOVLCAEhdQMv0Q6MU_CodecziArchiveziTarziIndexziStringTable_zdfShowStringTable3_bytes │ │ │ -base_GHCziInt_zdfIxInt32_closure │ │ │ arrayzm0zi5zi4zi0_DataziArrayziBase_zdfIArrayUArrayInt32_closure │ │ │ arrayzm0zi5zi4zi0_DataziArrayziBase_zdwzdsshowsIArray_closure │ │ │ +base_GHCziInt_zdfIxInt32_closure │ │ │ arrayzm0zi5zi4zi0_DataziArrayziBase_zdfIArrayUArrayWord32_closure │ │ │ base_GHCziWord_zdfShowWord32_closure │ │ │ base_GHCziInt_zdfShowInt32_closure │ │ │ arrayzm0zi5zi4zi0_DataziArrayziBase_zdwzdsshowsIArray_info │ │ │ tarzm0zi5zi1zi1zm3BHZZnOVLCAEhdQMv0Q6MU_CodecziArchiveziTarziIndexziStringTable_zdfShowStringTablezuzdcshow_info │ │ │ tarzm0zi5zi1zi1zm3BHZZnOVLCAEhdQMv0Q6MU_CodecziArchiveziTarziIndexziStringTable_zdfShowStringTablezuzdcshow_closure │ │ │ tarzm0zi5zi1zi1zm3BHZZnOVLCAEhdQMv0Q6MU_CodecziArchiveziTarziIndexziStringTable_zdfShowStringTable1_info │ │ ├── readelf --wide --decompress --hex-dump=.dynstr {} │ │ │ @@ -8202,24 +8202,24 @@ │ │ │ 0x0002a940 696c6465 725f636c 6f737572 65007461 ilder_closure.ta │ │ │ 0x0002a950 727a6d30 7a69357a 69317a69 317a6d33 rzm0zi5zi1zi1zm3 │ │ │ 0x0002a960 42485a5a 6e4f564c 43414568 64514d76 BHZZnOVLCAEhdQMv │ │ │ 0x0002a970 3051364d 555f436f 6465637a 69417263 0Q6MU_CodecziArc │ │ │ 0x0002a980 68697665 7a695461 727a6949 6e646578 hiveziTarziIndex │ │ │ 0x0002a990 7a695374 72696e67 5461626c 655f7a64 ziStringTable_zd │ │ │ 0x0002a9a0 6653686f 77537472 696e6754 61626c65 fShowStringTable │ │ │ - 0x0002a9b0 335f6279 74657300 62617365 5f474843 3_bytes.base_GHC │ │ │ - 0x0002a9c0 7a69496e 745f7a64 66497849 6e743332 ziInt_zdfIxInt32 │ │ │ - 0x0002a9d0 5f636c6f 73757265 00617272 61797a6d _closure.arrayzm │ │ │ - 0x0002a9e0 307a6935 7a69347a 69305f44 6174617a 0zi5zi4zi0_Dataz │ │ │ - 0x0002a9f0 69417272 61797a69 42617365 5f7a6466 iArrayziBase_zdf │ │ │ - 0x0002aa00 49417272 61795541 72726179 496e7433 IArrayUArrayInt3 │ │ │ - 0x0002aa10 325f636c 6f737572 65006172 7261797a 2_closure.arrayz │ │ │ - 0x0002aa20 6d307a69 357a6934 7a69305f 44617461 m0zi5zi4zi0_Data │ │ │ - 0x0002aa30 7a694172 7261797a 69426173 655f7a64 ziArrayziBase_zd │ │ │ - 0x0002aa40 777a6473 73686f77 73494172 7261795f wzdsshowsIArray_ │ │ │ + 0x0002a9b0 335f6279 74657300 61727261 797a6d30 3_bytes.arrayzm0 │ │ │ + 0x0002a9c0 7a69357a 69347a69 305f4461 74617a69 zi5zi4zi0_Datazi │ │ │ + 0x0002a9d0 41727261 797a6942 6173655f 7a646649 ArrayziBase_zdfI │ │ │ + 0x0002a9e0 41727261 79554172 72617949 6e743332 ArrayUArrayInt32 │ │ │ + 0x0002a9f0 5f636c6f 73757265 00617272 61797a6d _closure.arrayzm │ │ │ + 0x0002aa00 307a6935 7a69347a 69305f44 6174617a 0zi5zi4zi0_Dataz │ │ │ + 0x0002aa10 69417272 61797a69 42617365 5f7a6477 iArrayziBase_zdw │ │ │ + 0x0002aa20 7a647373 686f7773 49417272 61795f63 zdsshowsIArray_c │ │ │ + 0x0002aa30 6c6f7375 72650062 6173655f 4748437a losure.base_GHCz │ │ │ + 0x0002aa40 69496e74 5f7a6466 4978496e 7433325f iInt_zdfIxInt32_ │ │ │ 0x0002aa50 636c6f73 75726500 61727261 797a6d30 closure.arrayzm0 │ │ │ 0x0002aa60 7a69357a 69347a69 305f4461 74617a69 zi5zi4zi0_Datazi │ │ │ 0x0002aa70 41727261 797a6942 6173655f 7a646649 ArrayziBase_zdfI │ │ │ 0x0002aa80 41727261 79554172 72617957 6f726433 ArrayUArrayWord3 │ │ │ 0x0002aa90 325f636c 6f737572 65006261 73655f47 2_closure.base_G │ │ │ 0x0002aaa0 48437a69 576f7264 5f7a6466 53686f77 HCziWord_zdfShow │ │ │ 0x0002aab0 576f7264 33325f63 6c6f7375 72650062 Word32_closure.bThe part of the diff that demonstrates the issue best is this one:
│ │ ├── strings --all --bytes=8 {} │ │ │ @@ -1556,17 +1556,17 @@ │ │ │ tarzm0zi5zi1zi1zm3BHZZnOVLCAEhdQMv0Q6MU_CodecziArchiveziTarziIndexziStringTable_zdfShowStringTableBuilderzuzdcshow_closure │ │ │ tarzm0zi5zi1zi1zm3BHZZnOVLCAEhdQMv0Q6MU_CodecziArchiveziTarziIndexziStringTable_zdfShowStringTable2_closure │ │ │ tarzm0zi5zi1zi1zm3BHZZnOVLCAEhdQMv0Q6MU_CodecziArchiveziTarziIndexziStringTable_zdfShowStringTableBuilderzuzdcshowList_info │ │ │ tarzm0zi5zi1zi1zm3BHZZnOVLCAEhdQMv0Q6MU_CodecziArchiveziTarziIndexziStringTable_zdfShowStringTableBuilderzuzdcshowList_closure │ │ │ tarzm0zi5zi1zi1zm3BHZZnOVLCAEhdQMv0Q6MU_CodecziArchiveziTarziIndexziStringTable_zdfShowStringTableBuilder_info │ │ │ tarzm0zi5zi1zi1zm3BHZZnOVLCAEhdQMv0Q6MU_CodecziArchiveziTarziIndexziStringTable_zdfShowStringTableBuilder_closure │ │ │ tarzm0zi5zi1zi1zm3BHZZnOVLCAEhdQMv0Q6MU_CodecziArchiveziTarziIndexziStringTable_zdfShowStringTable3_bytes │ │ │ -base_GHCziInt_zdfIxInt32_closure │ │ │ arrayzm0zi5zi4zi0_DataziArrayziBase_zdfIArrayUArrayInt32_closure │ │ │ arrayzm0zi5zi4zi0_DataziArrayziBase_zdwzdsshowsIArray_closure │ │ │ +base_GHCziInt_zdfIxInt32_closure │ │ │ arrayzm0zi5zi4zi0_DataziArrayziBase_zdfIArrayUArrayWord32_closure │ │ │ base_GHCziWord_zdfShowWord32_closure │ │ │ base_GHCziInt_zdfShowInt32_closure │ │ │ arrayzm0zi5zi4zi0_DataziArrayziBase_zdwzdsshowsIArray_info │ │ │ tarzm0zi5zi1zi1zm3BHZZnOVLCAEhdQMv0Q6MU_CodecziArchiveziTarziIndexziStringTable_zdfShowStringTablezuzdcshow_info │ │ │ tarzm0zi5zi1zi1zm3BHZZnOVLCAEhdQMv0Q6MU_CodecziArchiveziTarziIndexziStringTable_zdfShowStringTablezuzdcshow_closure │ │ │ tarzm0zi5zi1zi1zm3BHZZnOVLCAEhdQMv0Q6MU_CodecziArchiveziTarziIndexziStringTable_zdfShowStringTable1_infoAs you can see it's only the position of
base_GHCziInt_zdfIxInt32_closurethat changed between those two builds. A lot of the other differences are offsets (the build id is a hash of everything).I've asked in irc and got this reply:
<hsyl20> but if it's the only source of non-determinism in this case, we could perhaps make the Ord instance of CLabel deterministic?Unfortunately I'm not an expert so I don't know, maybe somebody in here knows how this could be fixed?
If you have any questions or would like us to test something you're very welcome to reach out in libera/#archlinux-reproducible!
Edited by kpcyrdQuick update, mpickering on irc suggested trying
-j1and our package builds deterministically then:==> Leaving fakeroot environment. ==> Finished making: haskell-tar 0.5.1.1-68 (Tue 13 Jul 2021 07:58:15 PM CEST) -> built succeeded! built packages can be found in /var/lib/archbuild/reproducible/testenv/pkgdest ==> comparing artifacts... -> Package 'artifacts/a/haskell-tar-0.5.1.1-68-x86_64.pkg.tar.zst' successfully reproduced!I ran the rebuild multiple times and always got the same result.
My change looks like this:
@@ -27,7 +28,7 @@ --prefix=/usr --docdir=/usr/share/doc/$pkgname --enable-tests \ --dynlibdir=/usr/lib --libsubdir=\$compiler/site-local/\$pkgid \ -f-old-time - runhaskell Setup build $MAKEFLAGS + runhaskell Setup build -j1 runhaskell Setup register --gen-script runhaskell Setup unregister --gen-script sed -i -r -e "s|ghc-pkg.*update[^ ]* |&'--force' |" register.shThe value of
$MAKEFLAGSvaries, on our regular buildsystem it's-j65, which seems to cause non-determinism in ghc.Edited by kpcyrdIn openSUSE, we also made it deterministic using non-parallel builds for now = https://github.com/fedora-haskell/ghc-rpm-macros/commit/331f527a6d82c555f08fd5134a6b5cf27b2cf828
mentioned in merge request !7338 (closed)
mentioned in merge request !7353 (closed)
mentioned in issue #23299 (closed)
assigned to @alt-romes
I've got a working prototype in !12680 (merged) that I've successfully tested locally on macOS by diffing all object files produced from compiling the Cabal library in two separate runs.
On Linux or validate jobs it looks like there is still work to be done (going by CI). There are also possible non-deterministic sources that I haven't addressed like register allocation, but none of them have shown up in Cabal's object files.
Edited by Rodrigo Mesquita@alt-romes That's cool! Can you summarise your?
mentioned in commit bfd2259b
mentioned in commit 1f94ce0e
mentioned in commit 448e10da
mentioned in commit a7e58e41
mentioned in commit 3406f4d9
