GHC crashes after using up more than 16GB of RAM ("protocol-buffers-descriptor" package)
Summary
I am trying to compile https://hackage.haskell.org/package/protocol-buffers-descriptor using GHC 8.10.1
When it gets to the file src-auto-generated/Text/DescriptorProtos/FileOptions.hs it compiles for several minutes using up more and more memory until it finally crashes with out of memory (I have 16 GB + swap)
Steps to reproduce
$ ghc --version
8.10.1
$ cabal --version
3.2.0.0
$ git clone https://github.com/k-bx/protocol-buffers.git
$ cd protocol-buffers
$ git checkout 5986692b2de759458738049a484d5cda72b71f10
$ cd descriptor
$ cabal update
$ cabal new-build
....
....
[19 of 29] Compiling Text.DescriptorProtos.FileOptions ( src-auto-generated/Text/DescriptorProtos/FileOptions.hs,
/home/bitc/tmp/protocol-buffers/descriptor/dist-newstyle/build/x86_64-linux/ghc-8.10.1/protocol-buffers-descriptor-2.4.13/build/Text/DescriptorProtos/FileOptions.o, /home/bitc/tmp/protocol-buffers/descriptor/dist-newstyle/build/x86_64-linux/ghc-8.10.1/protocol-buffers-descriptor-2.4.13/build/Text/DescriptorProtos/FileOptions.dyn_o )(...hangs and uses up tons of memory)
Standalone reproducer
The following program is a condensed version of what goes wrong:
{-# LANGUAGE BangPatterns #-}
module T18140 where
class Mergeable a where
mergeAppend :: a -> a -> a
mergeAppend _a b = b
instance Mergeable Bool
instance Mergeable a => Mergeable (Maybe a) where
mergeAppend = mayMerge
mayMerge :: (Mergeable b) => Maybe b -> Maybe b -> Maybe b
mayMerge Nothing y = y
mayMerge x Nothing = x
mayMerge (Just x) (Just y) = Just (mergeAppend x y)
data D = D
!(Maybe Bool)
!(Maybe Bool)
!(Maybe Bool)
!(Maybe Bool)
!(Maybe Bool)
!(Maybe Bool)
!(Maybe Bool)
!(Maybe Bool)
!(Maybe Bool)
!(Maybe Bool)
!(Maybe Bool)
!(Maybe Bool)
!(Maybe Bool)
!(Maybe Bool)
!(Maybe Bool)
!(Maybe Bool)
!(Maybe Bool)
!(Maybe Bool)
instance Mergeable D where
mergeAppend
(D x'1 x'2 x'3 x'4 x'5 x'6 x'7 x'8 x'9 x'10 x'11 x'12 x'13 x'14 x'15 x'16 x'17 x'18)
(D y'1 y'2 y'3 y'4 y'5 y'6 y'7 y'8 y'9 y'10 y'11 y'12 y'13 y'14 y'15 y'16 y'17 y'18)
= D
(mergeAppend x'1 y'1)
(mergeAppend x'2 y'2)
(mergeAppend x'3 y'3)
(mergeAppend x'4 y'4)
(mergeAppend x'5 y'5)
(mergeAppend x'6 y'6)
(mergeAppend x'7 y'7)
(mergeAppend x'8 y'8)
(mergeAppend x'9 y'9)
(mergeAppend x'10 y'10)
(mergeAppend x'11 y'11)
(mergeAppend x'12 y'12)
(mergeAppend x'13 y'13)
(mergeAppend x'14 y'14)
(mergeAppend x'15 y'15)
(mergeAppend x'16 y'16)
(mergeAppend x'17 y'17)
(mergeAppend x'18 y'18)The behavior is exponential in the number of fields n, roughly 2^n or 3^n.
Environment
- GHC version used: 8.10.1
- Cabal: 3.2.0.0
Optional:
- Operating System: Linux
- System Architecture: x86_64
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
No child items are currently assigned. Use child items to break down this issue into smaller parts.
Activity
added needs triage label
I am fairly certain the problem is these:
(P'.mergeAppend x'8 y'8)lines. When I change each of those lines to
(Prelude'.undefined)then the file successfully compiles. I changed all of them (from 6 to 18) to(Prelude'.undefined)and then put them back one by one. the compile time becomes longer and longer (feels like exponentially longer)added Phigh Tbug compiler perf labels and removed needs triage label
Here is a self-contained reproducer (it seems). To be compiled with
-O1. The critical parts seem to be: strict fields of the datatype and wrapping fields inMaybe(removing any of those make the problem disappear).Edited by Artem Pelenitsyn-
Bisected it down to b78cc64e.
I can confirm that the Simplifier blows up on the new reproducer from #18140 (comment 270612):
*** Simplifier [Main]: Result size of Simplifier iteration=1 = {terms: 554, types: 396, coercions: 27, joins: 0/17} Result size of Simplifier iteration=2 = {terms: 1,100, types: 698, coercions: 27, joins: 0/49} Result size of Simplifier iteration=3 = {terms: 2,666, types: 1,556, coercions: 27, joins: 0/141} Result size of Simplifier iteration=4 = {terms: 7,148, types: 3,986, coercions: 27, joins: 0/405} Result size of Simplifier = {terms: 7,148, types: 3,986, coercions: 27, joins: 0/405} !!! Simplifier [Main]: finished in 43.52 milliseconds, allocated 37.833 megabytes *** Simplifier [Main]: Result size of Simplifier iteration=1 = {terms: 19,938, types: 10,828, coercions: 27, joins: 0/1,161} Result size of Simplifier iteration=2 = {terms: 56,388, types: 30,106, coercions: 27, joins: 0/3,321} Result size of Simplifier iteration=3 = {terms: 159,906, types: 84,052, coercions: 27, joins: 0/9,477} Result size of Simplifier iteration=4 = {terms: 452,964, types: 234,226, coercions: 27, joins: 0/26,973} Result size of Simplifier = {terms: 452,964, types: 234,226, coercions: 27, joins: 0/26,973} !!! Simplifier [Main]: finished in 1496.23 milliseconds, allocated 2140.383 megabytes *** Simplifier [Main]: Result size of Simplifier iteration=1 = {terms: 1,629,570, types: 1,170,262, coercions: 27, joins: 65,610/196,830} Result size of Simplifier iteration=2 = {terms: 1,667,089, types: 973,609, coercions: 27, joins: 59,322/71,898} Result size of Simplifier iteration=3 = {terms: 1,634,071, types: 945,015, coercions: 27, joins: 59,048/59,048} Result size of Simplifier = {terms: 1,634,071, types: 945,015, coercions: 27, joins: 59,048/59,048} !!! Simplifier [Main]: finished in 21195.57 milliseconds, allocated 26169.819 megabytesEvery additional
!(Maybe Bool)field ofDessentially triples the amount of terms after the second Simplifier run. So clearly unwanted exponential behavior.Edited by Sebastian Graf-
I updated the OP with the standalone reproducer. Feel free to change back if your original problem isn't fixed when we fix the reproducer, @bitc.
Apparently the fact that DataCon wrappers live longer now ofr some reason means that the Simplifier inlines join points more aggressively in the definition of
mergeAppend, or doesn't create join points in the first place.Previously (I used 8.4.3), we get this with 4 fields:
T18140.$fMergeableD_$cmergeAppend = \ (w_s14h :: D) (w1_s14i :: D) -> case w_s14h of { D ww1_s14l ww2_s14m ww3_s14n ww4_s14o -> case w1_s14i of { D ww6_s14s ww7_s14t ww8_s14u ww9_s14v -> join { $j_s12I [Dmd=<C(S),1*C1(U(U,U,U,U))>] :: Maybe Bool -> D [LclId[JoinId(1)], Arity=1, Str=<L,U>m, Unf=OtherCon []] $j_s12I (dt_XYH [OS=OneShot] :: Maybe Bool) = join { $j1_s12F [Dmd=<C(S),1*C1(U(U,U,U,U))>] :: Maybe Bool -> D [LclId[JoinId(1)], Arity=1, Str=<L,U>m, Unf=OtherCon []] $j1_s12F (dt1_XYJ [OS=OneShot] :: Maybe Bool) = case ww3_s14n of wild_X1r { Nothing -> case ww4_s14o of wild1_X2E { Nothing -> T18140.D dt_XYH dt1_XYJ ww8_s14u ww9_s14v; Just ipv_s11s -> case ww9_s14v of wild2_Xa { Nothing -> T18140.D dt_XYH dt1_XYJ ww8_s14u wild1_X2E; Just ipv1_s11v -> T18140.D dt_XYH dt1_XYJ ww8_s14u wild2_Xa } }; Just ipv_s11s -> case ww8_s14u of wild1_Xa { Nothing -> case ww4_s14o of wild2_X2J { Nothing -> T18140.D dt_XYH dt1_XYJ wild_X1r ww9_s14v; Just ipv1_X12M -> case ww9_s14v of wild3_X1x { Nothing -> T18140.D dt_XYH dt1_XYJ wild_X1r wild2_X2J; Just ipv2_s11v -> T18140.D dt_XYH dt1_XYJ wild_X1r wild3_X1x } }; Just ipv1_s11v -> case ww4_s14o of wild2_X2K { Nothing -> T18140.D dt_XYH dt1_XYJ wild1_Xa ww9_s14v; Just ipv2_X12N -> case ww9_s14v of wild3_X1y { Nothing -> T18140.D dt_XYH dt1_XYJ wild1_Xa wild2_X2K; Just ipv3_X12V -> T18140.D dt_XYH dt1_XYJ wild1_Xa wild3_X1y } } } } } in case ww2_s14m of wild_X1r { Nothing -> jump $j1_s12F ww7_s14t; Just ipv_s11s -> case ww7_s14t of wild1_Xa { Nothing -> jump $j1_s12F wild_X1r; Just ipv1_s11v -> jump $j1_s12F wild1_Xa } } } in case ww1_s14l of wild_X1r { Nothing -> jump $j_s12I ww6_s14s; Just ipv_s11s -> case ww6_s14s of wild1_Xa { Nothing -> jump $j_s12I wild_X1r; Just ipv1_s11v -> jump $j_s12I wild1_Xa } } } }Now we get this:
T18140.$fMergeableD_$cmergeAppend = \ (w_sNj :: D) (w1_sNk :: D) -> case w_sNj of ww_sNm { D ww1_sNn ww2_sNo ww3_sNp ww4_sNq -> case w1_sNk of ww5_sNt { D ww6_sNu ww7_sNv ww8_sNw ww9_sNx -> case ww1_sNn of wild_X8 { Nothing -> case ww2_sNo of wild1_X9 { Nothing -> case ww3_sNp of wild2_Xa { Nothing -> case ww4_sNq of wild3_Xb { Nothing -> ww5_sNt; Just ipv_sCI -> case ww9_sNx of { Nothing -> T18140.D ww6_sNu ww7_sNv ww8_sNw wild3_Xb; Just ipv1_sCL -> ww5_sNt } }; Just ipv_sCI -> case ww8_sNw of wild3_Xb { Nothing -> case ww4_sNq of wild4_Xc { Nothing -> T18140.D ww6_sNu ww7_sNv wild2_Xa ww9_sNx; Just ipv1_Xd -> case ww9_sNx of wild5_Xe { Nothing -> T18140.D ww6_sNu ww7_sNv wild2_Xa wild4_Xc; Just ipv2_sCL -> T18140.D ww6_sNu ww7_sNv wild2_Xa wild5_Xe } }; Just ipv1_sCL -> case ww4_sNq of wild4_Xc { Nothing -> ww5_sNt; Just ipv2_Xd -> case ww9_sNx of { Nothing -> T18140.D ww6_sNu ww7_sNv wild3_Xb wild4_Xc; Just ipv3_Xf -> ww5_sNt } } } }; Just ipv_sCI -> case ww7_sNv of wild2_Xa { Nothing -> case ww3_sNp of wild3_Xb { Nothing -> case ww4_sNq of wild4_Xc { Nothing -> T18140.D ww6_sNu wild1_X9 ww8_sNw ww9_sNx; Just ipv1_Xd -> case ww9_sNx of wild5_Xe { Nothing -> T18140.D ww6_sNu wild1_X9 ww8_sNw wild4_Xc; Just ipv2_sCL -> T18140.D ww6_sNu wild1_X9 ww8_sNw wild5_Xe } }; Just ipv1_Xc -> case ww8_sNw of wild4_Xd { Nothing -> case ww4_sNq of wild5_Xe { Nothing -> T18140.D ww6_sNu wild1_X9 wild3_Xb ww9_sNx; Just ipv2_Xf -> case ww9_sNx of wild6_Xg { Nothing -> T18140.D ww6_sNu wild1_X9 wild3_Xb wild5_Xe; Just ipv3_sCL -> T18140.D ww6_sNu wild1_X9 wild3_Xb wild6_Xg } }; Just ipv2_sCL -> case ww4_sNq of wild5_Xe { Nothing -> T18140.D ww6_sNu wild1_X9 wild4_Xd ww9_sNx; Just ipv3_Xf -> case ww9_sNx of wild6_Xg { Nothing -> T18140.D ww6_sNu wild1_X9 wild4_Xd wild5_Xe; Just ipv4_Xh -> T18140.D ww6_sNu wild1_X9 wild4_Xd wild6_Xg } } } }; Just ipv1_sCL -> case ww3_sNp of wild3_Xb { Nothing -> case ww4_sNq of wild4_Xc { Nothing -> ww5_sNt; Just ipv2_Xd -> case ww9_sNx of { Nothing -> T18140.D ww6_sNu wild2_Xa ww8_sNw wild4_Xc; Just ipv3_Xf -> ww5_sNt } }; Just ipv2_Xc -> case ww8_sNw of wild4_Xd { Nothing -> case ww4_sNq of wild5_Xe { Nothing -> T18140.D ww6_sNu wild2_Xa wild3_Xb ww9_sNx; Just ipv3_Xf -> case ww9_sNx of wild6_Xg { Nothing -> T18140.D ww6_sNu wild2_Xa wild3_Xb wild5_Xe; Just ipv4_Xh -> T18140.D ww6_sNu wild2_Xa wild3_Xb wild6_Xg } }; Just ipv3_Xe -> case ww4_sNq of wild5_Xf { Nothing -> ww5_sNt; Just ipv4_Xg -> case ww9_sNx of { Nothing -> T18140.D ww6_sNu wild2_Xa wild4_Xd wild5_Xf; Just ipv5_Xi -> ww5_sNt } } } } } }; Just ipv_sCI -> case ww6_sNu of wild1_X9 { Nothing -> case ww2_sNo of wild2_Xa { Nothing -> case ww3_sNp of wild3_Xb { Nothing -> case ww4_sNq of wild4_Xc { Nothing -> T18140.D wild_X8 ww7_sNv ww8_sNw ww9_sNx; Just ipv1_Xd -> case ww9_sNx of wild5_Xe { Nothing -> T18140.D wild_X8 ww7_sNv ww8_sNw wild4_Xc; Just ipv2_sCL -> T18140.D wild_X8 ww7_sNv ww8_sNw wild5_Xe } }; Just ipv1_Xc -> case ww8_sNw of wild4_Xd { Nothing -> case ww4_sNq of wild5_Xe { Nothing -> T18140.D wild_X8 ww7_sNv wild3_Xb ww9_sNx; Just ipv2_Xf -> case ww9_sNx of wild6_Xg { Nothing -> T18140.D wild_X8 ww7_sNv wild3_Xb wild5_Xe; Just ipv3_sCL -> T18140.D wild_X8 ww7_sNv wild3_Xb wild6_Xg } }; Just ipv2_sCL -> case ww4_sNq of wild5_Xe { Nothing -> T18140.D wild_X8 ww7_sNv wild4_Xd ww9_sNx; Just ipv3_Xf -> case ww9_sNx of wild6_Xg { Nothing -> T18140.D wild_X8 ww7_sNv wild4_Xd wild5_Xe; Just ipv4_Xh -> T18140.D wild_X8 ww7_sNv wild4_Xd wild6_Xg } } } }; Just ipv1_Xb -> case ww7_sNv of wild3_Xc { Nothing -> case ww3_sNp of wild4_Xd { Nothing -> case ww4_sNq of wild5_Xe { Nothing -> T18140.D wild_X8 wild2_Xa ww8_sNw ww9_sNx; Just ipv2_Xf -> case ww9_sNx of wild6_Xg { Nothing -> T18140.D wild_X8 wild2_Xa ww8_sNw wild5_Xe; Just ipv3_sCL -> T18140.D wild_X8 wild2_Xa ww8_sNw wild6_Xg } }; Just ipv2_Xe -> case ww8_sNw of wild5_Xf { Nothing -> case ww4_sNq of { Nothing -> T18140.D wild_X8 wild2_Xa wild4_Xd ww9_sNx; Just ipv3_Xh -> case ww9_sNx of wild7_Xi { Nothing -> ww_sNm; Just ipv4_sCL -> T18140.D wild_X8 wild2_Xa wild4_Xd wild7_Xi } }; Just ipv3_sCL -> case ww4_sNq of wild6_Xg { Nothing -> T18140.D wild_X8 wild2_Xa wild5_Xf ww9_sNx; Just ipv4_Xh -> case ww9_sNx of wild7_Xi { Nothing -> T18140.D wild_X8 wild2_Xa wild5_Xf wild6_Xg; Just ipv5_Xj -> T18140.D wild_X8 wild2_Xa wild5_Xf wild7_Xi } } } }; Just ipv2_sCL -> case ww3_sNp of wild4_Xd { Nothing -> case ww4_sNq of wild5_Xe { Nothing -> T18140.D wild_X8 wild3_Xc ww8_sNw ww9_sNx; Just ipv3_Xf -> case ww9_sNx of wild6_Xg { Nothing -> T18140.D wild_X8 wild3_Xc ww8_sNw wild5_Xe; Just ipv4_Xh -> T18140.D wild_X8 wild3_Xc ww8_sNw wild6_Xg } }; Just ipv3_Xe -> case ww8_sNw of wild5_Xf { Nothing -> case ww4_sNq of wild6_Xg { Nothing -> T18140.D wild_X8 wild3_Xc wild4_Xd ww9_sNx; Just ipv4_Xh -> case ww9_sNx of wild7_Xi { Nothing -> T18140.D wild_X8 wild3_Xc wild4_Xd wild6_Xg; Just ipv5_Xj -> T18140.D wild_X8 wild3_Xc wild4_Xd wild7_Xi } }; Just ipv4_Xg -> case ww4_sNq of wild6_Xh { Nothing -> T18140.D wild_X8 wild3_Xc wild5_Xf ww9_sNx; Just ipv5_Xi -> case ww9_sNx of wild7_Xj { Nothing -> T18140.D wild_X8 wild3_Xc wild5_Xf wild6_Xh; Just ipv6_Xk -> T18140.D wild_X8 wild3_Xc wild5_Xf wild7_Xj } } } } } }; Just ipv1_sCL -> case ww2_sNo of wild2_Xa { Nothing -> case ww3_sNp of wild3_Xb { Nothing -> case ww4_sNq of wild4_Xc { Nothing -> ww5_sNt; Just ipv2_Xd -> case ww9_sNx of { Nothing -> T18140.D wild1_X9 ww7_sNv ww8_sNw wild4_Xc; Just ipv3_Xf -> ww5_sNt } }; Just ipv2_Xc -> case ww8_sNw of wild4_Xd { Nothing -> case ww4_sNq of wild5_Xe { Nothing -> T18140.D wild1_X9 ww7_sNv wild3_Xb ww9_sNx; Just ipv3_Xf -> case ww9_sNx of wild6_Xg { Nothing -> T18140.D wild1_X9 ww7_sNv wild3_Xb wild5_Xe; Just ipv4_Xh -> T18140.D wild1_X9 ww7_sNv wild3_Xb wild6_Xg } }; Just ipv3_Xe -> case ww4_sNq of wild5_Xf { Nothing -> ww5_sNt; Just ipv4_Xg -> case ww9_sNx of { Nothing -> T18140.D wild1_X9 ww7_sNv wild4_Xd wild5_Xf; Just ipv5_Xi -> ww5_sNt } } } }; Just ipv2_Xb -> case ww7_sNv of wild3_Xc { Nothing -> case ww3_sNp of wild4_Xd { Nothing -> case ww4_sNq of wild5_Xe { Nothing -> T18140.D wild1_X9 wild2_Xa ww8_sNw ww9_sNx; Just ipv3_Xf -> case ww9_sNx of wild6_Xg { Nothing -> T18140.D wild1_X9 wild2_Xa ww8_sNw wild5_Xe; Just ipv4_Xh -> T18140.D wild1_X9 wild2_Xa ww8_sNw wild6_Xg } }; Just ipv3_Xe -> case ww8_sNw of wild5_Xf { Nothing -> case ww4_sNq of wild6_Xg { Nothing -> T18140.D wild1_X9 wild2_Xa wild4_Xd ww9_sNx; Just ipv4_Xh -> case ww9_sNx of wild7_Xi { Nothing -> T18140.D wild1_X9 wild2_Xa wild4_Xd wild6_Xg; Just ipv5_Xj -> T18140.D wild1_X9 wild2_Xa wild4_Xd wild7_Xi } }; Just ipv4_Xg -> case ww4_sNq of wild6_Xh { Nothing -> T18140.D wild1_X9 wild2_Xa wild5_Xf ww9_sNx; Just ipv5_Xi -> case ww9_sNx of wild7_Xj { Nothing -> T18140.D wild1_X9 wild2_Xa wild5_Xf wild6_Xh; Just ipv6_Xk -> T18140.D wild1_X9 wild2_Xa wild5_Xf wild7_Xj } } } }; Just ipv3_Xd -> case ww3_sNp of wild4_Xe { Nothing -> case ww4_sNq of wild5_Xf { Nothing -> ww5_sNt; Just ipv4_Xg -> case ww9_sNx of { Nothing -> T18140.D wild1_X9 wild3_Xc ww8_sNw wild5_Xf; Just ipv5_Xi -> ww5_sNt } }; Just ipv4_Xf -> case ww8_sNw of wild5_Xg { Nothing -> case ww4_sNq of wild6_Xh { Nothing -> T18140.D wild1_X9 wild3_Xc wild4_Xe ww9_sNx; Just ipv5_Xi -> case ww9_sNx of wild7_Xj { Nothing -> T18140.D wild1_X9 wild3_Xc wild4_Xe wild6_Xh; Just ipv6_Xk -> T18140.D wild1_X9 wild3_Xc wild4_Xe wild7_Xj } }; Just ipv5_Xh -> case ww4_sNq of wild6_Xi { Nothing -> ww5_sNt; Just ipv6_Xj -> case ww9_sNx of { Nothing -> T18140.D wild1_X9 wild3_Xc wild5_Xg wild6_Xi; Just ipv7_Xl -> ww5_sNt } } } } } } } } } }Edited by Sebastian Graf Ah. That commit b78cc64e is relatively recent:
Make constructor wrappers inline only during the final phase For case-of-known constructor to continue triggering early, exprIsConApp_maybe is now capable of looking through lets and cases. See #15840I have not yet investigated.
changed milestone to %9.0.1
mentioned in commit dc66d978
mentioned in merge request !3497 (closed)
See !3497 (closed) which fixes this ticket, I think.
Can someone check in the original setting?
mentioned in commit b3163d88
-
I can confirm that !3497 (closed) DOES seem to fix the problem here.
Edited by Artem Pelenitsyn marked this issue as related to #10421 (closed)
marked this issue as related to #13253 (closed)
mentioned in commit d83d8c83
mentioned in commit 212ce778
mentioned in commit a6247763
mentioned in commit 6b9eab2f
mentioned in commit df525f2b
mentioned in commit d6d70c7e
mentioned in commit 35bcb5fa
mentioned in commit f2506067
mentioned in commit 818aba30
mentioned in commit adcf83e2
mentioned in commit dfa4a337
mentioned in commit e8b04d89
mentioned in commit 9b1412f1
mentioned in commit df18a355
mentioned in commit 8d686760
mentioned in commit d738eb62
mentioned in commit 3f65f1e0
mentioned in commit ddb908e7
mentioned in commit 0bd60059
Fixed by !3497 (closed)
Test:
perf/compiler/T18140mentioned in commit 7f519481
mentioned in commit cf9a6c17
mentioned in issue #18923 (closed)
mentioned in merge request !4409 (closed)
marked this issue as related to #18923 (closed)
marked this issue as related to #19121 (closed)
mentioned in issue #19121 (closed)
mentioned in issue #19103
mentioned in commit ff59c315
mentioned in commit e3c244f5
marked this issue as related to #20246 (closed)
mentioned in issue #20246 (closed)
Please register or sign in to reply