Starting with 9.2, GHC OOMs while building the propellor package
$ git clone -b ghc-oom https://github.com/sjakobi/propellor.branchable.com
$ cd propellor.branchable.com
$ cabal build -w ghc-9.4.1 --allow-newer=hashable:ghc-bignum,base
<...>
[160 of 180] Compiling Propellor.Property.SiteSpecific.JoeySites ( src/Propellor/Property/SiteSpecific/JoeySites.hs, /home/simon/tmp/propellor-5.13/dist-newstyle/build/x86_64-linux/ghc-9.4.1/propellor-5.13/build/Propellor/Property/SiteSpecific/JoeySites.o, /home/simon/tmp/propellor-5.13/dist-newstyle/build/x86_64-linux/ghc-9.4.1/propellor-5.13/build/Propellor/Property/SiteSpecific/JoeySites.dyn_o )
Error: cabal: Failed to build propellor-5.13 (which is required by
exe:propellor-config from propellor-5.13 and exe:propellor from
propellor-5.13). The build process was killed (i.e. SIGKILL). The typical
reason for this is that there is not enough memory available (e.g. the OS
killed a process using lots of memory).On my machine GHC takes more than 15GB of memory before being killed. With GHC 8.6.5 and 9.0.2 the memory usage is unremarkable and the build succeeds.
This is the problematic module: https://github.com/sjakobi/propellor.branchable.com/blob/d0dd9f43d5edbefcaf995920de5ee02074fc1169/src/Propellor/Property/SiteSpecific/JoeySites.hs
(CC @trac-joeyhess)
Activity
added needs triage label
According to
-ddump-timings, the problem seems to be somewhere in the renamer/typechecker phase:[160 of 179] Compiling Propellor.Property.SiteSpecific.JoeySites ( src/Propellor/Property/SiteSpecific/JoeySites.hs, /home/simon/src/propellor.branchable.com/dist-newstyle/build/x86_64-linux/ghc-9.4.1/propellor-5.13/build/Propellor/Property/SiteSpecific/JoeySites.o, /home/simon/src/propellor.branchable.com/dist-newstyle/build/x86_64-linux/ghc-9.4.1/propellor-5.13/build/Propellor/Property/SiteSpecific/JoeySites.dyn_o ) *** Parser [Propellor.Property.SiteSpecific.JoeySites]: Parser [Propellor.Property.SiteSpecific.JoeySites]: alloc=28331504 time=12.184 *** Renamer/typechecker [Propellor.Property.SiteSpecific.JoeySites]: Error: cabal: Failed to build propellor-5.13 (which is required by <...>I'll try to minimize the issue a bit.
I have somewhat reduced the size of the package on my
ghc-oom-reducedbranch (link to the reduced problematic module). At this stage the renamer/typechecker phase takes only around 5s, so it's still noticeably slow. But the majority of the compile time is now used by the simplifier, which takes around 22s:[86 of 86] Compiling Propellor.Property.SiteSpecific.JoeySites ( src/Propellor/Property/SiteSpecific/JoeySites.hs, /home/simon/src/propellor.branchable.com/dist-newstyle/build/x86_64-linux/ghc-9.4.1/propellor-5.13/build/Propellor/Property/SiteSpecific/JoeySites.o, /home/simon/src/propellor.branchable.com/dist-newstyle/build/x86_64-linux/ghc-9.4.1/propellor-5.13/build/Propellor/Property/SiteSpecific/JoeySites.dyn_o ) *** Parser [Propellor.Property.SiteSpecific.JoeySites]: Parser [Propellor.Property.SiteSpecific.JoeySites]: alloc=3219952 time=1.582 *** Renamer/typechecker [Propellor.Property.SiteSpecific.JoeySites]: Renamer/typechecker [Propellor.Property.SiteSpecific.JoeySites]: alloc=3548793864 time=5841.314 *** Desugar [Propellor.Property.SiteSpecific.JoeySites]: Desugar [Propellor.Property.SiteSpecific.JoeySites]: alloc=3520960 time=1.665 *** Simplifier [Propellor.Property.SiteSpecific.JoeySites]: Simplifier [Propellor.Property.SiteSpecific.JoeySites]: alloc=31011112544 time=22223.230 *** CoreTidy [Propellor.Property.SiteSpecific.JoeySites]: CoreTidy [Propellor.Property.SiteSpecific.JoeySites]: alloc=1435552648 time=4062.040 *** CorePrep [Propellor.Property.SiteSpecific.JoeySites]: CorePrep [Propellor.Property.SiteSpecific.JoeySites]: alloc=712896 time=10.836 *** CoreToStg [Propellor.Property.SiteSpecific.JoeySites]: CoreToStg [Propellor.Property.SiteSpecific.JoeySites]: alloc=1513448 time=432.639 *** CodeGen [Propellor.Property.SiteSpecific.JoeySites]: CodeGen [Propellor.Property.SiteSpecific.JoeySites]: alloc=70487328 time=191.616 *** WriteIface [/home/simon/src/propellor.branchable.com/dist-newstyle/build/x86_64-linux/ghc-9.4.1/propellor-5.13/build/Propellor/Property/SiteSpecific/JoeySites.hi]: WriteIface [/home/simon/src/propellor.branchable.com/dist-newstyle/build/x86_64-linux/ghc-9.4.1/propellor-5.13/build/Propellor/Property/SiteSpecific/JoeySites.hi]: alloc=1348968 time=0.207 *** systool:as: systool:as: alloc=120064 time=0.312 *** CorePrep [Propellor.Property.SiteSpecific.JoeySites]: CorePrep [Propellor.Property.SiteSpecific.JoeySites]: alloc=713048 time=27.245 *** CoreToStg [Propellor.Property.SiteSpecific.JoeySites]: CoreToStg [Propellor.Property.SiteSpecific.JoeySites]: alloc=1489256 time=427.179 *** CodeGen [Propellor.Property.SiteSpecific.JoeySites]: CodeGen [Propellor.Property.SiteSpecific.JoeySites]: alloc=78063304 time=34.922 *** WriteIface [/home/simon/src/propellor.branchable.com/dist-newstyle/build/x86_64-linux/ghc-9.4.1/propellor-5.13/build/Propellor/Property/SiteSpecific/JoeySites.dyn_hi]: WriteIface [/home/simon/src/propellor.branchable.com/dist-newstyle/build/x86_64-linux/ghc-9.4.1/propellor-5.13/build/Propellor/Property/SiteSpecific/JoeySites.dyn_hi]: alloc=1349936 time=0.227 *** systool:as: systool:as: alloc=111504 time=0.315Here is the
-dshow-passesoutput:*** Parser [Propellor.Property.SiteSpecific.JoeySites]: !!! Parser [Propellor.Property.SiteSpecific.JoeySites]: finished in 5.31 milliseconds, allocated 3.072 megabytes *** Renamer/typechecker [Propellor.Property.SiteSpecific.JoeySites]: !!! Renamer/typechecker [Propellor.Property.SiteSpecific.JoeySites]: finished in 5435.71 milliseconds, allocated 3402.887 megabytes *** Desugar [Propellor.Property.SiteSpecific.JoeySites]: Result size of Desugar (before optimization) = {terms: 828, types: 2,376,049, coercions: 27,748,196, joins: 0/162} Result size of Desugar (after optimization) = {terms: 592, types: 1,188,888, coercions: 30,474,872, joins: 0/55} !!! Desugar [Propellor.Property.SiteSpecific.JoeySites]: finished in 10541.98 milliseconds, allocated 11407.044 megabytes *** Simplifier [Propellor.Property.SiteSpecific.JoeySites]: Result size of Simplifier iteration=1 = {terms: 636, types: 1,188,947, coercions: 30,474,805, joins: 0/55} Result size of Simplifier = {terms: 636, types: 1,188,947, coercions: 30,474,805, joins: 0/55} !!! Simplifier [Propellor.Property.SiteSpecific.JoeySites]: finished in 18226.95 milliseconds, allocated 18469.317 megabytes *** CoreTidy [Propellor.Property.SiteSpecific.JoeySites]: !!! CoreTidy [Propellor.Property.SiteSpecific.JoeySites]: finished in 1514.89 milliseconds, allocated 1369.008 megabytes Result size of Tidy Core = {terms: 636, types: 1,188,947, coercions: 30,474,805, joins: 0/55} *** CorePrep [Propellor.Property.SiteSpecific.JoeySites]: Result size of CorePrep = {terms: 1,102, types: 2,375,885, coercions: 30,474,805, joins: 0/287} !!! CorePrep [Propellor.Property.SiteSpecific.JoeySites]: finished in 689.98 milliseconds, allocated 2.406 megabytes *** CoreToStg [Propellor.Property.SiteSpecific.JoeySites]: *** Stg2Stg: !!! CoreToStg [Propellor.Property.SiteSpecific.JoeySites]: finished in 0.54 milliseconds, allocated 0.708 megabytes *** CodeGen [Propellor.Property.SiteSpecific.JoeySites]: !!! CodeGen [Propellor.Property.SiteSpecific.JoeySites]: finished in 1296.84 milliseconds, allocated 67.466 megabytes *** WriteIface [/home/simon/src/propellor.branchable.com/dist-newstyle/build/x86_64-linux/ghc-9.4.1/propellor-5.13/build/Propellor/Property/SiteSpecific/JoeySites.hi]: !!! WriteIface [/home/simon/src/propellor.branchable.com/dist-newstyle/build/x86_64-linux/ghc-9.4.1/propellor-5.13/build/Propellor/Property/SiteSpecific/JoeySites.hi]: finished in 0.21 milliseconds, allocated 1.286 megabytes *** systool:as: *** Assembler: !!! systool:as: finished in 0.42 milliseconds, allocated 0.125 megabytes *** CorePrep [Propellor.Property.SiteSpecific.JoeySites]: Result size of CorePrep = {terms: 1,102, types: 2,375,885, coercions: 30,474,805, joins: 0/287} !!! CorePrep [Propellor.Property.SiteSpecific.JoeySites]: finished in 973.53 milliseconds, allocated 1.500 megabytes *** CoreToStg [Propellor.Property.SiteSpecific.JoeySites]: *** Stg2Stg: !!! CoreToStg [Propellor.Property.SiteSpecific.JoeySites]: finished in 0.40 milliseconds, allocated 0.711 megabytes *** CodeGen [Propellor.Property.SiteSpecific.JoeySites]: !!! CodeGen [Propellor.Property.SiteSpecific.JoeySites]: finished in 34.14 milliseconds, allocated 74.305 megabytes *** WriteIface [/home/simon/src/propellor.branchable.com/dist-newstyle/build/x86_64-linux/ghc-9.4.1/propellor-5.13/build/Propellor/Property/SiteSpecific/JoeySites.dyn_hi]: !!! WriteIface [/home/simon/src/propellor.branchable.com/dist-newstyle/build/x86_64-linux/ghc-9.4.1/propellor-5.13/build/Propellor/Property/SiteSpecific/JoeySites.dyn_hi]: finished in 0.24 milliseconds, allocated 1.287 megabytes *** systool:as: *** Assembler: !!! systool:as: finished in 0.30 milliseconds, allocated 0.116 megabytes
Seems like we have many coercions so cc @sheaf
-dshow-passesoutput for the reduced module:*** Parser [Propellor.Property.SiteSpecific.JoeySites]: !!! Parser [Propellor.Property.SiteSpecific.JoeySites]: finished in 0.97 milliseconds, allocated 3.073 megabytes *** Renamer/typechecker [Propellor.Property.SiteSpecific.JoeySites]: !!! Renamer/typechecker [Propellor.Property.SiteSpecific.JoeySites]: finished in 5377.99 milliseconds, allocated 3402.887 megabytes *** Desugar [Propellor.Property.SiteSpecific.JoeySites]: Result size of Desugar (before optimization) = {terms: 828, types: 2,376,049, coercions: 27,748,196, joins: 0/162} Result size of Desugar (after optimization) = {terms: 592, types: 1,188,888, coercions: 27,748,206, joins: 0/55} !!! Desugar [Propellor.Property.SiteSpecific.JoeySites]: finished in 2741.83 milliseconds, allocated 339.682 megabytes *** Simplifier [Propellor.Property.SiteSpecific.JoeySites]: Result size of Simplifier iteration=1 = {terms: 636, types: 1,188,947, coercions: 27,748,215, joins: 0/55} Result size of Simplifier = {terms: 636, types: 1,188,947, coercions: 27,748,215, joins: 0/55} !!! Simplifier [Propellor.Property.SiteSpecific.JoeySites]: finished in 3016.88 milliseconds, allocated 5.156 megabytes *** CoreTidy [Propellor.Property.SiteSpecific.JoeySites]: !!! CoreTidy [Propellor.Property.SiteSpecific.JoeySites]: finished in 3819.53 milliseconds, allocated 1327.422 megabytes Result size of Tidy Core = {terms: 636, types: 1,188,947, coercions: 27,748,215, joins: 0/55} *** CorePrep [Propellor.Property.SiteSpecific.JoeySites]: Result size of CorePrep = {terms: 1,102, types: 2,375,885, coercions: 27,748,215, joins: 0/287} !!! CorePrep [Propellor.Property.SiteSpecific.JoeySites]: finished in 929.88 milliseconds, allocated 2.410 megabytes *** CoreToStg [Propellor.Property.SiteSpecific.JoeySites]: *** Stg2Stg: !!! CoreToStg [Propellor.Property.SiteSpecific.JoeySites]: finished in 0.46 milliseconds, allocated 0.708 megabytes *** CodeGen [Propellor.Property.SiteSpecific.JoeySites]: !!! CodeGen [Propellor.Property.SiteSpecific.JoeySites]: finished in 51.50 milliseconds, allocated 67.463 megabytes *** WriteIface [/home/simon/src/propellor.branchable.com/dist-newstyle/build/x86_64-linux/ghc-9.4.1/propellor-5.13/build/Propellor/Property/SiteSpecific/JoeySites.hi]: !!! WriteIface [/home/simon/src/propellor.branchable.com/dist-newstyle/build/x86_64-linux/ghc-9.4.1/propellor-5.13/build/Propellor/Property/SiteSpecific/JoeySites.hi]: finished in 0.24 milliseconds, allocated 1.286 megabytes *** systool:as: *** Assembler: !!! systool:as: finished in 0.42 milliseconds, allocated 0.125 megabytes *** CorePrep [Propellor.Property.SiteSpecific.JoeySites]: Result size of CorePrep = {terms: 1,102, types: 2,375,885, coercions: 27,748,215, joins: 0/287} !!! CorePrep [Propellor.Property.SiteSpecific.JoeySites]: finished in 935.35 milliseconds, allocated 1.500 megabytes *** CoreToStg [Propellor.Property.SiteSpecific.JoeySites]: *** Stg2Stg: !!! CoreToStg [Propellor.Property.SiteSpecific.JoeySites]: finished in 0.41 milliseconds, allocated 0.710 megabytes *** CodeGen [Propellor.Property.SiteSpecific.JoeySites]: !!! CodeGen [Propellor.Property.SiteSpecific.JoeySites]: finished in 36.67 milliseconds, allocated 74.302 megabytes *** WriteIface [/home/simon/src/propellor.branchable.com/dist-newstyle/build/x86_64-linux/ghc-9.4.1/propellor-5.13/build/Propellor/Property/SiteSpecific/JoeySites.dyn_hi]: !!! WriteIface [/home/simon/src/propellor.branchable.com/dist-newstyle/build/x86_64-linux/ghc-9.4.1/propellor-5.13/build/Propellor/Property/SiteSpecific/JoeySites.dyn_hi]: finished in 0.25 milliseconds, allocated 1.287 megabytes *** systool:as: *** Assembler: !!! systool:as: finished in 0.33 milliseconds, allocated 0.116 megabytesThe original reproducer still OOMs during renaming/typechecking.
I don't have much insight to offer here, but one problem propellor has often had is that when the type checker does have an error to report, the error message is often so enormous that it OOMS ghc when ghc tries to display it. So, if the new version of ghc is somehow finding a type error that previous versions do not, that seems like a possible reason.
(See http://propellor.branchable.com/todo/avoid_ghc_memory_blow_up_for_huge_metatypes_error_messages/ for one case of this that I worked around by using type-errors.)
Here is a diff for the
-dshow-passesoutput with GHC 9.0.2 vs. 9.4.1:--- a/show-passes-9.0.2-ghc-oom-reduced.log +++ b/show-passes-9.4.1-ghc-oom-reduced.log @@ -1,40 +1,69 @@ *** Parser [Propellor.Property.SiteSpecific.JoeySites]: -!!! Parser [Propellor.Property.SiteSpecific.JoeySites]: finished in 2.37 milliseconds, allocated 3.935 megabytes +!!! Parser [Propellor.Property.SiteSpecific.JoeySites]: finished in 5.31 milliseconds, allocated 3.072 megabytes *** Renamer/typechecker [Propellor.Property.SiteSpecific.JoeySites]: -!!! Renamer/typechecker [Propellor.Property.SiteSpecific.JoeySites]: finished in 191.22 milliseconds, allocated 77.307 megabytes +!!! Renamer/typechecker [Propellor.Property.SiteSpecific.JoeySites]: finished in 5435.71 milliseconds, allocated 3402.887 megabytes *** Desugar [Propellor.Property.SiteSpecific.JoeySites]: Result size of Desugar (before optimization) - = {terms: 832, types: 4,557, coercions: 115,324, joins: 0/164} + = {terms: 828, + types: 2,376,049, + coercions: 27,748,196, + joins: 0/162} Result size of Desugar (after optimization) - = {terms: 598, types: 3,064, coercions: 109,002, joins: 0/58} -!!! Desugar [Propellor.Property.SiteSpecific.JoeySites]: finished in 70.20 milliseconds, allocated 64.550 megabytes + = {terms: 592, + types: 1,188,888, + coercions: 30,474,872, + joins: 0/55} +!!! Desugar [Propellor.Property.SiteSpecific.JoeySites]: finished in 10541.98 milliseconds, allocated 11407.044 megabytes *** Simplifier [Propellor.Property.SiteSpecific.JoeySites]: Result size of Simplifier iteration=1 - = {terms: 637, types: 3,093, coercions: 108,936, joins: 0/55} + = {terms: 636, + types: 1,188,947, + coercions: 30,474,805, + joins: 0/55} Result size of Simplifier - = {terms: 637, types: 3,093, coercions: 108,936, joins: 0/55} -!!! Simplifier [Propellor.Property.SiteSpecific.JoeySites]: finished in 106.92 milliseconds, allocated 102.678 megabytes + = {terms: 636, + types: 1,188,947, + coercions: 30,474,805, + joins: 0/55} +!!! Simplifier [Propellor.Property.SiteSpecific.JoeySites]: finished in 18226.95 milliseconds, allocated 18469.317 megabytes *** CoreTidy [Propellor.Property.SiteSpecific.JoeySites]: +!!! CoreTidy [Propellor.Property.SiteSpecific.JoeySites]: finished in 1514.89 milliseconds, allocated 1369.008 megabytes Result size of Tidy Core - = {terms: 637, types: 3,093, coercions: 108,936, joins: 0/55} -!!! CoreTidy [Propellor.Property.SiteSpecific.JoeySites]: finished in 10.00 milliseconds, allocated 13.229 megabytes + = {terms: 636, + types: 1,188,947, + coercions: 30,474,805, + joins: 0/55} *** CorePrep [Propellor.Property.SiteSpecific.JoeySites]: Result size of CorePrep - = {terms: 1,101, types: 4,226, coercions: 108,936, joins: 0/286} -!!! CorePrep [Propellor.Property.SiteSpecific.JoeySites]: finished in 9.07 milliseconds, allocated 1.890 megabytes + = {terms: 1,102, + types: 2,375,885, + coercions: 30,474,805, + joins: 0/287} +!!! CorePrep [Propellor.Property.SiteSpecific.JoeySites]: finished in 689.98 milliseconds, allocated 2.406 megabytes +*** CoreToStg [Propellor.Property.SiteSpecific.JoeySites]: *** Stg2Stg: +!!! CoreToStg [Propellor.Property.SiteSpecific.JoeySites]: finished in 0.54 milliseconds, allocated 0.708 megabytes *** CodeGen [Propellor.Property.SiteSpecific.JoeySites]: -!!! CodeGen [Propellor.Property.SiteSpecific.JoeySites]: finished in 40.35 milliseconds, allocated 75.520 megabytes +!!! CodeGen [Propellor.Property.SiteSpecific.JoeySites]: finished in 1296.84 milliseconds, allocated 67.466 megabytes +*** WriteIface [/home/simon/src/propellor.branchable.com/dist-newstyle/build/x86_64-linux/ghc-9.4.1/propellor-5.13/build/Propellor/Property/SiteSpecific/JoeySites.hi]: +!!! WriteIface [/home/simon/src/propellor.branchable.com/dist-newstyle/build/x86_64-linux/ghc-9.4.1/propellor-5.13/build/Propellor/Property/SiteSpecific/JoeySites.hi]: finished in 0.21 milliseconds, allocated 1.286 megabytes *** systool:as: *** Assembler: -!!! systool:as: finished in 0.53 milliseconds, allocated 0.136 megabytes +!!! systool:as: finished in 0.42 milliseconds, allocated 0.125 megabytes *** CorePrep [Propellor.Property.SiteSpecific.JoeySites]: Result size of CorePrep - = {terms: 1,101, types: 4,226, coercions: 108,936, joins: 0/286} -!!! CorePrep [Propellor.Property.SiteSpecific.JoeySites]: finished in 5.73 milliseconds, allocated 1.893 megabytes + = {terms: 1,102, + types: 2,375,885, + coercions: 30,474,805, + joins: 0/287} +!!! CorePrep [Propellor.Property.SiteSpecific.JoeySites]: finished in 973.53 milliseconds, allocated 1.500 megabytes +*** CoreToStg [Propellor.Property.SiteSpecific.JoeySites]: *** Stg2Stg: +!!! CoreToStg [Propellor.Property.SiteSpecific.JoeySites]: finished in 0.40 milliseconds, allocated 0.711 megabytes *** CodeGen [Propellor.Property.SiteSpecific.JoeySites]: -!!! CodeGen [Propellor.Property.SiteSpecific.JoeySites]: finished in 42.79 milliseconds, allocated 83.642 megabytes +!!! CodeGen [Propellor.Property.SiteSpecific.JoeySites]: finished in 34.14 milliseconds, allocated 74.305 megabytes +*** WriteIface [/home/simon/src/propellor.branchable.com/dist-newstyle/build/x86_64-linux/ghc-9.4.1/propellor-5.13/build/Propellor/Property/SiteSpecific/JoeySites.dyn_hi]: +!!! WriteIface [/home/simon/src/propellor.branchable.com/dist-newstyle/build/x86_64-linux/ghc-9.4.1/propellor-5.13/build/Propellor/Property/SiteSpecific/JoeySites.dyn_hi]: finished in 0.24 milliseconds, allocated 1.287 megabytes *** systool:as: *** Assembler: -!!! systool:as: finished in 0.48 milliseconds, allocated 0.128 megabytes +!!! systool:as: finished in 0.30 milliseconds, allocated 0.116 megabytesIt's remarkable how, already in the unoptimized desugarer output, types are inflated by roughly 520x and coercions by 240x.
The
propellorlibrary (and the reduced reproducer) already usesHaskell2010as thedefault-language, with onlyTypeOperatorsin thedefault-extensions. EnablingLANGUAGE Haskell2010explicitly in the problematicPropellor.Property.SiteSpecific.JoeySitesmodule doesn't change anything about the size of the generated Core.
-
Please can you diagnose this @sheaf ?
added Coercions Phigh Tbug compiler perf labels and removed needs triage label
assigned to @sheaf
Here's a standalone reproducer:
--{-# OPTIONS_GHC -ddump-timings #-} {-# LANGUAGE Haskell2010 #-} {-# LANGUAGE DataKinds #-} {-# LANGUAGE FlexibleContexts #-} {-# LANGUAGE GADTs #-} {-# LANGUAGE PolyKinds #-} {-# LANGUAGE StandaloneKindSignatures #-} {-# LANGUAGE TypeFamilies #-} {-# LANGUAGE TypeOperators #-} {-# LANGUAGE UndecidableInstances #-} module JoeySites ( house ) where import Data.Kind ( Type ) import Data.Type.Bool ( type (&&), type (||), If, Not ) data family Sing (x :: k) type MetaTypes = Sing data TargetOS = OSDebian | OSBuntish | OSArchLinux | OSFreeBSD data MetaType = Targeting TargetOS | WithInfo type UnixLike = MetaTypes '[ 'Targeting 'OSDebian , 'Targeting 'OSBuntish , 'Targeting 'OSArchLinux , 'Targeting 'OSFreeBSD ] type Linux = MetaTypes '[ 'Targeting 'OSDebian , 'Targeting 'OSBuntish , 'Targeting 'OSArchLinux ] type DebianLike = MetaTypes '[ 'Targeting 'OSDebian, 'Targeting 'OSBuntish ] type ArchLinux = MetaTypes '[ 'Targeting 'OSArchLinux ] type HasInfo = MetaTypes '[ 'WithInfo ] type (+) :: Type -> Type -> Type type family a + b where MetaTypes a + MetaTypes b = MetaTypes (Concat a b) type Concat :: [a] -> [a] -> [a] type family Concat l1 l2 where Concat '[] bs = bs Concat (a ': as) bs = a ': (Concat as bs) type IsTarget :: k -> Bool type family IsTarget a where IsTarget ('Targeting a) = 'True IsTarget 'WithInfo = 'False type Targets :: [k] -> [k] type family Targets l where Targets '[] = '[] Targets (x ': xs) = If (IsTarget x) (x ': Targets xs) (Targets xs) type NonTargets :: [k] -> [k] type family NonTargets l where NonTargets '[] = '[] NonTargets (x ': xs) = If (IsTarget x) (NonTargets xs) (x ': NonTargets xs) type Combine :: [a] -> [a] -> [a] type family Combine l1 l2 where Combine (list1 :: [a]) (list2 :: [a]) = (Concat (NonTargets list1 `Union` NonTargets list2) (Targets list1 `Intersect` Targets list2) ) type Elem :: t -> [t] -> Bool type family Elem a list where Elem a '[] = 'False Elem a (a ': bs) = 'True Elem a (_ ': bs) = Elem a bs type Union :: [a] -> [a] -> [a] type family Union l1 l2 where Union '[] list2 = list2 Union (a ': rest) list2 = If (Elem a list2 || Elem a rest) (Union rest list2) (a ': Union rest list2) type Intersect :: [a] -> [a] -> [a] type family Intersect l1 l2 where Intersect '[] list2 = '[] Intersect (a ': rest) list2 = If (Elem a list2 && Not (Elem a rest)) (a ': Intersect rest list2) (Intersect rest list2) data Property metatypes = Property data Props metatypes = Props [()] type family GetMetaTypes x where GetMetaTypes (Property (MetaTypes t)) = MetaTypes t propertyList :: Props (MetaTypes metatypes) -> Property (MetaTypes metatypes) propertyList = undefined props :: Props UnixLike props = Props [] (&) :: ( MetaTypes y ~ GetMetaTypes p ) => Props (MetaTypes x) -> p -> Props (MetaTypes (Combine x y)) (&) = undefined house :: Property (HasInfo + DebianLike) house = propertyList $ props & bad1 & bad1 & bad1 & bad2 & bad2 & bad2 & bad2 & bad2 & bad2 & bad2 & bad2 & bad2 & bad2 & bad2 & bad2 & bad3 where bad1 :: Property (DebianLike + ArchLinux) bad1 = undefined bad2 :: Property DebianLike bad2 = undefined bad3 :: Property (HasInfo + UnixLike) bad3 = undefinedThanks for the reproducer.
I think the compile time regression is to do with increased residency.
Here is HEAD
23,034,675,928 bytes allocated in the heap 14,777,558,520 bytes copied during GC 2,094,319,728 bytes maximum residency (18 sample(s)) 8,337,296 bytes maximum slop 4075 MiB total memory in use (0 MB lost due to fragmentation) INIT time 0.002s ( 0.006s elapsed) MUT time 28.491s ( 28.440s elapsed) GC time 52.440s ( 52.626s elapsed) EXIT time 0.001s ( 0.009s elapsed) Total time 80.933s ( 81.080s elapsed) Alloc rate 808,477,384 bytes per MUT secondHere is GHC 9.2:
*** Deleting temp dirs: 27,408,935,912 bytes allocated in the heap 1,304,456 bytes copied during GC 2,305,006,472 bytes maximum residency (22 sample(s)) 9,139,320 bytes maximum slop 4485 MiB total memory in use (0 MB lost due to fragmentation) INIT time 0.001s ( 0.000s elapsed) MUT time 17.672s ( 17.620s elapsed) GC time 18.024s ( 18.029s elapsed) EXIT time 0.001s ( 0.001s elapsed) Total time 35.698s ( 35.650s elapsed)GHC 9.0 does this
109,873,640 bytes allocated in the heap 44,683,152 bytes copied during GC 10,008,712 bytes maximum residency (6 sample(s)) 128,888 bytes maximum slop 24 MiB total memory in use (0 MB lost due to fragmentation) Tot time (elapsed) Avg pause Max pause Gen 0 44 colls, 0 par 0.023s 0.023s 0.0005s 0.0023s Gen 1 6 colls, 0 par 0.081s 0.199s 0.0331s 0.1465s TASKS: 4 (1 bound, 3 peak workers (3 total), using -N1) SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled) INIT time 0.001s ( 0.004s elapsed) MUT time 0.058s ( 0.080s elapsed) GC time 0.104s ( 0.222s elapsed) EXIT time 0.001s ( 0.004s elapsed) Total time 0.165s ( 0.310s elapsed) Alloc rate 1,880,914,833 bytes per MUT second Productivity 35.5% of total user, 25.8% of total elapsedSuper fast.
Looking at
-dshow-passesI see this:ghc-9.0
!!! Renamer/typechecker [JoeySites]: finished in 63.27 milliseconds, allocated 43.489 megabytes *** Desugar [JoeySites]: Result size of Desugar (before optimization) = {terms: 438, types: 2,262, coercions: 2,089, joins: 0/7}HEAD
!!! Renamer/typechecker [JoeySites]: finished in 40181.93 milliseconds, allocated 18980.365 megabytes *** Desugar [JoeySites]: Result size of Desugar (before optimization) = {terms: 430, types: 16,122,875, coercions: 32,243,474, joins: 0/7}That's quite a difference!
changed milestone to %9.4.3
changed milestone to %9.2.5
@sjakobi, I do indeed hope that we can address this for 9.6. It would be great if someone could take @sheaf's minimized reproducer and try running a bisection between 9.0 and 9.2.
One thing that should be noted is that the source program was arguably always somewhat fragile in that it uses suboptimal type-level data structures for its use-case. Specifically, the existence of
ElemandUnionis surely a sign that a type-level set would be more appropriate. Given the under-defined nature of GHC's typechecker behavior, it's not terribly surprising (but still quite unfortunate) that this program manifested in a high compile-time heap residency. Refactoring the package to use a more appropriate type-level data structure (e.g. the structure provided by thetype-level-setspackage) may be a sufficient workaround.Edited by Ben Gamari
Please register or sign in to reply