compile-time memory-usage regression for DynFlags between GHC 8.0 and GHC 8.2

I notice that buildbots were having a hard-time building GHC 8.2 snapshots... so I investigated...

One thing that jumped at me was the memory usage while compiling DynFlags which is the worst memory-offender with 4777MiB of resident size:

"inplace/bin/ghc-stage1" -hisuf hi -osuf  o -hcsuf hc -static  -H32m -O -Wall   -Iincludes -Iincludes/dist -Iincludes/dist-derivedconstants/header -Iincludes/dist-ghcconstants/header   -this-unit-id ghc-8.2.0.201
70313 -hide-all-packages -i -icompiler/backpack -icompiler/basicTypes -icompiler/cmm -icompiler/codeGen -icompiler/coreSyn -icompiler/deSugar -icompiler/ghci -icompiler/hsSyn -icompiler/iface -icompiler/llvmGen -
icompiler/main -icompiler/nativeGen -icompiler/parser -icompiler/prelude -icompiler/profiling -icompiler/rename -icompiler/simplCore -icompiler/simplStg -icompiler/specialise -icompiler/stgSyn -icompiler/stranal 
-icompiler/typecheck -icompiler/types -icompiler/utils -icompiler/vectorise -icompiler/stage2/build -Icompiler/stage2/build -icompiler/stage2/build/./autogen -Icompiler/stage2/build/./autogen -Icompiler/. -Icompi
ler/parser -Icompiler/utils -Icompiler/../rts/dist/build -Icompiler/stage2   -optP-DGHCI -optP-include -optPcompiler/stage2/build/./autogen/cabal_macros.h -package-id base-4.10.0.0 -package-id deepseq-1.4.3.0 -pa
ckage-id directory-1.3.0.2 -package-id process-1.4.3.0 -package-id bytestring-0.10.8.2 -package-id binary-0.8.4.1 -package-id time-1.8.0.1 -package-id containers-0.5.10.2 -package-id array-0.5.1.2 -package-id fil
epath-1.4.1.1 -package-id template-haskell-2.12.0.0 -package-id hpc-0.6.0.3 -package-id transformers-0.5.2.0 -package-id ghc-boot-8.2.0.20170313 -package-id ghc-boot-th-8.2.0.20170313 -package-id ghci-8.2.0.20170
313 -package-id hoopl-3.10.2.2 -package-id unix-2.7.2.1 -package-id terminfo-0.4.0.2 -Wall -fno-warn-name-shadowing -this-unit-id ghc -XHaskell2010 -optc-DTHREADED_RTS -DGHCI_TABLES_NEXT_TO_CODE -DSTAGE=2 -Rghc-t
iming -O2  -no-user-package-db -rtsopts      -Wnoncanonical-monad-instances  -odir compiler/stage2/build -hidir compiler/stage2/build -stubdir compiler/stage2/build   -dynamic-too -c compiler/main/DynFlags.hs -o 
compiler/stage2/build/DynFlags.o -dyno compiler/stage2/build/DynFlags.dyn_o

<<ghc: 63014533832 bytes, 4032 GCs, 395131448/1973969928 avg/max bytes residency (24 samples), 4777M in use, 0.000 INIT (0.000 elapsed), 32.207 MUT (33.221 elapsed), 26.682 GC (26.685 elapsed) :ghc>>

Then I did two clean full builds of the ghc-8.0 and ghc-8.2 branch, and collected the -Rghc-timings output into simple 3-columns .csv files;

Here's are the respective top-12 entries for GHC 8.0 and GHC 8.2 respectively:

$ sort -t, -k2n build.80.csv | tail -n12
"inplace/bin/ghc-stage1",560,compiler/stage2/build/TcRnDriver.o
"inplace/bin/ghc-stage1",572,compiler/stage2/build/TcRnDriver.p_o
"inplace/bin/ghc-stage1",572,compiler/stage2/build/TcSplice.o
"inplace/bin/ghc-stage1",847,compiler/stage2/build/OptCoercion.p_o
"/opt/ghc/bin/ghc",979,compiler/stage1/build/OptCoercion.o
"/opt/ghc/bin/ghc",1036,compiler/stage1/build/DynFlags.o
"/opt/ghc/bin/ghc",1040,compiler/stage1/build/X86/CodeGen.o
"inplace/bin/ghc-stage1",1119,compiler/stage2/build/DynFlags.p_o
"inplace/bin/ghc-stage1",1123,compiler/stage2/build/DynFlags.o
"inplace/bin/ghc-stage1",1178,compiler/stage2/build/OptCoercion.o
"inplace/bin/ghc-stage1",1233,compiler/stage2/build/X86/CodeGen.o
"inplace/bin/ghc-stage1",1282,compiler/stage2/build/X86/CodeGen.p_o

$ sort -t, -k2n build.82.csv | tail -n12
"inplace/bin/ghc-stage1",891,compiler/stage2/build/CmmParse.p_o
"/opt/ghc/bin/ghc",1053,compiler/stage1/build/DynFlags.o
"/opt/ghc/bin/ghc",1058,compiler/stage1/build/OptCoercion.o
"inplace/bin/ghc-stage1",1284,compiler/stage2/build/HsExpr.p_o
"inplace/bin/ghc-stage1",1313,compiler/stage2/build/HsExpr.o
"/opt/ghc/bin/ghc",1394,compiler/stage1/build/X86/CodeGen.o
"inplace/bin/ghc-stage1",1495,compiler/stage2/build/X86/CodeGen.o
"inplace/bin/ghc-stage1",1605,compiler/stage2/build/X86/CodeGen.p_o
"inplace/bin/ghc-stage1",1620,compiler/stage2/build/OptCoercion.o
"inplace/bin/ghc-stage1",1733,compiler/stage2/build/OptCoercion.p_o
"inplace/bin/ghc-stage1",3989,compiler/stage2/build/DynFlags.p_o
"inplace/bin/ghc-stage1",4777,compiler/stage2/build/DynFlags.o

What's apparent from this data:

• GHC stage0 (which was GHC 8.0.2) is able to build DynFlags.o for GHC 8.0 and GHC 8.2 w/ roughtly the same amount of max resident memory, i.e. 1 GiB

• **GHC 8.2 stage1 however needs 4 times the amount of memory more than GHC 8.0 to build the same DynFlags.hs module!**

Trac metadata

Trac field	Value
Version	8.1
Type	Bug
TypeOfFailure	OtherFailure
Priority	high
Resolution	Unresolved
Component	Compiler
Test case
Differential revisions
BlockedBy
Related
Blocking
CC
Operating system
Architecture