Compile time regression in OpenGLRaw

As reported by Paolo Veronelli:

Hello, I'm using ghc 7.10.1 to compile OpenGLRaw which is now impossible with -O1 and -O2 due to "ghc : out of memory error" on a 4GB linux host. The file making memory explode is Graphics.Rendering.OpenGL.Raw.Functions. With -O0 it uses 600 MB.

The file is really huge, but I could compile it with prior versions of ghc.

I've had a look and indeed the file takes way more memory to compile with HEAD. Looking at the heap profiles, it seems that the problem is in GHC itself (space leak somewhere?). I'll attach the heap profiles (from compiling the single module itself, i.e., touching the file and running prof GHC on it).

Paolo also says: ere is another file , which is small, which cannot be compiled within 4GB memory.

https://raw.githubusercontent.com/benl23x5/gloss/master/gloss-examples/raster/Fluid/src-repa/Stage/Linear.hs

changed weight to 5

added Tbug Trac import labels

Attached file module-hc.png (download).

profiled with "-RTS -hc -p"

Attached file module-hd.png (download).

profiled with "-RTS -hd -p"

Attached file module-hy.png (download).

profiled with "-RTS -hy -p"

I've had another quick look at this and compiled profiled GHC with addition of {-# OPTIONS_GHC -fprof-auto #-} to simplCore/Simplify.hs. The results are interesting:

	Sun May  3 17:33 2015 Time and Allocation Profiling Report  (Final)

	   ghc +RTS -hc -L100 -p -RTS [...] Graphics.Rendering.OpenGL.Raw.Functions

	total time  =       42.38 secs   (42385 ticks @ 1000 us, 1 processor)
	total alloc = 65,222,063,200 bytes  (excludes profiling overheads)

COST CENTRE               MODULE       %time %alloc

tc_rn_src_decls           TcRnDriver    15.1   10.4
completeCall.(...)        Simplify       9.5   22.0
simplCast.add_coerce.arg' Simplify       9.1   21.8
pprNativeCode             AsmCodeGen     4.2    3.2
Parser                    HscMain        3.3    4.2
RegAlloc                  AsmCodeGen     3.0    2.7
regLiveness               AsmCodeGen     2.5    2.0
simplLam                  Simplify       2.5    1.3
OccAnal                   SimplCore      2.5    1.8
StgCmm                    HscMain        2.3    1.2
CallArity                 SimplCore      2.2    3.4
FloatOutwards             SimplCore      1.7    1.1
sink                      CmmPipeline    1.7    1.1
simplAlts                 Simplify       1.5    0.8
genMachCode               AsmCodeGen     1.5    0.9
deSugar                   HscMain        1.5    1.3
simplAlt                  Simplify       1.4    0.7
simplUnfolding            Simplify       1.3    0.8
layoutStack               CmmPipeline    1.2    0.9
completeCall.info         Simplify       1.2    0.6
completeBind              Simplify       1.2    0.6
NativeCodeGen             CodeOutput     1.2    0.8
rebuildCall               Simplify       1.1    1.0
Simplify                  SimplCore      1.1    0.0

Looking at git blame the Simplify.simplCast didn't change much recently... Next week I'll try to have a look at Core generated by 7.8.4 and HEAD (maybe some stage before simplifier generates something weird?).

I created a synthetic test case that exhibits the same space blow-up:

modu = unlines $
       "module Out where" :
       "import Control.Monad (forever)" :
       [ var ++ " :: IO (); " ++ var ++ " = forever $ putStrLn \"" ++ var ++ "\"" |
         n <- [1..3000], let var = "a" ++ show n ]
main = writeFile "Out.hs" modu

The string literals seem to be important, I didn't manage to find a variation that blew up without them.

Sven Panne says: To alleviate the pain a bit, I've uploaded a new version of OpenGLRaw (2.5.0.0) to Hackage, containing 2 improvements:

• foreign import "dynamic"s with the same signature are re-used, cutting down their number from 3062 to 864.
• Those foreign import "dynamic"s live in a separate module now.

Travis CI seems to be happy with these changes (the VMs there don't have much memory, either), although Haddock still seems to eat memory like hell. But that's a different story...

This may or may not be related, but Greg Weber reports: We have observed issues with compile-time inlining taking much longer in newer versions of GHC in some cases: https://github.com/larskuhtz/toCaseFoldBuildTimes. This particular issue was reported to the text repo: https://github.com/bos/text/issues/116

The former link has a helpful collection of associated data.

changed the description

@rwbarton Thanks a lot for a smaller use case!

I had a brief look at generated Core (for both 7.8.4 and HEAD) and couldn't really find anything suspicious. Also core stats seem similar -- we generate almost identical number of terms/types/coercions. (although I'm far from an expert so take that with a grain of salt)

The good news is that I managed to narrow it down to b8392ae7 What I did: initially I just wanted to bisect but couldn't really get GHC to build before Jan 2015. So I checkout a commit from early January (the problem is reproducible there) and reverted the above patch (at which point the problem is no longer reproducible).

I'll try to look into that commit tomorrow.

Trac metadata

Trac field	Value
CC	ghc.haskell.org@liyang.hu, paolo.veronelli@gmail.com, simonpj@microsoft.com → ghc.haskell.org@liyang.hu, michalt, paolo.veronelli@gmail.com, simonpj@microsoft.com

I'm a bit stuck. I've had a look at that commit, but I don't really understand simplifier well enough to find the reason of the blow-up.

Any ideas are welcome! :-)

Can you give the output of -dshow-passes before and after reverting the patch? That should show if there is a big change in code size. Thanks.

Attached file ghc-show-passes-with-revert (download).

Attached file ghc-show-passes-without-revert (download).

tl;dr: No difference

If anyone wants to try to reproduce: checkout b0379819 and then revert b8392ae7

mentioned in commit c89bd681

mentioned in commit 45d9a15c