Avoid unneccessarily re-serialising the `ModIface`
To reduce memory usage of `ModIface`, we serialise `ModIface` to an in-memory byte array, which implicitly shares duplicated values. This serialised byte array can be reused to avoid work when we actually write the `ModIface` to disk. We introduce a new field to `ModIface` which allows us to save the byte array, and write it direclty to disk if the `ModIface` wasn't changed after the initial serialisation. This requires us to change absolute offsets, for example to jump to the deduplication table for `Name` or `FastString` with relative offsets, as the deduplication byte array doesn't contain header information, such as fingerprints. To allow us to dump the binary blob to disk, we need to replace all absolute offsets with relative ones. We introduce additional helpers for `ModIface` binary serialisation, which construct relocatable binary blobs. We say the binary blob is relocatable, if the binary representation can be moved and does not contain any absolute offsets. Further, we introduce new primitives for `Binary` that allow to create relocatable binaries, such as `forwardGetRel` and `forwardPutRel`. ------------------------- Metric Decrease: MultiLayerModulesDefsGhcWithCore Metric Increase: MultiComponentModules MultiLayerModules T10421 T12150 T12234 T12425 T13035 T13253-spj T13701 T13719 T14697 T15703 T16875 T18140 T18304 T18698a T18730 T18923 T20049 T24582 T5837 T6048 T9198 T9961 mhu-perf ------------------------- These metric increases may look bad, but they are all completely benign, we simply allocate 1 MB per module for `shareIface`. As this allocation is quite quick, it has a negligible impact on run-time performance. In fact, the performance difference wasn't measurable on my local machine. Reducing the size of the pre-allocated 1 MB buffer avoids these test failures, but also requires us to reallocate the buffer if the interface file is too big. These reallocations *did* have an impact on performance, which is why I have opted to accept all these metric increases, as the number of allocated bytes is merely a guidance. This 1MB allocation increase causes a lot of tests to fail that generally have a low allocation number. E.g., increasing from 40MB to 41MB is a 2.5% increase. In particular, the tests T12150, T13253-spj, T18140, T18304, T18698a, T18923, T20049, T24582, T5837, T6048, and T9961 only fail on i386-darwin job, where the number of allocated bytes seems to be lower than in other jobs. The test T16875 fails on i386-linux-debian10 for the same reason.
parent
c5e7b418
No related branches found
No related tags found
Pipeline #96783 canceled
Stage: tool-lint
Stage: quick-build
Stage: full-build
Stage: packaging
Stage: testing
Showing
- compiler/GHC.hs 29 additions, 1 deletioncompiler/GHC.hs
- compiler/GHC/Iface/Binary.hs 29 additions, 16 deletionscompiler/GHC/Iface/Binary.hs
- compiler/GHC/Iface/Ext/Binary.hs 1 addition, 1 deletioncompiler/GHC/Iface/Ext/Binary.hs
- compiler/GHC/Iface/Ext/Fields.hs 3 additions, 3 deletionscompiler/GHC/Iface/Ext/Fields.hs
- compiler/GHC/Iface/Load.hs 16 additions, 16 deletionscompiler/GHC/Iface/Load.hs
- compiler/GHC/Iface/Make.hs 67 additions, 35 deletionscompiler/GHC/Iface/Make.hs
- compiler/GHC/Iface/Recomp.hs 2 additions, 1 deletioncompiler/GHC/Iface/Recomp.hs
- compiler/GHC/Iface/Rename.hs 9 additions, 7 deletionscompiler/GHC/Iface/Rename.hs
- compiler/GHC/Tc/Errors/Hole.hs 1 addition, 1 deletioncompiler/GHC/Tc/Errors/Hole.hs
- compiler/GHC/Tc/Gen/Splice.hs 4 additions, 2 deletionscompiler/GHC/Tc/Gen/Splice.hs
- compiler/GHC/Tc/Utils/Backpack.hs 4 additions, 4 deletionscompiler/GHC/Tc/Utils/Backpack.hs
- compiler/GHC/Unit/Module/ModIface.hs 509 additions, 134 deletionscompiler/GHC/Unit/Module/ModIface.hs
- compiler/GHC/Utils/Binary.hs 171 additions, 14 deletionscompiler/GHC/Utils/Binary.hs
- testsuite/tests/plugins/simple-plugin/Simple/RemovePlugin.hs 4 additions, 3 deletionstestsuite/tests/plugins/simple-plugin/Simple/RemovePlugin.hs
- utils/haddock/haddock-api/src/Haddock/InterfaceFile.hs 4 additions, 4 deletionsutils/haddock/haddock-api/src/Haddock/InterfaceFile.hs
Loading
Please register or sign in to comment