Commits on Source (6)
-
Hannes Siebenhandl authored
We add an `Ord` instance so that we can store `IfaceType` in a `Data.Map` container. This is required to deduplicate `IfaceType` while writing `.hi` files to disk. Deduplication has many beneficial consequences to both file size and memory usage, as the deduplication enables implicit sharing of values. See issue #24540 for more motivation. The `Ord` instance would be unnecessary if we used a `TrieMap` instead of `Data.Map` for the deduplication process. While in theory this is clerarly the better option, experiments on the agda code base showed that a `TrieMap` implementation has worse run-time performance characteristics. To the change itself, we mostly derive `Eq` and `Ord`. This requires us to change occurrences of `FastString` with `LexicalFastString`, since `FastString` has no `Ord` instance. We change the definition of `IfLclName` to a newtype of `LexicalFastString`, to make such changes in the future easier. Bump haddock submodule for IfLclName changes
1b2f0ff0 -
Hannes Siebenhandl authoreddd01fcb4
-
Hannes Siebenhandl authored
The type `IfaceType` is a highly redundant, tree-like data structure. While benchmarking, we realised that the high redundancy of `IfaceType` causes high memory consumption in GHCi sessions when byte code is embedded into the `.hi` file via `-fwrite-if-simplified-core` or `-fbyte-code-and-object-code`. Loading such `.hi` files from disk introduces many duplicates of memory expensive values in `IfaceType`, such as `IfaceTyCon`, `IfaceTyConApp`, `IA_Arg` and many more. We improve the memory behaviour of GHCi by adding an additional deduplication table for `IfaceType` to the serialisation of `ModIface`, similar to how we deduplicate `Name`s and `FastString`s. When reading the interface file back, the table allows us to automatically share identical values of `IfaceType`. To provide some numbers, we evaluated this patch on the agda code base. We loaded the full library from the `.hi` files, which contained the embedded core expressions (`-fwrite-if-simplified-core`). Before this patch: * Load time: 11.7 s, 2.5 GB maximum residency. After this patch: * Load time: 7.3 s, 1.7 GB maximum residency. This deduplication has the beneficial side effect to additionally reduce the size of the on-disk interface files tremendously. For example, on agda, we reduce the size of `.hi` files (with `-fwrite-if-simplified-core`): * Before: 101 MB on disk * Now: 24 MB on disk This has even a beneficial side effect on the cabal store. We reduce the size of the store on disk: * Before: 341 MB on disk * Now: 310 MB on disk Note, none of the dependencies have been compiled with `-fwrite-if-simplified-core`, but `IfaceType` occurs in multiple locations in a `ModIface`. We also add IfaceType deduplication table to .hie serialisation and refactor .hie file serialisation to use the same infrastrucutre as `putWithTables`. Bump haddock submodule to accomodate for changes to the deduplication table layout and binary interface.
7e5a678d -
Introduce the flag `-fwrite-if-compression=<n>` which allows to configure the compression level of writing .hi files. The motivation is that some deduplication operations are too expensive for the average use case. Hence, we introduce multiple compression levels that have a minimal impact on performance, but still reduce the memory residency and `.hi` file size on disk considerably. We introduce three compression levels: * `1`: `Normal` mode. This is the least amount of compression. It deduplicates only `Name` and `FastString`s, and is naturally the fastest compression mode. * `2`: `Safe` mode. It has a noticeable impact on .hi file size and is marginally slower than `Normal` mode. In general, it should be safe to always use `Safe` mode. * `3`: `Full` deduplication mode. Deduplicate as much as we can, resulting in minimal .hi files, but at the cost of additional compilation time. Reading .hi files doesn't need to know the initial compression level, and can always deserialise a `ModIface`. This allows users to experiment with different compression levels for packages, without recompilation of dependencies. Note, the deduplication also has an additional side effect of reduced memory consumption to implicit sharing of deduplicated elements. See #24540 for example where that matters. ------------------------- Metric Decrease: T21839c T24471 -------------------------
a98ae613 -
Add regression tests to track how `-fwrite-if-compression` levels affect the size of `.hi` files.
45e812de -
Hannes Siebenhandl authoredf6d5a5c2
Showing
- compiler/GHC/Core/Map/Expr.hs 3 additions, 0 deletionscompiler/GHC/Core/Map/Expr.hs
- compiler/GHC/Core/TyCo/Rep.hs 2 additions, 2 deletionscompiler/GHC/Core/TyCo/Rep.hs
- compiler/GHC/CoreToIface.hs 13 additions, 13 deletionscompiler/GHC/CoreToIface.hs
- compiler/GHC/Data/FastString.hs 4 additions, 1 deletioncompiler/GHC/Data/FastString.hs
- compiler/GHC/Data/TrieMap.hs 83 additions, 5 deletionscompiler/GHC/Data/TrieMap.hs
- compiler/GHC/Driver/DynFlags.hs 2 additions, 0 deletionscompiler/GHC/Driver/DynFlags.hs
- compiler/GHC/Driver/Main.hs 3 additions, 3 deletionscompiler/GHC/Driver/Main.hs
- compiler/GHC/Driver/Session.hs 3 additions, 0 deletionscompiler/GHC/Driver/Session.hs
- compiler/GHC/Iface/Binary.hs 114 additions, 35 deletionscompiler/GHC/Iface/Binary.hs
- compiler/GHC/Iface/Decl.hs 3 additions, 4 deletionscompiler/GHC/Iface/Decl.hs
- compiler/GHC/Iface/Env.hs 7 additions, 8 deletionscompiler/GHC/Iface/Env.hs
- compiler/GHC/Iface/Ext/Binary.hs 72 additions, 91 deletionscompiler/GHC/Iface/Ext/Binary.hs
- compiler/GHC/Iface/Ext/Utils.hs 3 additions, 3 deletionscompiler/GHC/Iface/Ext/Utils.hs
- compiler/GHC/Iface/Load.hs 11 additions, 3 deletionscompiler/GHC/Iface/Load.hs
- compiler/GHC/Iface/Recomp.hs 2 additions, 2 deletionscompiler/GHC/Iface/Recomp.hs
- compiler/GHC/Iface/Recomp/Binary.hs 7 additions, 2 deletionscompiler/GHC/Iface/Recomp/Binary.hs
- compiler/GHC/Iface/Syntax.hs 2 additions, 1 deletioncompiler/GHC/Iface/Syntax.hs
- compiler/GHC/Iface/Type.hs 116 additions, 46 deletionscompiler/GHC/Iface/Type.hs
- compiler/GHC/Iface/Type/Map.hs 180 additions, 0 deletionscompiler/GHC/Iface/Type/Map.hs
- compiler/GHC/IfaceToCore.hs 14 additions, 14 deletionscompiler/GHC/IfaceToCore.hs
compiler/GHC/Iface/Type/Map.hs
0 → 100644