HIE file generation is inefficient (#16233) · Issues · Glasgow Haskell Compiler / GHC

HIE file generation is inefficient

Generating HIE files eats up a lot of memory and takes a long time. There are at least two big parts to this:

Every expression in the AST is desugared via deSugarExpr for the purpose of extracting a type. GHC ends up doing the same work multiple times for the same nodes. Ex: in order to desugar HsApp _ fun arg, GHC desugars both fun and arg, yet the HIE code doesn't cache that (so it ends up desugaring fun and arg at least once more).
There is a lot of stuff being written into these HIE files, making them quite big (even after compressing/deduplicating type information). One data point: the largest HIE file at time of writing is Parser.hie, which is 7.2MB, compared to just 58K for Parser.hiQ Writing large files is naturally going to take up memory and be slow.

Trac metadata

Trac field	Value
Version	8.7
Type	Bug
TypeOfFailure	OtherFailure
Priority	normal
Resolution	Unresolved
Component	Compiler
Test case
Differential revisions
BlockedBy
Related
Blocking
CC
Operating system
Architecture

Generating HIE files eats up a lot of memory and takes a long time. There are at least two big parts to this:

- Every expression in the AST is desugared via `deSugarExpr` for the purpose of extracting a type. GHC ends up doing the same work multiple times for the same nodes. Ex: in order to desugar `HsApp _ fun arg`, GHC desugars both `fun` and `arg`, yet the HIE code doesn't cache that (so it ends up desugaring `fun` and `arg` at least once more).
- There is a lot of stuff being written into these HIE files, making them quite big (even after compressing/deduplicating type information). One data point: the largest HIE file at time of writing is `Parser.hie`, which is 7.2MB, compared to just 58K for `Parser.hi`Q Writing large files is naturally going to take up memory and be slow.

<details><summary>Trac metadata</summary>

| Trac field             | Value        |
| ---------------------- | ------------ |
| Version                | 8.7          |
| Type                   | Bug          |
| TypeOfFailure          | OtherFailure |
| Priority               | normal       |
| Resolution             | Unresolved   |
| Component              | Compiler     |
| Test case              |              |
| Differential revisions |              |
| BlockedBy              |              |
| Related                |              |
| Blocking               |              |
| CC                     |              |
| Operating system       |              |
| Architecture           |              |

</details>