Make heap profiling less guesswork by giving more control to Haskell program
Currently figuring out just how much memory is used by specific data-structures in a complex program is quite difficult. With heap profiling we can break the overall memory usage down by various keys. However this only provides a rather global view not just of a specific part of interest to the programmer.
We can answer questions such as how much memory is used by, for example, the list constructor
Prelude. by breaking down by type, but this only accounts for the list closures and not even the list contents, furthermore lists are used everywhere! So we have to resort to measuring what we can with the current capabilities and guessing what is really going on. It's pretty much the same with the other breakdowns we always have to resort to guessing outside of some simple cases.
That is hardly satisfying.
As a first step towards solving this problem I propose adding a new profiling mode which lets the Haskell program being profiled specify objects whose size is to be accounted over the course of the run.
I am currently working on generalizing the heap traversal code in the retainer profiler to allow this.
While working on this and seeing how horrible traversing the heap is in C a thought occurred. Can we use
ghc-heap to do all of this stuff in Haskell instead of C? At first that sounds kind of crazy. If we attempt to do this in the same process as the code being profiled we have to make sure GC doesn't run or else heap objects might move. It could still be possible by reserving a large nursery before running the profiling code. I'm not sure.
Another suggestion by @mpickering is simply running the profiling code in a different process, which would certainly be less problematic from an RTS perspective.