ByteString pinned memory can be leaky
My question on IRC:
How does memory allocation for pinned blocks work? Let's say pinned blocks are 4KB in size, and I allocate first a 3 KB ByteString A and then an 8-byte ByteString B. Now I GC A, no longer need it. Then according to https://ghc.haskell.org/trac/ghc/wiki/Commentary/Rts/Storage/GC/Pinned "a single pinned object keeps alive the whole block in which it resides", my small ByteString B keeps the entire block alive. But what happens with the 3KB in the front of that block? Will it be re-used by the next ByteString allocation (say 1KB)? In other words, how smart is allocatePinned as an allocator?
slyfox: nh2: allocatePinned it quite dump. it only allocated from free tail space
nh2: slyfox: that sounds like a huge potential for memory leak then slyfox: yes, fragmentation is quite bad for bytestrings
So it seems that I can get into the unfortunate situation where a super short
ByteString of a few bytes can waste an entire 4 KB block of memory; some migth call this a leak.
One idea to solve it seems to be to change standard
ByteStrings to not pinned, and to allocate pinned ones explicitly when needed. This seems to be an often-discussed topic and not trivial because many
ByteString functions are implemented using libc FFI functions.
However, it seems there will always be some need for pinned memory, so we should better have an efficient way to manage it in any case.
Efficient here means, for example, to re-use freed memory inside a block instead of only using free tail space.
It seems that
jemalloc has a feature to use given blocks of memory and provide a
malloc() functionality inside them:
Perhaps this could be used to provide GHC with a simple method to use pinned memory more efficiently?