ByteString pinned memory can be leaky

My question on IRC:

How does memory allocation for pinned blocks work?

Let's say pinned blocks are 4KB in size, and I allocate first a 3 KB
ByteString A and then an 8-byte ByteString B.

Now I GC A, no longer need it.

Then according to
  https://ghc.haskell.org/trac/ghc/wiki/Commentary/Rts/Storage/GC/Pinned
"a single pinned object keeps alive the whole block in which it resides",
my small ByteString B keeps the entire block alive.

But what happens with the 3KB in the front of that block?
Will it be re-used by the next ByteString allocation (say 1KB)?
In other words, how smart is allocatePinned as an allocator?

The answer:

slyfox: nh2: allocatePinned it quite dump. it only allocated from free tail space

nh2: slyfox: that sounds like a huge potential for memory leak then
slyfox: yes, fragmentation is quite bad for bytestrings

So it seems that I can get into the unfortunate situation where a super short ByteString of a few bytes can waste an entire 4 KB block of memory; some migth call this a leak.

One idea to solve it seems to be to change standard ByteStrings to not pinned, and to allocate pinned ones explicitly when needed. This seems to be an often-discussed topic and not trivial because many ByteString functions are implemented using libc FFI functions.

However, it seems there will always be some need for pinned memory, so we should better have an efficient way to manage it in any case.

Efficient here means, for example, to re-use freed memory inside a block instead of only using free tail space.

It seems that jemalloc has a feature to use given blocks of memory and provide a malloc() functionality inside them:

https://stackoverflow.com/questions/30836359/jemalloc-mmap-and-shared-memory

Perhaps this could be used to provide GHC with a simple method to use pinned memory more efficiently?

Trac metadata

Trac field	Value
Version	8.0.1
Type	Bug
TypeOfFailure	OtherFailure
Priority	normal
Resolution	Unresolved
Component	Runtime System
Test case
Differential revisions
BlockedBy
Related
Blocking
CC	nh2, slyfox
Operating system
Architecture