Admin message

Due to a large amount of spam we do not allow new users to create repositories, they are "external" users. If you are a new user and want to create a repository, for example for forking GHC, open a new issue on ghc/ghc using the "get-verified" issue template

Awkward design around pinned ByteArray# and compaction
According to the documentation for `GHC.Compact`: > Pinned `ByteArray#` objects cannot be compacted. This is for a good reason: the memory is pinned so that it can be referenced by address (the address might be stored in a C data structure, for example), so we can't make a copy of it to store in the `Compact`. We also provide a primop `isByteArrayPinned#` to allow users to opportunistically exploit existing pinned-ness of `ByteArray#`s to avoid copying when using the FFI or `Storable`. However, these features disagree about when a `ByteArray#` is pinned: - Compaction considers a `ByteArray#` to be pinned only if it was explicitly pinned at creation-time with `newPinnedByteArray#` or `newAlignedPinnedByteArray#`. - In addition to explicitly pinned `ByteArray#`s, `isByteArrayPinned#` considers `ByteArray#`s in large-object heap blocks or in compact regions to be pinned. This mismatch can easily lead to memory-unsafety. Here's a small example program: <details> ```haskell import qualified Data.ByteString.Short as SBS import qualified Data.ByteString.Char8 as BS import GHC.Compact (compact, getCompact) import Control.Monad (forM_) import System.Mem (performMajorGC) main :: IO () main = do x <- compact (SBS.fromShort (SBS.replicate 8000 48)) let printCompact v = BS.putStrLn (BS.take 20 (getCompact v)) printCompact x performMajorGC forM_ [8000..9000] $ \i -> compact (SBS.fromShort (SBS.replicate i 49)) printCompact x ``` </details> This program _seems_ innocent. But under the hood, `SBS.replicate 8000 48` creates a large pinned-but-not-_explicitly_-pinned `ByteArray#`, and `SBS.fromShort` (since `bytestring-0.11.1.0`) sees that it is pinned and opportunistically creates a non-GC reference into it instead of making a copy. Then the underlying buffer is copied into the compact region `x` without issue since it was not explicitly pinned, but the non-GC reference in the `ByteString` cannot be updated and becomes a dangling pointer. (The subsequent code tries to demonstrate this by making two calls to `printCompact x` produce different output, though of course this isn't truly guaranteed to work.) Of course, `GHC.Compact` is not a "Safe Haskell" module and `ByteString`s are typically not compactible. But it's still a little unsatisfactory that this performs unsafe memory accesses instead of raising a `CompactionFailed` exception. Here are a few options: 2. Prevent compaction of these _implicitly_ pinned `ByteArray#`s. 1. Do nothing, and just document this infelicity. 3. Weaken `isByteArrayPinned#` to only consider _explicitly_ pinned `ByteArray#`s. 4. Provide a primop that makes an implictly pinned `ByteArray#` explicitly pinned, without copying its contents. 5. Distinguish between `ByteArray#` references (which may happen to refer to pinned objects) and `PinnedByteArray#` references, and only refuse to compact the latter. (This means the distinction must exist at runtime. Perhaps both can be references to `ARR_WORDS` heap objects, but with different pointer tags?)
issue