Skip to content

File read/write operations >= 2GiB fail on macOS with an "Invalid argument" exception

Summary

File reads/writes bigger than 2GB result in an "Invalid argument" exception on macOS. Files bigger than 2GB still work, but individual read/write operations bigger than 2GB fail.

This was already reported for the bytestring library, but the actual problem is in base. I didn't see an earlier report.

bytestring report: https://github.com/haskell/bytestring/issues/153

Python has a workaround for this issue and more discussion: https://bugs.python.org/issue24658

Steps to reproduce

{-# LANGUAGE ScopedTypeVariables #-}
import Foreign.Ptr (Ptr)
import Foreign.Marshal.Alloc (mallocBytes, free)
import System.IO (hPutBuf, withBinaryFile, IOMode (WriteMode)) 

-- more than 2GiB
numBytes :: Int
numBytes = 2264375889

main :: IO ()
main = do
  (ptr :: Ptr ()) <- mallocBytes numBytes
  -- the next line produces the exception on macOS
  withBinaryFile "test.out" WriteMode (\h -> hPutBuf h ptr numBytes)
  free ptr

Expected behavior

The base library provides operating system independent buffering and encoding functionality. I think it should work around operating system peculiarities and write all the data to the test.out file without throwing an excepion.

Environment

Tested with GHC 8.6.5 on macOS 10.14.6

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information