Skip to content

GitLab

Open
Created by Niklas Hambüchen@nh2Reporter

stat() calls can block Haskell runtime

getFileStatus (stat() syscall) is marked as unsafe, which means that if I have e.g. +RTS -N4, I can't stat more than 4 files at the same time without completely stopping the Haskell world.

This is an issue on network file systems, where a single stat() can easily take 2 milliseconds, so you typically want to do them in parallel (but due to the above you can't).

The underlying problem is that there are some Linux syscalls you typically need on networked file systems that have no asynchronous equivalent; according to http://blog.libtorrent.org/2012/10/asynchronous-disk-io/ these are at least:

  • stat()
  • open()
  • fallocate()
  • rename()

A quick skim through libraries/base/System/Posix/Internals.hs reveals the situation:

  • stat() only exists as unsafe
  • open() has both safe and unsafe variants (I haven't checked which one is used in practice)

The remaining ones are in the unix package

  • fallocate() is safe
  • rename() is unsafe

It seems to me that there are two issues here:

  1. None of these calls should be unsafe because they may block for a very long time (e.g. > 0.5 ms even on the fastest LANs).

  2. We need to answer the question: If we marked them as safe, how many of them would the RTS execute in parallel? To my current knowledge (thanks rwbarton and slyfox on #ghc), there's a pool of Haskell executing threads (the usual -RTS -N), and a pool of FFI threads. Are there any restrictions on the size of that latter pool? The docs https://downloads.haskell.org/\~ghc/latest/docs/html/users_guide/ffi-chap.html\#foreign-imports-and-multi-threading are not specific on that topic, simply mentioning "but there may be an arbitrary number of foreign calls in progress at any one time, regardless of the +RTS -N value". Does that mean the amount of FFI threads is truly unbounded?

Trac metadata
Trac field Value
Version 8.0.2
Type Bug
TypeOfFailure OtherFailure
Priority normal
Resolution Unresolved
Component Runtime System
Test case
Differential revisions
BlockedBy
Related
Blocking
CC rwbarton, simonmar, slyfox
Operating system
Architecture
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information