GHC issueshttps://gitlab.haskell.org/ghc/ghc/-/issues2019-07-07T18:03:44Zhttps://gitlab.haskell.org/ghc/ghc/-/issues/15615Rename some mutable closure types for consistency2019-07-07T18:03:44ZAndreas KlebingerRename some mutable closure types for consistencyDescription taken from the diff:
```
SMALL_MUT_ARR_PTRS_FROZEN0 -> SMALL_MUT_ARR_PTRS_FROZEN_DIRTY
SMALL_MUT_ARR_PTRS_FROZEN -> SMALL_MUT_ARR_PTRS_FROZEN_CLEAN
MUT_ARR_PTRS_FROZEN0 -> MUT_ARR_PTRS_FROZEN_DIRTY
MUT_ARR_PTRS_FROZEN -> MUT...Description taken from the diff:
```
SMALL_MUT_ARR_PTRS_FROZEN0 -> SMALL_MUT_ARR_PTRS_FROZEN_DIRTY
SMALL_MUT_ARR_PTRS_FROZEN -> SMALL_MUT_ARR_PTRS_FROZEN_CLEAN
MUT_ARR_PTRS_FROZEN0 -> MUT_ARR_PTRS_FROZEN_DIRTY
MUT_ARR_PTRS_FROZEN -> MUT_ARR_PTRS_FROZEN_CLEAN
```
Naming is now consistent with other CLEAR/DIRTY objects (MVAR, MUT_VAR,
MUT_ARR_PTRS).
(alternatively we could rename MVAR_DIRTY/MVAR_CLEAN etc. to MVAR0/MVAR)
Removed a few comments in Scav.c about FROZEN0 being on the mut_list
because it's now clear from the closure type.
<details><summary>Trac metadata</summary>
| Trac field | Value |
| ---------------------- | -------------- |
| Version | 8.4.3 |
| Type | Task |
| TypeOfFailure | OtherFailure |
| Priority | normal |
| Resolution | Unresolved |
| Component | Runtime System |
| Test case | |
| Differential revisions | |
| BlockedBy | |
| Related | |
| Blocking | |
| CC | |
| Operating system | |
| Architecture | |
</details>
<!-- {"blocked_by":[],"summary":"Rename some mutable closure types for consistency","status":"New","operating_system":"","component":"Runtime System","related":[],"milestone":"8.6.1","resolution":"Unresolved","owner":{"tag":"Unowned"},"version":"8.4.3","keywords":[],"differentials":[],"test_case":"","architecture":"","cc":[""],"type":"Task","description":"Description taken from the diff:\r\n\r\n{{{\r\nSMALL_MUT_ARR_PTRS_FROZEN0 -> SMALL_MUT_ARR_PTRS_FROZEN_DIRTY\r\nSMALL_MUT_ARR_PTRS_FROZEN -> SMALL_MUT_ARR_PTRS_FROZEN_CLEAN\r\nMUT_ARR_PTRS_FROZEN0 -> MUT_ARR_PTRS_FROZEN_DIRTY\r\nMUT_ARR_PTRS_FROZEN -> MUT_ARR_PTRS_FROZEN_CLEAN\r\n}}}\r\n\r\nNaming is now consistent with other CLEAR/DIRTY objects (MVAR, MUT_VAR,\r\nMUT_ARR_PTRS).\r\n\r\n(alternatively we could rename MVAR_DIRTY/MVAR_CLEAN etc. to MVAR0/MVAR)\r\n\r\nRemoved a few comments in Scav.c about FROZEN0 being on the mut_list\r\nbecause it's now clear from the closure type.\r\n\r\n\r\n","type_of_failure":"OtherFailure","blocking":[]} -->8.6.1https://gitlab.haskell.org/ghc/ghc/-/issues/15485GHC uses 300% CPU when calling into blocking C call2023-12-01T13:50:01ZoconnoreGHC uses 300% CPU when calling into blocking C callHello,
I'm trying to write a program that modifies /etc/passwd safely, and so I wrote a function that looks like this:
```hs
lockPwd f = bracket main recover (\_ -> f)
where
mode = unionFileModes ownerReadMode ownerWriteMode
...Hello,
I'm trying to write a program that modifies /etc/passwd safely, and so I wrote a function that looks like this:
```hs
lockPwd f = bracket main recover (\_ -> f)
where
mode = unionFileModes ownerReadMode ownerWriteMode
main = do
fd <- openFd "/etc/.pwd.lock" WriteOnly (Just mode) defaultFileFlags
putStrLn "waiting to set lock"
waitToSetLock fd (WriteLock, AbsoluteSeek, 0, 0)
putStrLn "got lock"
return fd
recover = flip setLock (Unlock, AbsoluteSeek, 0, 0)
```
When I run it, my fans go wild, and CPU usage hits 300%.
/u/gelisam did some more in depth investigation here: https://www.reddit.com/r/haskell/comments/94wbfc/systemunixiowaittosetlock_call_results_in_300_cpu/e3p7sks/
> Googling confirms that the parallel gc is using spin locks. So I think what is happening is that the waitToSetLock call makes the current thread unavailable for garbage collection (I don't know if that means the thread is "descheduled" as in the linked issue), which then causes the other threads to spin-lock while waiting for that thread at the next parallel GC. The problem still occurs with the latest release, GHC 8.4.3, and I could not find an existing issue describing the problem, so please file a ticket.
Maybe an argument could be made that this waitToSetLock call should be converted to be [interruptible](https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/ffi-chap.html#interruptible-foreign-calls), but it also seems like I should be able to make my haskell program block patiently if I want it to.
<details><summary>Trac metadata</summary>
| Trac field | Value |
| ---------------------- | -------------- |
| Version | 8.4.3 |
| Type | Bug |
| TypeOfFailure | OtherFailure |
| Priority | normal |
| Resolution | Unresolved |
| Component | Runtime System |
| Test case | |
| Differential revisions | |
| BlockedBy | |
| Related | |
| Blocking | |
| CC | |
| Operating system | |
| Architecture | |
</details>
<!-- {"blocked_by":[],"summary":"GHC uses 300% CPU when calling into blocking C call","status":"New","operating_system":"","component":"Runtime System","related":[],"milestone":"8.6.1","resolution":"Unresolved","owner":{"tag":"Unowned"},"version":"8.4.3","keywords":[],"differentials":[],"test_case":"","architecture":"","cc":[""],"type":"Bug","description":"Hello,\r\n\r\nI'm trying to write a program that modifies /etc/passwd safely, and so I wrote a function that looks like this:\r\n\r\n{{{#!hs\r\nlockPwd f = bracket main recover (\\_ -> f)\r\n where\r\n mode = unionFileModes ownerReadMode ownerWriteMode\r\n main = do\r\n fd <- openFd \"/etc/.pwd.lock\" WriteOnly (Just mode) defaultFileFlags\r\n putStrLn \"waiting to set lock\"\r\n waitToSetLock fd (WriteLock, AbsoluteSeek, 0, 0)\r\n putStrLn \"got lock\"\r\n return fd\r\n recover = flip setLock (Unlock, AbsoluteSeek, 0, 0)\r\n}}}\r\n\r\nWhen I run it, my fans go wild, and CPU usage hits 300%.\r\n\r\n/u/gelisam did some more in depth investigation here: https://www.reddit.com/r/haskell/comments/94wbfc/systemunixiowaittosetlock_call_results_in_300_cpu/e3p7sks/\r\n\r\n> Googling confirms that the parallel gc is using spin locks. So I think what is happening is that the waitToSetLock call makes the current thread unavailable for garbage collection (I don't know if that means the thread is \"descheduled\" as in the linked issue), which then causes the other threads to spin-lock while waiting for that thread at the next parallel GC. The problem still occurs with the latest release, GHC 8.4.3, and I could not find an existing issue describing the problem, so please file a ticket.\r\n\r\nMaybe an argument could be made that this waitToSetLock call should be converted to be [[https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/ffi-chap.html#interruptible-foreign-calls|interruptible]], but it also seems like I should be able to make my haskell program block patiently if I want it to. ","type_of_failure":"OtherFailure","blocking":[]} -->8.6.1https://gitlab.haskell.org/ghc/ghc/-/issues/15440Flush eventlog in hs_init_ghc2019-07-07T18:04:53ZMitsutoshi AoeFlush eventlog in hs_init_ghcCurrently GHC RTS traces some basic information about the process in `hs_init_ghc` but the information is often flushed to the eventlog at the very end of the process lifetime. Or even worse, it's never written to it if the process termi...Currently GHC RTS traces some basic information about the process in `hs_init_ghc` but the information is often flushed to the eventlog at the very end of the process lifetime. Or even worse, it's never written to it if the process terminates abnormally.
This is very inconvenient because the information contains some useful stuff like `EVENT_WALL_CLOCK_TIME`, which is quite important if user wants to sync timestamps in the eventlog with clock.
The simplest way to fix the issue is to flush the eventlog buffer in `hs_init_ghc`.
This is implemented at https://github.com/maoe/ghc/commit/30e20e4e66adccc28c59c8876331918d30eacef2.8.6.1https://gitlab.haskell.org/ghc/ghc/-/issues/15306First attempt at starting GHCI failed2019-07-07T18:13:21ZDaleBFirst attempt at starting GHCI failed```
PS C:\WINDOWS\system32> ghci
GHCi, version 8.4.3: http://www.haskell.org/ghc/ :? for help
ghc.exe: C:\F\G77\lib\libmingw32.a: Not a x86_64 PE+ file.
ghc.exe: getNumberOfSymbols: error whilst reading `CRTglob.o' header in `C:\F\G77\l...```
PS C:\WINDOWS\system32> ghci
GHCi, version 8.4.3: http://www.haskell.org/ghc/ :? for help
ghc.exe: C:\F\G77\lib\libmingw32.a: Not a x86_64 PE+ file.
ghc.exe: getNumberOfSymbols: error whilst reading `CRTglob.o' header in `C:\F\G77\lib\libmingw32.a': Unknown COFF_OBJ_TYPE.
ghc.exe: loadArchive: error whilst reading `C:\F\G77\lib\libmingw32.a'
ghc.exe: panic! (the 'impossible' happened)
(GHC version 8.4.3 for x86_64-unknown-mingw32):
loadArchive "C:\\F\\G77\\lib\\libmingw32.a": failed
Please report this as a GHC bug: http://www.haskell.org/ghc/reportabug
```8.6.1https://gitlab.haskell.org/ghc/ghc/-/issues/15285"strange closure type" in T7919 with the threaded2 way2019-07-07T18:13:27ZAlp Mestanogullari"strange closure type" in T7919 with the threaded2 wayLast night's `./validate --slow` run on Circle CI revealed the following failure:
```
Wrong exit code for T7919(threaded2)(expected 0 , actual 134 )
Stderr ( T7919 ):
T7919: internal error: evacuate: strange closure type 32554624
(G...Last night's `./validate --slow` run on Circle CI revealed the following failure:
```
Wrong exit code for T7919(threaded2)(expected 0 , actual 134 )
Stderr ( T7919 ):
T7919: internal error: evacuate: strange closure type 32554624
(GHC version 8.5.20180617 for x86_64_unknown_linux)
Please report this as a GHC bug: http://www.haskell.org/ghc/reportabug
Aborted
*** unexpected failure for T7919(threaded2)
```
The full log is available [here](https://circleci.com/api/v1.1/project/github/ghc/ghc/6205/output/107/0?file=true). Among other things, you can see that it indeed only fails in the threaded2 way.
<details><summary>Trac metadata</summary>
| Trac field | Value |
| ---------------------- | -------------- |
| Version | 8.5 |
| Type | Bug |
| TypeOfFailure | OtherFailure |
| Priority | normal |
| Resolution | Unresolved |
| Component | Runtime System |
| Test case | |
| Differential revisions | |
| BlockedBy | |
| Related | |
| Blocking | |
| CC | |
| Operating system | |
| Architecture | |
</details>
<!-- {"blocked_by":[],"summary":"\"strange closure type\" in T7919 with the threaded2 way","status":"New","operating_system":"","component":"Runtime System","related":[],"milestone":"","resolution":"Unresolved","owner":{"tag":"Unowned"},"version":"8.5","keywords":[],"differentials":[],"test_case":"","architecture":"","cc":[""],"type":"Bug","description":"Last night's `./validate --slow` run on Circle CI revealed the following failure:\r\n\r\n{{{\r\nWrong exit code for T7919(threaded2)(expected 0 , actual 134 )\r\nStderr ( T7919 ):\r\nT7919: internal error: evacuate: strange closure type 32554624\r\n (GHC version 8.5.20180617 for x86_64_unknown_linux)\r\n Please report this as a GHC bug: http://www.haskell.org/ghc/reportabug\r\nAborted\r\n*** unexpected failure for T7919(threaded2)\r\n}}}\r\n\r\nThe full log is available [https://circleci.com/api/v1.1/project/github/ghc/ghc/6205/output/107/0?file=true here]. Among other things, you can see that it indeed only fails in the threaded2 way.","type_of_failure":"OtherFailure","blocking":[]} -->8.6.1https://gitlab.haskell.org/ghc/ghc/-/issues/15171Add HeapView functionality breaks validate (stage 0 ghc is 8.4.2)2019-07-07T18:13:56Zjrp2014Add HeapView functionality breaks validate (stage 0 ghc is 8.4.2)```
"/opt/ghc/bin/ghc" -hisuf hi -osuf o -hcsuf hc -static -O0 -H64m -Wall -package-db libraries/bootstrapping.conf -this-unit-id mtl-2.2.2 -hide-all-packages -i -ilibraries/mtl/. -ilibraries/mtl/dist-boot/build -Ilibraries/mtl/dist...```
"/opt/ghc/bin/ghc" -hisuf hi -osuf o -hcsuf hc -static -O0 -H64m -Wall -package-db libraries/bootstrapping.conf -this-unit-id mtl-2.2.2 -hide-all-packages -i -ilibraries/mtl/. -ilibraries/mtl/dist-boot/build -Ilibraries/mtl/dist-boot/build -ilibraries/mtl/dist-boot/build/./autogen -Ilibraries/mtl/dist-boot/build/./autogen -Ilibraries/mtl/. -optP-include -optPlibraries/mtl/dist-boot/build/./autogen/cabal_macros.h -package-id base-4.11.1.0 -package-id transformers-0.5.5.0 -Wall -fno-warn-unused-imports -fno-warn-warnings-deprecations -Wcompat -Wnoncanonical-monad-instances -Wnoncanonical-monadfail-instances -XHaskell2010 -no-user-package-db -rtsopts -fno-warn-deprecated-flags -odir libraries/mtl/dist-boot/build -hidir libraries/mtl/dist-boot/build -stubdir libraries/mtl/dist-boot/build -c libraries/mtl/./Control/Monad/RWS/Lazy.hs -o libraries/mtl/dist-boot/build/Control/Monad/RWS/Lazy.o
libraries/ghc-heap/GHC/Exts/Heap.hs:119:48: error:
• Couldn't match expected type ‘ByteArray#’
with actual type ‘Array# b0’
• In the first argument of ‘sizeofByteArray#’, namely ‘dat’
In the first argument of ‘I#’, namely ‘(sizeofByteArray# dat)’
In the first argument of ‘div’, namely
‘(I# (sizeofByteArray# dat))’
• Relevant bindings include
dat :: Array# b0
(bound at libraries/ghc-heap/GHC/Exts/Heap.hs:115:18)
|
119 | let nelems = (I# (sizeofByteArray# dat)) `div` wORD_SIZE
| ^^^
libraries/ghc-heap/GHC/Exts/Heap.hs:121:47: error:
• Couldn't match expected type ‘ByteArray#’
with actual type ‘Array# b0’
• In the first argument of ‘indexWordArray#’, namely ‘dat’
In the first argument of ‘W#’, namely ‘(indexWordArray# dat i)’
In the expression: W# (indexWordArray# dat i)
• Relevant bindings include
dat :: Array# b0
(bound at libraries/ghc-heap/GHC/Exts/Heap.hs:115:18)
|
121 | rawWds = [W# (indexWordArray# dat i) | I# i <- [0.. end] ]
| ^^^
libraries/ghc-heap/GHC/Exts/Heap.hs:122:43: error:
• Couldn't match expected type ‘Array# a0’
with actual type ‘ByteArray#’
• In the first argument of ‘sizeofArray#’, namely ‘pointers’
In the first argument of ‘I#’, namely ‘(sizeofArray# pointers)’
In the expression: I# (sizeofArray# pointers)
|
122 | pelems = I# (sizeofArray# pointers)
| ^^^^^^^^
libraries/ghc-heap/GHC/Exts/Heap.hs:123:67: error:
• Couldn't match expected type ‘Array# Any’
with actual type ‘ByteArray#’
• In the fourth argument of ‘Array’, namely ‘pointers’
In the second argument of ‘($)’, namely
‘Array 0 (pelems - 1) pelems pointers’
In the expression: amap' Box $ Array 0 (pelems - 1) pelems pointers
|
123 | ptrList = amap' Box $ Array 0 (pelems - 1) pelems pointers
| ^^^^^^^^
libraries/ghc-heap/ghc.mk:3: recipe for target 'libraries/ghc-heap/dist-boot/build/GHC/Exts/Heap.o' failed
make[1]: *** [libraries/ghc-heap/dist-boot/build/GHC/Exts/Heap.o] Error 1
```
<details><summary>Trac metadata</summary>
| Trac field | Value |
| ---------------------- | -------------- |
| Version | 8.5 |
| Type | Bug |
| TypeOfFailure | OtherFailure |
| Priority | high |
| Resolution | Unresolved |
| Component | Runtime System |
| Test case | |
| Differential revisions | |
| BlockedBy | |
| Related | |
| Blocking | |
| CC | patrickdoc |
| Operating system | |
| Architecture | |
</details>
<!-- {"blocked_by":[],"summary":"Add HeapView functionality breaks validate (stage 0 ghc is 8.4.2)","status":"New","operating_system":"","component":"Runtime System","related":[],"milestone":"8.6.1","resolution":"Unresolved","owner":{"tag":"Unowned"},"version":"8.5","keywords":[],"differentials":[],"test_case":"","architecture":"","cc":["patrickdoc"],"type":"Bug","description":"{{{\r\n\"/opt/ghc/bin/ghc\" -hisuf hi -osuf o -hcsuf hc -static -O0 -H64m -Wall -package-db libraries/bootstrapping.conf -this-unit-id mtl-2.2.2 -hide-all-packages -i -ilibraries/mtl/. -ilibraries/mtl/dist-boot/build -Ilibraries/mtl/dist-boot/build -ilibraries/mtl/dist-boot/build/./autogen -Ilibraries/mtl/dist-boot/build/./autogen -Ilibraries/mtl/. -optP-include -optPlibraries/mtl/dist-boot/build/./autogen/cabal_macros.h -package-id base-4.11.1.0 -package-id transformers-0.5.5.0 -Wall -fno-warn-unused-imports -fno-warn-warnings-deprecations -Wcompat -Wnoncanonical-monad-instances -Wnoncanonical-monadfail-instances -XHaskell2010 -no-user-package-db -rtsopts -fno-warn-deprecated-flags -odir libraries/mtl/dist-boot/build -hidir libraries/mtl/dist-boot/build -stubdir libraries/mtl/dist-boot/build -c libraries/mtl/./Control/Monad/RWS/Lazy.hs -o libraries/mtl/dist-boot/build/Control/Monad/RWS/Lazy.o \r\n\r\nlibraries/ghc-heap/GHC/Exts/Heap.hs:119:48: error:\r\n • Couldn't match expected type ‘ByteArray#’\r\n with actual type ‘Array# b0’\r\n • In the first argument of ‘sizeofByteArray#’, namely ‘dat’\r\n In the first argument of ‘I#’, namely ‘(sizeofByteArray# dat)’\r\n In the first argument of ‘div’, namely\r\n ‘(I# (sizeofByteArray# dat))’\r\n • Relevant bindings include\r\n dat :: Array# b0\r\n (bound at libraries/ghc-heap/GHC/Exts/Heap.hs:115:18)\r\n |\r\n119 | let nelems = (I# (sizeofByteArray# dat)) `div` wORD_SIZE\r\n | ^^^\r\n\r\nlibraries/ghc-heap/GHC/Exts/Heap.hs:121:47: error:\r\n • Couldn't match expected type ‘ByteArray#’\r\n with actual type ‘Array# b0’\r\n • In the first argument of ‘indexWordArray#’, namely ‘dat’\r\n In the first argument of ‘W#’, namely ‘(indexWordArray# dat i)’\r\n In the expression: W# (indexWordArray# dat i)\r\n • Relevant bindings include\r\n dat :: Array# b0\r\n (bound at libraries/ghc-heap/GHC/Exts/Heap.hs:115:18)\r\n |\r\n121 | rawWds = [W# (indexWordArray# dat i) | I# i <- [0.. end] ]\r\n | ^^^\r\n\r\nlibraries/ghc-heap/GHC/Exts/Heap.hs:122:43: error:\r\n • Couldn't match expected type ‘Array# a0’\r\n with actual type ‘ByteArray#’\r\n • In the first argument of ‘sizeofArray#’, namely ‘pointers’\r\n In the first argument of ‘I#’, namely ‘(sizeofArray# pointers)’\r\n In the expression: I# (sizeofArray# pointers)\r\n |\r\n122 | pelems = I# (sizeofArray# pointers)\r\n | ^^^^^^^^\r\n\r\nlibraries/ghc-heap/GHC/Exts/Heap.hs:123:67: error:\r\n • Couldn't match expected type ‘Array# Any’\r\n with actual type ‘ByteArray#’\r\n • In the fourth argument of ‘Array’, namely ‘pointers’\r\n In the second argument of ‘($)’, namely\r\n ‘Array 0 (pelems - 1) pelems pointers’\r\n In the expression: amap' Box $ Array 0 (pelems - 1) pelems pointers\r\n |\r\n123 | ptrList = amap' Box $ Array 0 (pelems - 1) pelems pointers\r\n | ^^^^^^^^\r\nlibraries/ghc-heap/ghc.mk:3: recipe for target 'libraries/ghc-heap/dist-boot/build/GHC/Exts/Heap.o' failed\r\nmake[1]: *** [libraries/ghc-heap/dist-boot/build/GHC/Exts/Heap.o] Error 1\r\n}}}","type_of_failure":"OtherFailure","blocking":[]} -->8.6.1https://gitlab.haskell.org/ghc/ghc/-/issues/15136High CPU when asynchronous exception and unblocking retry on TVar raced2019-07-07T18:14:07ZnshimazaHigh CPU when asynchronous exception and unblocking retry on TVar racedDetail: https://github.com/nshimaza/race-tmvar-async-exception
Runtime falls into high CPU under racing condition between async exception and unblocking retry on TVar.
- Reproduces with +RTS -Nx where x \> 1
- Does NOT reproduce with +...Detail: https://github.com/nshimaza/race-tmvar-async-exception
Runtime falls into high CPU under racing condition between async exception and unblocking retry on TVar.
- Reproduces with +RTS -Nx where x \> 1
- Does NOT reproduce with +RTS -N1
- Program stalls at `killThread`
- High CPU based on given -Nx
- CPU won't be 100% if you gave x smaller than available hardware threads of your platform.
- Does NOT reproduce if TVar/retry is replaced by MVar
- Reproduced with GHC 8.4.2 (macOS High Sierra (10.13.4))
- Reproduced with GHC 8.4.2 (Docker for Mac Version 18.03.1-ce-mac65)
- Reproduced with ghc-8.5.20180506 (Docker for Mac Version 18.03.1-ce-mac65)
Minimal reproducing code here. (You can find more verbose code on the above github repo.)
```hs
main :: IO ()
main = do
let volume = 1000
forM_ [1..1000] $ \i -> do
putStrFlush $ show i ++ " "
-- Spawn massive number of threads.
threads <- replicateM volume $ do
trigger <- newTVarIO False
tid <- forkIO $ void $ atomically $ do
t <- readTVar trigger
if t then pure t else retry
pure (trigger, tid)
-- Make sure all threads are spawned.
threadDelay 30000
-- Let threads start to exit normally.
forkIO $ forM_ threads $ \(trigger, _) -> threadDelay 1 *> atomically (writeTVar trigger True)
-- Concurrently kill threads in order to create race.
-- TMVar operation and asynchronous exception can hit same thread simultaneously.
-- Adjust threadDelay if you don't reproduce very well.
threadDelay 1000
forM_ threads $ \(_, tid) -> do
putCharFlush 'A'
killThread tid -- When the issue reproduced, this killThread doesn't return.
putCharFlush '\b'
```
This program intentionally creates race condition between asynchronous exception
and unblocking operation of `retry` on TVar. From one side, a `writeTVar
trigger True` is attempted from external thread while target thread is blocking
at `retry` on the same `TVar`. On the other side, an asynchronous exception
`ThreadKilled` is thrown by yet another external thread to the same target
thread.
In other word, it attempts to kill a thread about to unblock.
I guess when the above two operation hit the same thread at the same time in
parallel in SMP environment, GHC runtime falls into high CPU.
<details><summary>Trac metadata</summary>
| Trac field | Value |
| ---------------------- | -------------- |
| Version | 8.4.2 |
| Type | Bug |
| TypeOfFailure | OtherFailure |
| Priority | highest |
| Resolution | Unresolved |
| Component | Runtime System |
| Test case | |
| Differential revisions | |
| BlockedBy | |
| Related | |
| Blocking | |
| CC | |
| Operating system | |
| Architecture | |
</details>
<!-- {"blocked_by":[],"summary":"High CPU when asynchronous exception and unblocking retry on TVar raced","status":"New","operating_system":"","component":"Runtime System","related":[],"milestone":"8.6.1","resolution":"Unresolved","owner":{"tag":"Unowned"},"version":"8.4.2","keywords":[],"differentials":[],"test_case":"","architecture":"","cc":[""],"type":"Bug","description":"Detail: https://github.com/nshimaza/race-tmvar-async-exception\r\n\r\nRuntime falls into high CPU under racing condition between async exception and unblocking retry on TVar.\r\n\r\n* Reproduces with +RTS -Nx where x > 1\r\n* Does NOT reproduce with +RTS -N1\r\n* Program stalls at `killThread`\r\n* High CPU based on given -Nx\r\n * CPU won't be 100% if you gave x smaller than available hardware threads of your platform.\r\n* Does NOT reproduce if TVar/retry is replaced by MVar\r\n* Reproduced with GHC 8.4.2 (macOS High Sierra (10.13.4))\r\n* Reproduced with GHC 8.4.2 (Docker for Mac Version 18.03.1-ce-mac65)\r\n* Reproduced with ghc-8.5.20180506 (Docker for Mac Version 18.03.1-ce-mac65)\r\n\r\n\r\nMinimal reproducing code here. (You can find more verbose code on the above github repo.)\r\n{{{#!hs\r\nmain :: IO ()\r\nmain = do\r\n let volume = 1000\r\n forM_ [1..1000] $ \\i -> do\r\n putStrFlush $ show i ++ \" \"\r\n\r\n -- Spawn massive number of threads.\r\n threads <- replicateM volume $ do\r\n trigger <- newTVarIO False\r\n tid <- forkIO $ void $ atomically $ do\r\n t <- readTVar trigger\r\n if t then pure t else retry\r\n pure (trigger, tid)\r\n\r\n -- Make sure all threads are spawned.\r\n threadDelay 30000\r\n\r\n -- Let threads start to exit normally.\r\n forkIO $ forM_ threads $ \\(trigger, _) -> threadDelay 1 *> atomically (writeTVar trigger True)\r\n\r\n -- Concurrently kill threads in order to create race.\r\n -- TMVar operation and asynchronous exception can hit same thread simultaneously.\r\n -- Adjust threadDelay if you don't reproduce very well.\r\n threadDelay 1000\r\n forM_ threads $ \\(_, tid) -> do\r\n putCharFlush 'A'\r\n killThread tid -- When the issue reproduced, this killThread doesn't return.\r\n putCharFlush '\\b'\r\n}}}\r\n\r\nThis program intentionally creates race condition between asynchronous exception\r\nand unblocking operation of `retry` on TVar. From one side, a `writeTVar\r\ntrigger True` is attempted from external thread while target thread is blocking\r\nat `retry` on the same `TVar`. On the other side, an asynchronous exception\r\n`ThreadKilled` is thrown by yet another external thread to the same target\r\nthread.\r\n\r\nIn other word, it attempts to kill a thread about to unblock.\r\n\r\nI guess when the above two operation hit the same thread at the same time in\r\nparallel in SMP environment, GHC runtime falls into high CPU.\r\n","type_of_failure":"OtherFailure","blocking":[]} -->8.6.1Ömer Sinan AğacanÖmer Sinan Ağacanhttps://gitlab.haskell.org/ghc/ghc/-/issues/15060outofmem testcase sometimes crashes on i3862019-07-07T18:14:28ZBen Gamarioutofmem testcase sometimes crashes on i386Specifically in the `normal` way with,
```
Actual stderr output differs from expected:
--- ./rts/outofmem.run/outofmem.stderr-ws-32.normalised 2018-04-19 13:57:35.386373403 +0000
+++ ./rts/outofmem.run/outofmem.run.stderr.normalised 201...Specifically in the `normal` way with,
```
Actual stderr output differs from expected:
--- ./rts/outofmem.run/outofmem.stderr-ws-32.normalised 2018-04-19 13:57:35.386373403 +0000
+++ ./rts/outofmem.run/outofmem.run.stderr.normalised 2018-04-19 13:57:35.386373403 +0000
@@ -1 +1,4 @@
-outofmem: out of memory (requested 1074790400 bytes)
+outofmem: internal getMBlock: mmap: Invalid argument
+ (GHC version 8.5.20180419 for i386_unknown_linux)
+ Please report this as a GHC bug: http://www.haskell.org/ghc/reportabug
+Aborted
*** unexpected failure for outofmem(normal)
```
<details><summary>Trac metadata</summary>
| Trac field | Value |
| ---------------------- | -------------- |
| Version | 8.2.2 |
| Type | Bug |
| TypeOfFailure | OtherFailure |
| Priority | normal |
| Resolution | Unresolved |
| Component | Runtime System |
| Test case | |
| Differential revisions | |
| BlockedBy | |
| Related | |
| Blocking | |
| CC | |
| Operating system | |
| Architecture | |
</details>
<!-- {"blocked_by":[],"summary":"outofmem testcase sometimes crashes on i386","status":"New","operating_system":"","component":"Runtime System","related":[],"milestone":"8.6.1","resolution":"Unresolved","owner":{"tag":"Unowned"},"version":"8.2.2","keywords":[],"differentials":[],"test_case":"","architecture":"","cc":[""],"type":"Bug","description":"Specifically in the `normal` way with,\r\n{{{\r\nActual stderr output differs from expected:\r\n--- ./rts/outofmem.run/outofmem.stderr-ws-32.normalised\t2018-04-19 13:57:35.386373403 +0000\r\n+++ ./rts/outofmem.run/outofmem.run.stderr.normalised\t2018-04-19 13:57:35.386373403 +0000\r\n@@ -1 +1,4 @@\r\n-outofmem: out of memory (requested 1074790400 bytes)\r\n+outofmem: internal getMBlock: mmap: Invalid argument\r\n+ (GHC version 8.5.20180419 for i386_unknown_linux)\r\n+ Please report this as a GHC bug: http://www.haskell.org/ghc/reportabug\r\n+Aborted\r\n*** unexpected failure for outofmem(normal)\r\n}}}","type_of_failure":"OtherFailure","blocking":[]} -->8.6.1https://gitlab.haskell.org/ghc/ghc/-/issues/15049NUMA not detected on Win10 with AMD Ryzen Threadripper2019-07-07T18:14:33ZkanetwNUMA not detected on Win10 with AMD Ryzen ThreadripperAMD's Threadripper can be configured to be a 1-socket 2-die NUMA system:
Coreinfo.exe:
```
Logical to Physical Processor Map:
**------------------------------ Physical Processor 0 (Hyperthreaded)
--**---------------------------- Phys...AMD's Threadripper can be configured to be a 1-socket 2-die NUMA system:
Coreinfo.exe:
```
Logical to Physical Processor Map:
**------------------------------ Physical Processor 0 (Hyperthreaded)
--**---------------------------- Physical Processor 1 (Hyperthreaded)
----**-------------------------- Physical Processor 2 (Hyperthreaded)
------**------------------------ Physical Processor 3 (Hyperthreaded)
--------**---------------------- Physical Processor 4 (Hyperthreaded)
----------**-------------------- Physical Processor 5 (Hyperthreaded)
------------**------------------ Physical Processor 6 (Hyperthreaded)
--------------**---------------- Physical Processor 7 (Hyperthreaded)
----------------**-------------- Physical Processor 8 (Hyperthreaded)
------------------**------------ Physical Processor 9 (Hyperthreaded)
--------------------**---------- Physical Processor 10 (Hyperthreaded)
----------------------**-------- Physical Processor 11 (Hyperthreaded)
------------------------**------ Physical Processor 12 (Hyperthreaded)
--------------------------**---- Physical Processor 13 (Hyperthreaded)
----------------------------**-- Physical Processor 14 (Hyperthreaded)
------------------------------** Physical Processor 15 (Hyperthreaded)
Logical Processor to Socket Map:
******************************** Socket 0
Logical Processor to NUMA Node Map:
****************---------------- NUMA Node 0
----------------**************** NUMA Node 1
Approximate Cross-NUMA Node Access Cost (relative to fastest):
00 01
00: 1.0 1.4
01: 1.7 1.2
```
However, ghc-8.4.1 doesn't detect the NUMA topology when running with `--numa`
```
ghc.EXE: --numa: OS reports NUMA is not available
```
I haven't attempted dualbooting into Linux, yet, so I can't tell whether it's a Windows-only issue.
<details><summary>Trac metadata</summary>
| Trac field | Value |
| ---------------------- | -------------- |
| Version | 8.4.1 |
| Type | Bug |
| TypeOfFailure | OtherFailure |
| Priority | normal |
| Resolution | Unresolved |
| Component | Runtime System |
| Test case | |
| Differential revisions | |
| BlockedBy | |
| Related | |
| Blocking | |
| CC | |
| Operating system | |
| Architecture | |
</details>
<!-- {"blocked_by":[],"summary":"NUMA not detected on Win10 with AMD Ryzen Threadripper","status":"New","operating_system":"","component":"Runtime System","related":[],"milestone":"8.6.1","resolution":"Unresolved","owner":{"tag":"Unowned"},"version":"8.4.1","keywords":[],"differentials":[],"test_case":"","architecture":"","cc":[""],"type":"Bug","description":"AMD's Threadripper can be configured to be a 1-socket 2-die NUMA system:\r\n\r\nCoreinfo.exe:\r\n{{{\r\nLogical to Physical Processor Map:\r\n**------------------------------ Physical Processor 0 (Hyperthreaded)\r\n--**---------------------------- Physical Processor 1 (Hyperthreaded)\r\n----**-------------------------- Physical Processor 2 (Hyperthreaded)\r\n------**------------------------ Physical Processor 3 (Hyperthreaded)\r\n--------**---------------------- Physical Processor 4 (Hyperthreaded)\r\n----------**-------------------- Physical Processor 5 (Hyperthreaded)\r\n------------**------------------ Physical Processor 6 (Hyperthreaded)\r\n--------------**---------------- Physical Processor 7 (Hyperthreaded)\r\n----------------**-------------- Physical Processor 8 (Hyperthreaded)\r\n------------------**------------ Physical Processor 9 (Hyperthreaded)\r\n--------------------**---------- Physical Processor 10 (Hyperthreaded)\r\n----------------------**-------- Physical Processor 11 (Hyperthreaded)\r\n------------------------**------ Physical Processor 12 (Hyperthreaded)\r\n--------------------------**---- Physical Processor 13 (Hyperthreaded)\r\n----------------------------**-- Physical Processor 14 (Hyperthreaded)\r\n------------------------------** Physical Processor 15 (Hyperthreaded)\r\n\r\nLogical Processor to Socket Map:\r\n******************************** Socket 0\r\n\r\nLogical Processor to NUMA Node Map:\r\n****************---------------- NUMA Node 0\r\n----------------**************** NUMA Node 1\r\n\r\nApproximate Cross-NUMA Node Access Cost (relative to fastest):\r\n 00 01\r\n00: 1.0 1.4\r\n01: 1.7 1.2\r\n}}}\r\n\r\nHowever, ghc-8.4.1 doesn't detect the NUMA topology when running with `--numa`\r\n{{{\r\nghc.EXE: --numa: OS reports NUMA is not available\r\n}}}\r\n\r\nI haven't attempted dualbooting into Linux, yet, so I can't tell whether it's a Windows-only issue.","type_of_failure":"OtherFailure","blocking":[]} -->8.6.1kanetwkanetwhttps://gitlab.haskell.org/ghc/ghc/-/issues/14900Calling isByteArrayPinned# on compact ByteArray2022-12-27T16:11:44ZAndrew MartinCalling isByteArrayPinned# on compact ByteArrayThe documentation explaining the relationship between pinnedness and compact regions is incomplete. From `Data.Compact`:
> Pinned ByteArray\# objects cannot be compacted. This is for a good reason: the memory is pinned so that it can be...The documentation explaining the relationship between pinnedness and compact regions is incomplete. From `Data.Compact`:
> Pinned ByteArray\# objects cannot be compacted. This is for a good reason: the memory is pinned so that it can be referenced by address (the address might be stored in a C data structure, for example), so we can't make a copy of it to store in the Compact.
This is half-way true since it only considers one a the ways in which the GHC runtime tracks the pinnedness of an object. From experimenting with compact regions, it seems like there are two different notions of pinnedness:
1. Did the user explicitly ask for the `ByteArray` to be pinned?
1. Is the `ByteArray` pinned?
If (1) is true, then (2) must always be true, but if (1) is false, then could be true or false. `ByteArray`s over 3KB are pinned, and `ByteArray`s under 3KB are not (or somewhere around that number).
With that background information in place, here's the scenario I've encountered:
```
{-# LANGUAGE MagicHash #-}
import Data.Primitive
import Data.Compact
import GHC.Int
import GHC.Prim
main :: IO ()
main = do
ByteArray arr1# <- fmap getCompact $ newByteArray 65000 >>= unsafeFreezeByteArray >>= compact
ByteArray arr2# <- newByteArray 65000 >>= unsafeFreezeByteArray
print (I# (isByteArrayPinned# arr1#))
print (I# (isByteArrayPinned# arr2#))
putStrLn "Finished"
```
When compiled and run, this gives:
```
0
1
Finished
```
We can see that the 65KiB `ByteArray` that was not compacted let's the user know that it is pinned. The compacted `ByteArray` claims to not be pinned, but this is not true. The docs in `Data.Compact` claim:
> Data in a compact doesn't ever move, so compacting data is also a way to pin arbitrary data structures in memory.
I propose that the behavior of `compact` be modified to accurately convey the pinnedness of the `ByteArray`s that copies. This would mean that even small, previously unpinned, `ByteArray`s would also be designated as pinned. It's a small change, but it makes the information the runtime gives us more accurate. This is occasionally handy when dealing with the FFI.
<details><summary>Trac metadata</summary>
| Trac field | Value |
| ---------------------- | ---------------- |
| Version | 8.2.2 |
| Type | Bug |
| TypeOfFailure | OtherFailure |
| Priority | normal |
| Resolution | Unresolved |
| Component | Compiler |
| Test case | |
| Differential revisions | |
| BlockedBy | |
| Related | |
| Blocking | |
| CC | ezyang, simonmar |
| Operating system | |
| Architecture | |
</details>
<!-- {"blocked_by":[],"summary":"Calling isByteArrayPinned# on compact ByteArray","status":"New","operating_system":"","component":"Compiler","related":[],"milestone":"","resolution":"Unresolved","owner":{"tag":"Unowned"},"version":"8.2.2","keywords":[],"differentials":[],"test_case":"","architecture":"","cc":["ezyang","simonmar"],"type":"Bug","description":"The documentation explaining the relationship between pinnedness and compact regions is incomplete. From `Data.Compact`:\r\n\r\n> Pinned ByteArray# objects cannot be compacted. This is for a good reason: the memory is pinned so that it can be referenced by address (the address might be stored in a C data structure, for example), so we can't make a copy of it to store in the Compact.\r\n\r\nThis is half-way true since it only considers one a the ways in which the GHC runtime tracks the pinnedness of an object. From experimenting with compact regions, it seems like there are two different notions of pinnedness:\r\n\r\n1. Did the user explicitly ask for the `ByteArray` to be pinned?\r\n2. Is the `ByteArray` pinned?\r\n\r\nIf (1) is true, then (2) must always be true, but if (1) is false, then could be true or false. `ByteArray`s over 3KB are pinned, and `ByteArray`s under 3KB are not (or somewhere around that number).\r\n\r\nWith that background information in place, here's the scenario I've encountered:\r\n\r\n{{{\r\n{-# LANGUAGE MagicHash #-}\r\n\r\nimport Data.Primitive\r\nimport Data.Compact\r\nimport GHC.Int\r\nimport GHC.Prim\r\n\r\nmain :: IO ()\r\nmain = do\r\n ByteArray arr1# <- fmap getCompact $ newByteArray 65000 >>= unsafeFreezeByteArray >>= compact\r\n ByteArray arr2# <- newByteArray 65000 >>= unsafeFreezeByteArray\r\n print (I# (isByteArrayPinned# arr1#))\r\n print (I# (isByteArrayPinned# arr2#))\r\n putStrLn \"Finished\"\r\n}}}\r\n\r\nWhen compiled and run, this gives:\r\n\r\n{{{\r\n0\r\n1\r\nFinished\r\n}}}\r\n\r\nWe can see that the 65KiB `ByteArray` that was not compacted let's the user know that it is pinned. The compacted `ByteArray` claims to not be pinned, but this is not true. The docs in `Data.Compact` claim:\r\n\r\n> Data in a compact doesn't ever move, so compacting data is also a way to pin arbitrary data structures in memory.\r\n\r\nI propose that the behavior of `compact` be modified to accurately convey the pinnedness of the `ByteArray`s that copies. This would mean that even small, previously unpinned, `ByteArray`s would also be designated as pinned. It's a small change, but it makes the information the runtime gives us more accurate. This is occasionally handy when dealing with the FFI.","type_of_failure":"OtherFailure","blocking":[]} -->8.6.1Simon MarlowSimon Marlowhttps://gitlab.haskell.org/ghc/ghc/-/issues/14886Add max GC pause to GHC.Stats/-t --machine-readable2019-07-07T18:15:11ZMateusz KowalczykAdd max GC pause to GHC.Stats/-t --machine-readableSometimes we're interested in latency of our programs. Often the upper highest percentiles are by garbage collector pauses due to stop-the-world implementation.
It seems that currently the only way to see the highest pause is to use the...Sometimes we're interested in latency of our programs. Often the upper highest percentiles are by garbage collector pauses due to stop-the-world implementation.
It seems that currently the only way to see the highest pause is to use the `-s` flag or alternatively `-S`. This is problematic in two scenarios:
1. You don't have a human at the monitor parsing the `-s` output.
1. You are unable to get the maximum pause statistic programmatically inside the program itself.
We do have `-t -machine-readable` which is nearly what I'd want but missing the actual pause information. Either that should include all the extra info from `-s` or `-s` should have machine-readable output format too.
There is the `-S` option which will print garbage collections as they happen. We could parse and manually track the highest pause time but it requires an external process and a parser. If you want to get that information back into the process then you have to do even more work.
`GHC.Stats` is of no help. It does not track or expose pause values. Even the "last GC info" APIs don't provide this (and they are not a solution anyway as you can only get info on last GC).
I would therefore like to request two similar features:
- Add relevant information from `-s` to `-t` machine readable output or add machine readable format option to `-s`.
- Add maximum pause information to GHC.Stats interface.
<details><summary>Trac metadata</summary>
| Trac field | Value |
| ---------------------- | -------------- |
| Version | 8.2.2 |
| Type | FeatureRequest |
| TypeOfFailure | OtherFailure |
| Priority | normal |
| Resolution | Unresolved |
| Component | Compiler |
| Test case | |
| Differential revisions | |
| BlockedBy | |
| Related | |
| Blocking | |
| CC | |
| Operating system | |
| Architecture | |
</details>
<!-- {"blocked_by":[],"summary":"Add max GC pause to GHC.Stats/-t --machine-readable","status":"New","operating_system":"","component":"Compiler","related":[],"milestone":"","resolution":"Unresolved","owner":{"tag":"Unowned"},"version":"8.2.2","keywords":[],"differentials":[],"test_case":"","architecture":"","cc":[""],"type":"FeatureRequest","description":"Sometimes we're interested in latency of our programs. Often the upper highest percentiles are by garbage collector pauses due to stop-the-world implementation.\r\n\r\nIt seems that currently the only way to see the highest pause is to use the {{{-s}}} flag or alternatively {{{-S}}}. This is problematic in two scenarios:\r\n\r\n1. You don't have a human at the monitor parsing the {{{-s}}} output.\r\n2. You are unable to get the maximum pause statistic programmatically inside the program itself.\r\n\r\nWe do have {{{-t -machine-readable}}} which is nearly what I'd want but missing the actual pause information. Either that should include all the extra info from {{{-s}}} or {{{-s}}} should have machine-readable output format too.\r\n\r\nThere is the {{{-S}}} option which will print garbage collections as they happen. We could parse and manually track the highest pause time but it requires an external process and a parser. If you want to get that information back into the process then you have to do even more work.\r\n\r\n{{{GHC.Stats}}} is of no help. It does not track or expose pause values. Even the \"last GC info\" APIs don't provide this (and they are not a solution anyway as you can only get info on last GC).\r\n\r\nI would therefore like to request two similar features:\r\n\r\n* Add relevant information from {{{-s}}} to {{{-t}}} machine readable output or add machine readable format option to {{{-s}}}.\r\n* Add maximum pause information to GHC.Stats interface.","type_of_failure":"OtherFailure","blocking":[]} -->8.6.1https://gitlab.haskell.org/ghc/ghc/-/issues/14702Runtime is keeping hold to too much memory2021-03-10T15:33:41ZblazeRuntime is keeping hold to too much memoryCurrently runtime only releases memory to OS when it has more than 4x of the amount used. This leads to heavy memory overuse when process is run using many processes and large nurseries. For example, if process is run on 16 CPUs with eac...Currently runtime only releases memory to OS when it has more than 4x of the amount used. This leads to heavy memory overuse when process is run using many processes and large nurseries. For example, if process is run on 16 CPUs with each having 32MB nursery, it will use total 512MB allocation area. This will push "release boundary" to the 2GB. If process is ever to allocate that much memory for a short term use, it will never be given back to OS and effectively wasted.
Nursery and other non-growing data areas should be excluded from this multiplier. Runtime should only keep an extra memory proportional to heap allocation.
<details><summary>Trac metadata</summary>
| Trac field | Value |
| ---------------------- | -------------- |
| Version | 8.2.2 |
| Type | Bug |
| TypeOfFailure | OtherFailure |
| Priority | normal |
| Resolution | Unresolved |
| Component | Runtime System |
| Test case | |
| Differential revisions | |
| BlockedBy | |
| Related | |
| Blocking | |
| CC | |
| Operating system | |
| Architecture | |
</details>
<!-- {"blocked_by":[],"summary":"Runtime is keeping hold to too much memory","status":"New","operating_system":"","component":"Runtime System","related":[],"milestone":"","resolution":"Unresolved","owner":{"tag":"Unowned"},"version":"8.2.2","keywords":[],"differentials":[],"test_case":"","architecture":"","cc":[""],"type":"Bug","description":"Currently runtime only releases memory to OS when it has more than 4x of the amount used. This leads to heavy memory overuse when process is run using many processes and large nurseries. For example, if process is run on 16 CPUs with each having 32MB nursery, it will use total 512MB allocation area. This will push \"release boundary\" to the 2GB. If process is ever to allocate that much memory for a short term use, it will never be given back to OS and effectively wasted.\r\n\r\nNursery and other non-growing data areas should be excluded from this multiplier. Runtime should only keep an extra memory proportional to heap allocation.","type_of_failure":"OtherFailure","blocking":[]} -->8.6.1blazeblazehttps://gitlab.haskell.org/ghc/ghc/-/issues/14660Improve +RTS -t --machine-readable2019-07-07T18:16:07ZDouglas Wilsondouglas@well-typed.comImprove +RTS -t --machine-readableThe output of +RTS -t --machine-readable is a shown Haskell list, so we can easily add fields. Let's add everything shown by +RTS -s .
I've posted a rough patch that does that at [D4303](https://phabricator.haskell.org/D4303).
If this i...The output of +RTS -t --machine-readable is a shown Haskell list, so we can easily add fields. Let's add everything shown by +RTS -s .
I've posted a rough patch that does that at [D4303](https://phabricator.haskell.org/D4303).
If this is a good idea, I'd like to also add the following:
- a data structure and parsing function to GHC.Stats in base.
- The ghc version as a field in the --machine-readable output
- The rts ways as a field in the --machine-readable output
<details><summary>Trac metadata</summary>
| Trac field | Value |
| ---------------------- | -------------- |
| Version | 8.3 |
| Type | Task |
| TypeOfFailure | OtherFailure |
| Priority | normal |
| Resolution | Unresolved |
| Component | Runtime System |
| Test case | |
| Differential revisions | |
| BlockedBy | |
| Related | |
| Blocking | |
| CC | |
| Operating system | |
| Architecture | |
</details>
<!-- {"blocked_by":[],"summary":"Improve +RTS -t --machine-readable","status":"New","operating_system":"","component":"Runtime System","related":[],"milestone":"","resolution":"Unresolved","owner":{"tag":"Unowned"},"version":"8.3","keywords":[],"differentials":[],"test_case":"","architecture":"","cc":[""],"type":"Task","description":"The output of +RTS -t --machine-readable is a shown Haskell list, so we can easily add fields. Let's add everything shown by +RTS -s .\r\nI've posted a rough patch that does that at Phab:D4303.\r\n\r\nIf this is a good idea, I'd like to also add the following:\r\n* a data structure and parsing function to GHC.Stats in base.\r\n* The ghc version as a field in the --machine-readable output\r\n* The rts ways as a field in the --machine-readable output\r\n","type_of_failure":"OtherFailure","blocking":[]} -->8.6.1https://gitlab.haskell.org/ghc/ghc/-/issues/14611Support LIBRARY_PATH and LD_LIBRARY_PATH in rts2019-07-07T18:16:18ZTamar ChristinaSupport LIBRARY_PATH and LD_LIBRARY_PATH in rtsTo complement the C compiler's usage of these, the Haskell runtime loader should use `LD_LIBRARY_PATH` to load shared libraries and the linker should use `LIBRARY_PATH` to find libraries.
<details><summary>Trac metadata</summary>
| Tra...To complement the C compiler's usage of these, the Haskell runtime loader should use `LD_LIBRARY_PATH` to load shared libraries and the linker should use `LIBRARY_PATH` to find libraries.
<details><summary>Trac metadata</summary>
| Trac field | Value |
| ---------------------- | ----------------------- |
| Version | 8.2.2 |
| Type | FeatureRequest |
| TypeOfFailure | OtherFailure |
| Priority | normal |
| Resolution | Unresolved |
| Component | Runtime System (Linker) |
| Test case | |
| Differential revisions | |
| BlockedBy | |
| Related | |
| Blocking | |
| CC | |
| Operating system | |
| Architecture | |
</details>
<!-- {"blocked_by":[],"summary":"Support LIBRARY_PATH and LD_LIBRARY_PATH in rts","status":"New","operating_system":"","component":"Runtime System (Linker)","related":[],"milestone":"8.6.1","resolution":"Unresolved","owner":{"tag":"Unowned"},"version":"8.2.2","keywords":[],"differentials":[],"test_case":"","architecture":"","cc":[""],"type":"FeatureRequest","description":"To complement the C compiler's usage of these, the Haskell runtime loader should use `LD_LIBRARY_PATH` to load shared libraries and the linker should use `LIBRARY_PATH` to find libraries.","type_of_failure":"OtherFailure","blocking":[]} -->8.6.1Tamar ChristinaTamar Christinahttps://gitlab.haskell.org/ghc/ghc/-/issues/14492Tiered memory allocation restricts available memory2019-07-07T18:16:49ZunodeTiered memory allocation restricts available memoryThis was first noticed when running a binary produced with GHC 8.0.2 through a queueing system (Sun Grid Engine) that limits available memory by setting `ulimit -v`.
When running with \*\*unlimited\*\* address space, the binary reserves...This was first noticed when running a binary produced with GHC 8.0.2 through a queueing system (Sun Grid Engine) that limits available memory by setting `ulimit -v`.
When running with \*\*unlimited\*\* address space, the binary reserves 1TB of RAM and everything works as expected.
When running with \*\*limited\*\* address space, reservation happens in powers of 2 leading to significant unused memory at higher memory values.
The tests are summarized in the following table:
```
Limit VIRT Unused (all in GB)
8 4 8
10 8 2
16 8 8
32 16 16
34 32 2
64 32 32
66 64 2
128 64 64
130 128 2
256 128 128
258 256 2
512 256 256
514 512 2
1024 512 512
```
In the last line, setting `ulimit -v` to 1024GB (1073741824) causes the binary to only be able to reserve 512GB.
If at runtime the software needs more than 512GB it will abort with "out of memory" instead of expanding to as close to the specified ulimit as possible.
PS: Is there any way to make the "out of memory" error a little bit more informative? Something that informs the user of how much memory was already being used and how much more was going to be reserved when the failure was seen?
<details><summary>Trac metadata</summary>
| Trac field | Value |
| ---------------------- | -------------- |
| Version | 8.0.2 |
| Type | Bug |
| TypeOfFailure | OtherFailure |
| Priority | normal |
| Resolution | Unresolved |
| Component | Runtime System |
| Test case | |
| Differential revisions | |
| BlockedBy | |
| Related | |
| Blocking | |
| CC | |
| Operating system | |
| Architecture | |
</details>
<!-- {"blocked_by":[],"summary":"Tiered memory allocation restricts available memory","status":"New","operating_system":"","component":"Runtime System","related":[],"milestone":"","resolution":"Unresolved","owner":{"tag":"Unowned"},"version":"8.0.2","keywords":["memory","ulimit"],"differentials":[],"test_case":"","architecture":"","cc":[""],"type":"Bug","description":"This was first noticed when running a binary produced with GHC 8.0.2 through a queueing system (Sun Grid Engine) that limits available memory by setting `ulimit -v`.\r\n\r\nWhen running with **unlimited** address space, the binary reserves 1TB of RAM and everything works as expected.\r\nWhen running with **limited** address space, reservation happens in powers of 2 leading to significant unused memory at higher memory values.\r\n\r\nThe tests are summarized in the following table:\r\n\r\n{{{\r\n Limit VIRT Unused (all in GB)\r\n 8 4 8\r\n 10 8 2\r\n 16 8 8\r\n 32 16 16\r\n 34 32 2\r\n 64 32 32\r\n 66 64 2\r\n 128 64 64\r\n 130 128 2\r\n 256 128 128\r\n 258 256 2\r\n 512 256 256\r\n 514 512 2\r\n 1024 512 512\r\n}}}\r\n\r\nIn the last line, setting `ulimit -v` to 1024GB (1073741824) causes the binary to only be able to reserve 512GB.\r\nIf at runtime the software needs more than 512GB it will abort with \"out of memory\" instead of expanding to as close to the specified ulimit as possible.\r\n\r\nPS: Is there any way to make the \"out of memory\" error a little bit more informative? Something that informs the user of how much memory was already being used and how much more was going to be reserved when the failure was seen?\r\n","type_of_failure":"OtherFailure","blocking":[]} -->8.6.1Ben GamariBen Gamarihttps://gitlab.haskell.org/ghc/ghc/-/issues/14456Windows runtime linker failure with icuuc2019-07-07T18:16:58ZRyan ScottWindows runtime linker failure with icuucFirst, install `mingw-w64-x86_64-icu` (I'm using version 58.2-2):
```
$ pacman -S mingw-w64-x86_64-icu
```
Now take this file:
```hs
module Main where
import Data.Int
import Foreign
import Foreign.C.String
import Foreign.C.Types
typ...First, install `mingw-w64-x86_64-icu` (I'm using version 58.2-2):
```
$ pacman -S mingw-w64-x86_64-icu
```
Now take this file:
```hs
module Main where
import Data.Int
import Foreign
import Foreign.C.String
import Foreign.C.Types
type UErrorCode = CInt
u_ZERO_ERROR :: UErrorCode
u_ZERO_ERROR = 0
foreign import ccall "ucnv_open_58"
c_ucnv_open :: CString -> Ptr UErrorCode -> IO (Ptr ())
foreign import ccall "ucnv_getMaxCharSize_58"
c_ucnv_getMaxCharSize :: Ptr () -> IO Int8
main :: IO ()
main = with u_ZERO_ERROR $ \status -> do
conv <- c_ucnv_open nullPtr status
c_ucnv_getMaxCharSize conv >>= print
```
GHC is able to compile this successfully:
```
$ ghc -licuuc Bug2.hs
[1 of 1] Compiling Main ( Bug2.hs, Bug2.o )
Linking Bug2.exe ...
$ ./Bug2.exe
1
```
But GHCi is unable to accomplish the same thing:
```
$ runghc -licuuc Bug2.hs
ghc.exe: | C:\Users\RyanGlScott\Documents\Hacking\Haskell\Bug2.o: unknown symbol `ucnv_open_58'
Bug2.hs:
```
Phyx- and I discussed this briefly on IRC. He suspected that the fact that `C:\Windows\System32` now contains its own copy of `icuuc.dll` is contributing to the issue.8.6.1Tamar ChristinaTamar Christinahttps://gitlab.haskell.org/ghc/ghc/-/issues/143468.2.1 regression: heap corruption after safe foreign calls2022-07-06T16:47:17Zandrewchen8.2.1 regression: heap corruption after safe foreign callsTest case: (compile with ghc 8.2.1 and -threaded option)
```haskell
module Main where
import Control.Concurrent
import Control.Monad
import Data.Word
import Foreign.Marshal.Alloc
import Foreign.Ptr
import Foreign.Storable
foreign impo...Test case: (compile with ghc 8.2.1 and -threaded option)
```haskell
module Main where
import Control.Concurrent
import Control.Monad
import Data.Word
import Foreign.Marshal.Alloc
import Foreign.Ptr
import Foreign.Storable
foreign import ccall safe "test"
c_test :: Ptr Word32 -> IO ()
main :: IO ()
main = do
replicateM_ 1000 $ threadDelay 1000
_ <- forkIO $ forever $ threadDelay 100
allocaBytes 4 $ \p -> forever $ do
c_test p
x <- peek p
unless (x == 0xDEADBEEF) $ putStrLn "value mismatch"
```
```c
void test(unsigned int *buf) {
*buf = 0xDEADBEEF;
}
```
On my machine, it detects a few value mismatches before crashing with sigsegv.
```
$ time ./.stack-work/install/x86_64-linux-nopie/nightly-2017-10-10/8.2.1/bin/bug
value mismatch
value mismatch
value mismatch
value mismatch
zsh: segmentation fault (core dumped) ./.stack-work/install/x86_64-linux-nopie/nightly-2017-10-10/8.2.1/bin/bug
./.stack-work/install/x86_64-linux-nopie/nightly-2017-10-10/8.2.1/bin/bug 2.11s user 0.25s system 66% cpu 3.543 total
```
I believe this is what is causing crashes in xmobar. See discussion: https://github.com/jaor/xmobar/issues/310. Note that the crash in xmobar still happens without -threaded option, while this example only breaks when compiled with -threaded.
<details><summary>Trac metadata</summary>
| Trac field | Value |
| ---------------------- | -------------- |
| Version | 8.2.1 |
| Type | Bug |
| TypeOfFailure | OtherFailure |
| Priority | highest |
| Resolution | Unresolved |
| Component | Runtime System |
| Test case | |
| Differential revisions | |
| BlockedBy | |
| Related | |
| Blocking | |
| CC | |
| Operating system | |
| Architecture | |
</details>
<!-- {"blocked_by":[],"summary":"8.2.1 regression: heap corruption after safe foreign calls","status":"New","operating_system":"","component":"Runtime System","related":[],"milestone":"","resolution":"Unresolved","owner":{"tag":"Unowned"},"version":"8.2.1","keywords":[],"differentials":[],"test_case":"","architecture":"","cc":[""],"type":"Bug","description":"Test case: (compile with ghc 8.2.1 and -threaded option)\r\n{{{#!haskell\r\nmodule Main where\r\n\r\nimport Control.Concurrent\r\nimport Control.Monad\r\nimport Data.Word\r\nimport Foreign.Marshal.Alloc\r\nimport Foreign.Ptr\r\nimport Foreign.Storable\r\n\r\nforeign import ccall safe \"test\"\r\n c_test :: Ptr Word32 -> IO ()\r\n\r\nmain :: IO ()\r\nmain = do\r\n replicateM_ 1000 $ threadDelay 1000\r\n _ <- forkIO $ forever $ threadDelay 100\r\n allocaBytes 4 $ \\p -> forever $ do\r\n c_test p\r\n x <- peek p\r\n unless (x == 0xDEADBEEF) $ putStrLn \"value mismatch\"\r\n}}}\r\n{{{#!c\r\nvoid test(unsigned int *buf) {\r\n *buf = 0xDEADBEEF;\r\n}\r\n}}}\r\n\r\nOn my machine, it detects a few value mismatches before crashing with sigsegv.\r\n{{{\r\n$ time ./.stack-work/install/x86_64-linux-nopie/nightly-2017-10-10/8.2.1/bin/bug\r\nvalue mismatch\r\nvalue mismatch\r\nvalue mismatch\r\nvalue mismatch\r\nzsh: segmentation fault (core dumped) ./.stack-work/install/x86_64-linux-nopie/nightly-2017-10-10/8.2.1/bin/bug\r\n./.stack-work/install/x86_64-linux-nopie/nightly-2017-10-10/8.2.1/bin/bug 2.11s user 0.25s system 66% cpu 3.543 total\r\n}}}\r\n\r\nI believe this is what is causing crashes in xmobar. See discussion: https://github.com/jaor/xmobar/issues/310. Note that the crash in xmobar still happens without -threaded option, while this example only breaks when compiled with -threaded.","type_of_failure":"OtherFailure","blocking":[]} -->8.6.1Ben GamariBen Gamarihttps://gitlab.haskell.org/ghc/ghc/-/issues/14006Heap profiling ghc gives hp2ps error2019-07-07T18:18:54ZDavid FeuerHeap profiling ghc gives hp2ps errorRunning GHC with heap profiling and then running `hp2ps` on the result gives
```
hp2ps: ghc-stage2.hp, line 17405, samples out of sequence
```
Reproduction (GHC commit 4700baaf8f9bf3c44a53a595d840c7c14cfd6f98)
```
cabal unpack skyligh...Running GHC with heap profiling and then running `hp2ps` on the result gives
```
hp2ps: ghc-stage2.hp, line 17405, samples out of sequence
```
Reproduction (GHC commit 4700baaf8f9bf3c44a53a595d840c7c14cfd6f98)
```
cabal unpack skylighting-0.1.1.5
cd skylighting-0.1.1.5
cabal install --enable-profiling --only-dependencies -fexecutable
cabal configure --with-ghc=$HOME/src/ghc-clean-july12/inplace/bin/ghc-stage2 \
--enable-profiling -fexecutable
cabal build
touch src/Skylighting/Syntax/Php.hs
/home/dfeuer/src/ghc-clean-july12/inplace/bin/ghc-stage2 --make -i \
-isrc -idist/build -O -prof -fprof-auto-exported -osuf p_o -hisuf \
p_hi -optPdist/build/autogen/cabal_macros.h -hide-all-packages \
-package-db dist/package.conf.inplace -package-id aeson-1.2.1.0 \
-package base-4.10.0.0 -package blaze-html-0.9.0.1 -package \
bytestring-0.10.8.2 -package case-insensitive-1.2.0.10 -package \
containers-0.5.10.2 -package directory-1.3.0.2 -package \
filepath-1.4.1.2 -package hxt-9.3.1.16 -package mtl-2.2.1 -package \
regex-pcre-builtin-0.94.4.8.8.35 -package safe-0.3.15 -package \
text-1.2.2.2 -package utf8-string-1.0.1.1 -XHaskell2010 \
Skylighting.Types Skylighting.Regex \
Skylighting.Tokenizer Skylighting.Syntax.Php -j1 +RTS \
-s -h -N1
hp2ps ghc-stage2.hp
```8.6.1https://gitlab.haskell.org/ghc/ghc/-/issues/13165Speed up the RTS hash table2020-12-21T04:09:30ZdobenourSpeed up the RTS hash tableThe RTS hash table is rather slow. Every lookup involves at least one indirect call (to get a hash code). It also uses separate chaining, which is itself slow.
Until recently, this didn't really matter, since the RTS hash table wasn't u...The RTS hash table is rather slow. Every lookup involves at least one indirect call (to get a hash code). It also uses separate chaining, which is itself slow.
Until recently, this didn't really matter, since the RTS hash table wasn't used for anything particularly performance critical other than `StableName` operations. However, it has since become the bottleneck when compacting with sharing, to the point that compacting without sharing is around 10x faster and is thus the default.
Fortunately, there are easy ways to make the hash table faster. These include:
- Use linear probing instead of separate chaining.
- Specialize for the case of pointer keys
- Don't use indirect calls for hashing
- Use a fast, universal hash function for pointers.
- Use SSE instructions where available to do fast searches.
- Minimize the number of indirections in the use of the table.
In both the case of the StablePtr table and that of compact regions, the keys of the table are not GC pointers, but the values are, so there needs to be a way to ensure that the GC handles the table correctly8.6.1https://gitlab.haskell.org/ghc/ghc/-/issues/12964Runtime regression to RTS change2019-07-07T18:24:26ZJoachim Breitnermail@joachim-breitner.deRuntime regression to RTS changeThe commit to fix #12799, namely changeset:1732d7ac43ca, seems to have introduced a runtime regression. The hidden and integer nofib benchmarks regress by 5.5% resp. 2.5%:
https://perf.haskell.org/ghc/\#revision/1732d7ac43ca578deca39ea5...The commit to fix #12799, namely changeset:1732d7ac43ca, seems to have introduced a runtime regression. The hidden and integer nofib benchmarks regress by 5.5% resp. 2.5%:
https://perf.haskell.org/ghc/\#revision/1732d7ac43ca578deca39ea5a63cbf34f3cd9dd5
I re-measured this commit and its parent once, and the numbers aid not change much.
The `hidden` regression seems to be offset almost completely by the later changeset:6f7ed1e51bf360621a3c2a447045ab3012f68575 while the `integer` regression got “fixed” by the later changeset:1732d7ac43ca578deca39ea5a63cbf34f3cd9dd5.
The perf builder is still new, so this might be bogus. But we should still check.
Also, maybe random changes to the RTS cause random changes to the code layout which surface here. Or there is a real cause.
Simon Marlow [suggested](https://phabricator.haskell.org/rGHC1732d7ac43ca#56419) to open a high priority ticket for this.
<details><summary>Trac metadata</summary>
| Trac field | Value |
| ---------------------- | -------------- |
| Version | 8.1 |
| Type | Bug |
| TypeOfFailure | OtherFailure |
| Priority | highest |
| Resolution | Unresolved |
| Component | Runtime System |
| Test case | |
| Differential revisions | |
| BlockedBy | |
| Related | |
| Blocking | |
| CC | simonmar |
| Operating system | |
| Architecture | |
</details>
<!-- {"blocked_by":[],"summary":"Runtime regression to RTS change","status":"New","operating_system":"","component":"Runtime System","related":[],"milestone":"8.2.1","resolution":"Unresolved","owner":{"tag":"Unowned"},"version":"8.1","keywords":[],"differentials":[],"test_case":"","architecture":"","cc":["simonmar"],"type":"Bug","description":"The commit to fix #12799, namely changeset:1732d7ac43ca, seems to have introduced a runtime regression. The hidden and integer nofib benchmarks regress by 5.5% resp. 2.5%:\r\n\r\nhttps://perf.haskell.org/ghc/#revision/1732d7ac43ca578deca39ea5a63cbf34f3cd9dd5\r\n\r\nI re-measured this commit and its parent once, and the numbers aid not change much.\r\n\r\nThe `hidden` regression seems to be offset almost completely by the later changeset:6f7ed1e51bf360621a3c2a447045ab3012f68575 while the `integer` regression got “fixed” by the later changeset:1732d7ac43ca578deca39ea5a63cbf34f3cd9dd5.\r\n\r\nThe perf builder is still new, so this might be bogus. But we should still check.\r\n\r\nAlso, maybe random changes to the RTS cause random changes to the code layout which surface here. Or there is a real cause.\r\n\r\nSimon Marlow [https://phabricator.haskell.org/rGHC1732d7ac43ca#56419 suggested] to open a high priority ticket for this.","type_of_failure":"OtherFailure","blocking":[]} -->8.6.1