STM exception annotation causes RTS assertion failure and can drop async exceptions
Commit e1ce1fc3ad38 makes `catchSTM` wrap the user handler in a
new `annotateSTM`:
```haskell
-- libraries/ghc-internal/src/GHC/Internal/STM.hs
catchSTM (STM m) handler = STM $ catchSTM# m handler'
where
handler' e = case fromException e of
Just e' -> unSTM (annotateSTM (WhileHandling e) (handler e'))
Nothing -> raiseIO# e
annotateSTM :: forall e a. ExceptionAnnotation e => e -> STM a -> STM a
annotateSTM ann (STM io) = STM (catch# io handler)
where
handler se = raiseIO# (addExceptionContext ann se)
```
`annotateSTM` is implemented with the *IO* primop `catch#`, which is incorrect.
We need to use `catchSTM#` instead.
## 1. `retry` in a handler trips an RTS assertion
`atomically (catchSTM act (\_ -> ... retry ...))` is a legitimate
program: `findRetryFrameHelper` explicitly handles `CATCH_STM_FRAME`
precisely because `retry` may propagate out of `catchSTM`. But with
this patch, the handler's `retry#` calls `findRetryFrameHelper` with
the freshly pushed `CATCH_FRAME` on the stack and hits the assertion.
The same assertion exists in `findAtomicallyFrameHelper`, reachable via
`stg_abort` when an `orElse` inside the handler fails to commit.
With F1.hs below, compiled with `-debug` against a master build
containing e1ce1fc3ad38 (10.1.20260710, x86_64-linux):
```
F1: internal error: ASSERTION FAILED: file rts/Schedule.c, line 3272
(GHC version 10.1.20260710 for x86_64_unknown_linux)
Please report this as a GHC bug: https://www.haskell.org/ghc/reportabug
Aborted (core dumped)
```
(Line 3272 is the `ASSERT(info->i.type != CATCH_FRAME)` in
`findRetryFrameHelper`'s default branch.) On a release RTS the default
branch just skips the frame, so `retry` semantics happen to survive
there — but the documented invariant is violated, and debug/validate
builds abort on ordinary user code.
<details><summary>F1.hs</summary>
```haskell
{-# LANGUAGE ScopedTypeVariables #-}
-- retry inside a catchSTM handler.
-- On an assertion-enabled RTS (compile with -debug) this trips
-- ASSERT(info->i.type != CATCH_FRAME) in findRetryFrameHelper.
-- Expected (correct) behaviour: prints "F1: completed".
import GHC.Conc
import Control.Exception
import Control.Concurrent.MVar
import Control.Monad (unless)
main :: IO ()
main = do
tv <- newTVarIO (0 :: Int)
done <- newEmptyMVar
_ <- forkIO $ do
atomically $
catchSTM (throwSTM (ErrorCall "boom"))
(\(_ :: SomeException) -> do
v <- readTVar tv
unless (v == 1) retry)
putMVar done ()
threadDelay 100000
atomically (writeTVar tv 1)
takeMVar done
putStrLn "F1: completed"
```
</details>
## 2. Async exceptions delivered during a handler can be lost
The `catchSTM` handler runs unmasked (the `CATCH_STM_FRAME` branch of
`stg_raisezh` does not set `TSO_BLOCKEX`, unlike the `CATCH_FRAME`
branch), so an async exception can arrive while it — and hence
`annotateSTM`'s `CATCH_FRAME` — is live. `raiseAsync`
(rts/RaiseAsync.c) walks the stack from the top and stops at the first
`CATCH_FRAME`, applying its handler directly; it never reaches the
`ATOMICALLY_FRAME` case, which is the one that aborts the transaction
and propagates async exceptions *unconditionally* — its comment for
the STM cases notes that "we do not care whether the transaction is
valid or not because its possible validity cannot have caused the
exception".
`annotateSTM`'s handler then re-raises via `raiseIO#`, i.e. as a
*synchronous* exception inside the still-active transaction. Now
`stg_raisezh` reaches the `ATOMICALLY_FRAME` and calls
`stmValidateNestOfTransactions`: if the transaction has been
invalidated concurrently, the RTS restarts the transaction and the
exception is dropped entirely.
Concretely: thread A runs `atomically (readTVar tv >> catchSTM m h)`
where `m` throws and handler `h` is executing; thread B writes `tv`
(invalidating A's read set) and calls `killThread A`. The
`ThreadKilled` is consumed by the transaction restart: `killThread`
returns, and A re-runs its transaction as if nothing happened. The
same applies to `timeout`. Before this patch there was no
`CATCH_FRAME` inside the transaction, so the async exception was
always propagated. Lost `killThread`/`timeout` exceptions mean
unkillable threads and timeouts that never fire.
F2.hs below demonstrates this deterministically (release RTS,
`-threaded`); on the same build as above it prints:
```
F2: BUG — thread survived killThread: transaction restarted, ThreadKilled dropped
```
With correct behaviour it would print
`F2: OK — thread died with: thread killed`.
<details><summary>F2.hs</summary>
```haskell
{-# LANGUAGE ScopedTypeVariables #-}
-- killThread delivered while a catchSTM handler is running, with the
-- enclosing transaction concurrently invalidated.
-- Expected (correct) behaviour: the target thread dies with ThreadKilled
-- ("F2: OK ..."). Buggy behaviour: the ThreadKilled is consumed by a
-- transaction restart and the thread survives ("F2: BUG ...").
import GHC.Conc
import Control.Exception
import Control.Concurrent.MVar
import Data.IORef
main :: IO ()
main = do
tv <- newTVarIO (0 :: Int)
started <- newEmptyMVar
result <- newEmptyMVar
stop <- newIORef False
t <- forkIO $ do
r <- try $ atomically $ do
v <- readTVar tv -- puts tv in the outer read set
catchSTM (throwSTM (ErrorCall "sync exception"))
(\(_ :: SomeException) ->
if v == 0
then do
unsafeIOToSTM (putMVar started ())
let loop = do
unsafeIOToSTM yield -- safe point for async delivery
s <- unsafeIOToSTM (readIORef stop)
if s then pure "handler survived killThread"
else loop
loop
else pure "transaction restarted, ThreadKilled dropped")
putMVar result (r :: Either SomeException String)
takeMVar started
atomically (writeTVar tv 1) -- invalidate t's transaction
killThread t -- returns once the exception was delivered
writeIORef stop True -- safety net so the loop always terminates
r <- takeMVar result
case r of
Left e -> putStrLn ("F2: OK — thread died with: " ++ show e)
Right s -> putStrLn ("F2: BUG — thread survived killThread: " ++ s)
```
</details>
<details><summary>run.sh</summary>
```sh
#!/bin/sh
# Reproduces the annotateSTM catch#-inside-STM bugs.
# Usage: GHC=/path/to/ghc sh run.sh (needs a GHC containing e1ce1fc3ad)
set -e
: "${GHC:=ghc}"
echo "== F1: retry inside a catchSTM handler (assertion RTS, -debug)"
$GHC -debug F1.hs -o F1 -v0
./F1 || echo "F1 exit: $?"
echo
echo "== F2: killThread during a catchSTM handler (normal threaded RTS)"
$GHC -threaded F2.hs -o F2 -v0
./F2 || echo "F2 exit: $?"
```
</details>
## Affected versions
The commit is on master since e1ce1fc3ad38 (in `ghc-10.0-start`; both
reproducers verified on 10.1.20260710 = d415f38a75) and was backported
to `ghc-9.14` as 154a4b5bac13, first shipped in 9.14.2-rc1 — it is not
in 9.14.1, so for the stable branch this is a new regression in the rc.
## Provenance
Found by [GlaHiCo](https://gitlab.haskell.org/sjakobi/GlaHiCo), an
LLM-based code-review pipeline, while reviewing the 9.14.2-rc1
backports. The published review of 154a4b5bac13 (master:
e1ce1fc3ad38) is at https://gitlab.haskell.org/sjakobi/GlaHiCo/-/snippets/6145
(findings f1 and f2).
The draft of this ticket was also written by an LLM agent.
The reproducers above were built and run, and both failures
confirmed, before filing.
issue