maxDmdType drops the implicit demands beyond the unfolding's depth, leaving a hole in the #26416 fix
<!--
GlaHiCo draft ticket — g095 / f1
Suggested labels: ~bug ~"demand analysis" ~"incorrect runtime result"
Suggested milestone: 9.14.2 (bug is in the #26416 fix, which is being
backported for 9.14.2; also present on master)
Related issues: #26416 (the fix this is a hole in), #20819, #25924
Commits: master 870243e4f2a24730539f01ee8e3f3949c42ff312,
ghc-9.14 backport 162955e00d597bbe35c759359792eb73f426c9b8
Reproducer files + run.sh: ./f1/ (verified crashing with
~/ghc/work/_build/stage1/bin/ghc, master d415f38a75, 10.1.20260710)
-->
## Summary
Commit 870243e4f2 ("DmdAnal: Take stable unfoldings into account when
determining argument demands", the fix for #26416, backported to `ghc-9.14`
as 162955e00d) introduces `maxDmdType` to combine the demand type of the
optimised RHS with that of the stable unfolding, so that an argument still
used by the unfolding cannot be marked absent.
`maxDmdType` mishandles argument positions beyond the unfolding's
`dmdTypeDepth`:
```haskell
-- GHC.Types.Demand, current master
maxDmdType (DmdType fv1 ds1) (DmdType fv2 ds2)
= DmdType combined_fv combined_ds
where
combined_fv = maxDmdEnv fv1 fv2
combined_ds = go ds1 ds2
-- If lists have different lengths, keep remaining ds1 (from RHS)
go rhs [] = rhs
go [] _ = []
go (r:rhs) (u:unfs) = maxDmd r u : go rhs unfs
```
By GHC's own convention, a `DmdType` with `n` argument demands implicitly
demands `defaultArgDmd (de_div env)` — `topDmd` ("used") in the common
`topDiv` case — on every further argument (see `splitDmdTy` and
`etaExpandDmdType`). `lubDmdType` respects this by eta-expanding both types
to a common depth before zipping. `maxDmdType`'s `go rhs [] = rhs` clause
instead keeps the RHS's demand — possibly Absent — where the sound result is
`maxDmd r topDmd` = used. So whenever the unfolding's `DmdType` is shallower
than the RHS's, the fix silently contributes nothing for the trailing
arguments, and the exact #26416 crash class survives.
## Why the unfolding's `DmdType` can be shallower than the RHS's
The RHS is eta-expanded to `idArity` by the simplifier. Stable unfolding
templates are usually eta-expanded to `idArity` too
(`Note [Eta-expand stable unfoldings]` in GHC.Core.Opt.Simplify.Iteration),
**but deliberately not when the template is trivial or a PAP**
(`wantEtaExpansion` in GHC.Core.Opt.Simplify.Utils returns `False` for
application chains headed by a variable). For a PAP template `mk g`,
`dmdAnal` produces a `DmdType` whose depth is the leftover of the head's
signature depth after peeling the applications — e.g. depth 0 when `mk` has
arity 1 — regardless of how many arguments the function actually takes.
## Reproducer
Verified with GHC 10.1.20260710 (master, commit d415f38a75, which contains
870243e4f2):
<details><summary>M2.hs</summary>
```haskell
module M2 where
-- f's stable unfolding template is the PAP `mk g`, which
-- Note [Eta-expand stable unfoldings] deliberately does not eta-expand
-- (wantEtaExpansion is False for PAPs). Its DmdType has depth 0, so
-- maxDmdType's `go rhs [] = rhs` keeps the RHS's Absent demand on x.
-- The big lambda keeps the template large enough that f does not
-- certainlyWillInline, so it is worker/wrappered.
{-# INLINABLE [1] f #-}
f :: Int -> Int -> Float
f = mk (\y -> succ . succ . succ . succ . succ . succ . succ . succ . succ $ y)
-- Arity 1: the `let` does real work, so mk is not eta-expanded past it.
-- The returned function really forces x.
{-# NOINLINE mk #-}
mk :: (Int -> Int) -> Int -> Int -> Float
mk t = let s = t 1 in \dummy x -> x `seq` fromIntegral (t (dummy + s))
-- "Optimised" variant: drops the seq on x (RULES may reduce strictness).
{-# INLINE mkFast #-}
mkFast :: (Int -> Int) -> Int -> Int -> Float
mkFast t = \dummy x -> fromIntegral (t dummy)
-- Active only *before* phase 1: rewrites f's RHS early, but is dead by
-- the time f's stable unfolding (activation [1]) is inlined at call sites.
{-# RULES "mk" [~1] forall t. mk t = mkFast t #-}
{-# NOINLINE g #-}
g :: Int -> Int
g x = x + 1
```
</details>
<details><summary>T26416b.hs</summary>
```haskell
module Main where
import M2 ( f )
main = print (f 19 12)
```
</details>
<details><summary>run.sh</summary>
```bash
#!/usr/bin/env bash
# Reproducer for the maxDmdType depth hole (same crash class as #26416).
# Needs a GHC containing commit 870243e4f2 ("DmdAnal: Take stable
# unfoldings into account when determining argument demands"),
# e.g. current master or ghc-9.14 (9.14.2-rc1).
set -euo pipefail
GHC=${GHC:-ghc}
$GHC -O -fforce-recomp -c M2.hs
# The raised threshold only persuades GHC to inline the worker's stable
# unfolding in this small demo; a larger program does that naturally.
$GHC -O -fforce-recomp -funfolding-use-threshold=400 -c T26416b.hs
$GHC T26416b.o M2.o -o t26416b
# Expected: prints 38.0. Actual: "internal error: Oops! Entered absent arg"
./t26416b
```
</details>
Output:
```
t26416b: internal error: Oops! Entered absent arg Arg: x
Type: Int
In module `M2'
(GHC version 10.1.20260710 for x86_64_unknown_linux)
Please report this as a GHC bug: https://www.haskell.org/ghc/reportabug
Aborted (core dumped)
```
Expected output: `38.0` (the value the un-rewritten `mk` computes).
What happens, step by step:
1. The RULE (active before phase 1 only) rewrites `f`'s RHS to
`mkFast (\y -> ...)`, which inlines to `\dummy x -> ...` with `x` unused.
Demand analysis gives `f` the signature `<1!P(SL)><A>`.
2. `addUnfoldingDemands` analyses the stable unfolding template `mk (\y ->
...)`. It is a PAP, so it was never eta-expanded; `mk` has arity 1, so the
template's `DmdType` has depth 0 and `maxDmdType` leaves the RHS's `A`
on `x` untouched — even though the template, via NOINLINE `mk`, really
forces `x`.
3. Worker/wrapper drops `x`; the worker's transferred stable unfolding is
`mk (\y -> ...) (I# ww) (absentError "Arg: x ...")`.
4. In `Main`, the wrapper and then the worker's stable unfolding (activation
`[1]`) are inlined at a phase where the RULE is no longer active, so the
call to `mk` remains, and at runtime `mk` forces the absent-arg filler:
"Entered absent arg", SIGABRT. (With an unboxed/unlifted argument this
is the `LitRubbish` segfault of #26416.)
The `-funfolding-use-threshold=400` on `Main` merely persuades the inliner to
inline the worker's stable unfolding in this small demo; it does not change
semantics, and a larger program (or e.g. specialisation) reaches the same
state without it. `M2.hs` is compiled with plain `-O`.
## Suggested fix
Eta-expand both `DmdType`s to their common (max) depth before zipping,
exactly as `lubDmdType` already does — each side is then padded with its own
`defaultArgDmd`:
```haskell
maxDmdType :: DmdType -> DmdType -> DmdType
maxDmdType d1 d2 = DmdType combined_fv combined_ds
where
n = max (dmdTypeDepth d1) (dmdTypeDepth d2)
DmdType fv1 ds1 = etaExpandDmdType n d1
DmdType fv2 ds2 = etaExpandDmdType n d2
combined_fv = maxDmdEnv fv1 fv2
combined_ds = zipWithEqual maxDmd ds1 ds2
```
## Provenance
This defect was found by an LLM-generated review
([GlaHiCo](https://gitlab.haskell.org/sjakobi/GlaHiCo)) of the 9.14 backport
commit 162955e00d (master: 870243e4f2), reviewed by a human before filing.
Published review: https://gitlab.haskell.org/sjakobi/GlaHiCo/-/snippets/6126
issue