Crash in long-running web process: Itimer: read(timerfd) failed

Summary

When running a Haskell process in a set of Kubernetes-style provisioned VMs, we've encountered VM instances that encounter this error:

semanticd: internal error: Itimer: read(timerfd) failed
    (GHC version 8.8.1 for x86_64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug

As per the above, I'm filing this (though it doesn't happen reliably).

Steps to reproduce

I apologize, but we haven't determined circumstances that reproduce this crash. I will update this report if this changes.

Expected behavior

Not crash.

Environment

• GHC version used: 8.8.1

Optional:

• Operating System: virtualized Debian stable
• System Architecture: x86

added needs triage label

mentioned in merge request !3056 (closed)

I've opened !3056 (closed) to make it easier to debug next time.

Thanks @hsyl20! It will be interesting to see what this turns up. I don't recall every seeing this particular fail mode in the past and it seems like we handle the obvious EINTR case so I'm at a bit of a loss as to what could be at fault.

I've found:

• https://github.com/Azure/sonic-swss-common/pull/302
• https://github.com/Azure/sonic-swss-common/pull/291
• https://github.com/Azure/sonic-swss-common/pull/295
• https://lkml.org/lkml/2019/7/31/442
• https://lkml.org/lkml/2019/8/16/335

It's not for the same arch but we may want to support read returning 0 with errno == 0.

@patrickt Which Linux version are you using?

We're using Debian stable-slim.

The bug linked above seems to be architecture specific. Are you using DELL hardware?

These Linux instances are virtualized, running under Kubernetes and Docker. I’m trying to find out more about the hardware they’re running on.

@hsyl20 I’ve confirmed that the servers are 14th/15th generation Dell servers. Nice call.

Great! Thanks for the confirmation. !3137 (closed) should fix the bug then.

@hsyl20 You’re awesome. Thanks on behalf of the GitHub Semantic team.

mentioned in commit e8e100ea

mentioned in commit 55edd915

mentioned in commit dce50062

added Phigh RTS Tbug labels and removed needs triage label

It sounds like @hsyl20 may have found a likely culprit. @hsyl20, perhaps you would like to open an MR accepting errno=0 (with a comment pointing to the LKML thread)?

Done: !3137 (closed)

mentioned in merge request !3137 (closed)

mentioned in commit ce47e6b4

closed via commit 8ea37b01

mentioned in commit 8ea37b01

mentioned in commit 3d9267f2

mentioned in commit f3f98a26

mentioned in commit 6bc44deb

mentioned in commit e3fcf26d

mentioned in merge request !3373 (merged)

mentioned in commit 66c1cf35

mentioned in commit df9671ef

mentioned in commit adb716fd

mentioned in commit 39af414c

mentioned in commit d6bc1fef

mentioned in commit 95e92052

mentioned in commit 6507c2bf

mentioned in commit 9037a2b6

Backported to %8.8.4 in !3373 (merged).

changed milestone to %8.8.4

mentioned in commit 285f92bb

mentioned in merge request !3392 (merged)

Thanks for the fix, but by now it's been found that it's not a kernel bug, so the added comment is wrong and the man-page provides the real reason, see my detail reply in:

!3137 (diffs, comment 292898)

It would be great if somebody could fix the comment!

I've just encountered this on github CI:

reopened

marked this issue as related to #20132

mentioned in merge request !6201 (closed)

mentioned in commit 60d6bbb8

mentioned in commit 3a815452

mentioned in commit 90d0e438

mentioned in commit 5f3991c7

mentioned in commit 7471a6c0

mentioned in commit 37ad8856

mentioned in commit d99e76ad

mentioned in commit 05396123

mentioned in commit 48cdaaef

marked #17568 (closed) as a duplicate of this issue

marked this issue as related to #17568 (closed)

mentioned in issue #17568 (closed)