## Summary On 8.10.7 and on 9.2.4 (but only with optimizations on afaict) I get segfaults and an occasional internal error running our app (https://github.com/hasura/graphql-engine/commits/master) with `--nonmoving-gc`. I've only started debugging with 9.2.4. There I see: sometimes the server throws an error on startup, sometimes only after sending a query, and sometimes it seems to function normally. A couple gdb sessions are attached; I don't really know how to get useful information from it. I’ve built with `-g2 ` and I'm using the dwarf bindist. [gdb2](/uploads/c7c602ecad8653806a6522be4c628ee2/gdb2) [gdb1](/uploads/d763fff5db5e2876b98e3a9a1e429900/gdb1) The one "internal error" I managed to get was: ``` graphql-engine: internal error: PAP object (0x4205af6858) entered! Stack trace: 0x7fffdffa048f set_initial_registers (rts/Libdw.c:294.5) 0x7fffde9435a8 dwfl_thread_getframes (/usr/lib/x86_64-linux-gnu/libdw-0.187.so) 0x7fffde94395c dwfl_getthread_frames (/usr/lib/x86_64-linux-gnu/libdw-0.187.so) 0x7fffdffa0ab7 libdwGetBacktrace (rts/Libdw.c:265.8) 0x7fffdff9610d rtsFatalInternalErrorFn (rts/RtsMessages.c:176.6) 0x7fffdff962d0 barf (rts/RtsMessages.c:49.3) 0x7fffdffc9093 stg_PAP_info (rts/Apply.cmm:215.1) 0x7ffff6e730e8 graphqlzmenginezm1zi0zi0zminplace_HasuraziRQLziDDLziSchemaziCache_buildRebuildableSchemaCacheWithReason1847_info (src-lib/Hasura/RQL/DDL/Schema/Cache.hs:337.23) 0x7fffdffc7230 stg_upd_frame_info (rts/Updates.cmm:31.1) 0x7ffff6df5ed0 (null) (libraries/base/GHC/Base.hs:1491.22) 0x7ffff6ea1b88 (null) (src-lib/Hasura/RQL/DDL/Schema/Cache.hs:156.10) 0x886ca8 (null) (src-lib/Hasura/App.hs:409.21) 0x7fffdffc75f8 stg_maskAsyncExceptionszh_ret_info (rts/Exception.cmm:115.1) 0x7fffdffc7b08 stg_catch_frame_info (rts/Exception.cmm:376.1) 0x87e2f0 (null) (libraries/base/GHC/Base.hs:1514.12) 0x7fffdffc74b0 stg_unmaskAsyncExceptionszh_ret_info (rts/Exception.cmm:63.1) 0x87e750 Main_zdsforkManagedT_info (libraries/base/GHC/IO.hs:290.36) 0x7fffdffc75f8 stg_maskAsyncExceptionszh_ret_info (rts/Exception.cmm:115.1) 0x7fffdffc7b08 stg_catch_frame_info (rts/Exception.cmm:376.1) 0x3d9ab0 (null) (libraries/base/Control/Exception/Base.hs:239.10) 0x7fffdffc74b0 stg_unmaskAsyncExceptionszh_ret_info (rts/Exception.cmm:63.1) 0x3d9f60 Main_zdszdwmkLoggerCtx_info (libraries/base/GHC/IO.hs:290.36) 0x7fffdffc7b08 stg_catch_frame_info (rts/Exception.cmm:376.1) 0x7fffdffc7b08 stg_catch_frame_info (rts/Exception.cmm:376.1) 0x7fffdffc6fc8 stg_stop_thread_info (rts/StgStartup.cmm:42.1) 0x7fffdffa0c5b StgRunIsImplementedInAssembler (rts/StgCRun.c:379.5) 0x7fffdffa4bc3 schedule (rts/Capability.h:226.12) 0x7fffdffa64c1 scheduleWaitThread (rts/Schedule.c:2634.11) 0x7fffdffa952c hs_main (rts/RtsMain.c:73.18) 0x12d7d70 _init (/home/me/Work/hasura/graphql-engine-mono/dist-newstyle/build/x86_64-linux/ghc-9.2.4.20220919/graphql-engine-1.0.0/x/graphql-engine/opt/build/graphql-engine/graphql-engine) 0x7fffde02920a __libc_start_call_main (../sysdeps/nptl/libc_start_call_main.h:58.16) 0x7fffde0292bc __libc_start_main@@GLIBC_2.34 (../csu/libc-start.c:128.20) 0x36f561 _start (/home/me/Work/hasura/graphql-engine-mono/dist-newstyle/build/x86_64-linux/ghc-9.2.4.20220919/graphql-engine-1.0.0/x/graphql-engine/opt/build/graphql-engine/graphql-engine) (GHC version 9.2.4.20220919 for x86_64_unknown_linux) Please report this as a GHC bug: https://www.haskell.org/ghc/reportabug ``` ## Steps to reproduce I will have to circle back with a small reproducer if I can find one, but you can do: - edit `scripts/dev.sh` and add `--nonmoving-gc` to `RUN_INVOCATION` - change line 28 of `cabal/dev-sh.project.local` to `flags: +optimize-hasura`; `ln -s cabal/dev-sh.project.local cabal.project.local` - `$ scripts/dev.sh graphql-engine` in one terminal - `$ scripts/dev.sh postgres` in another - if you haven’t segfaulted by this point: `cd server/benchmarks/benchmark_sets/chinook && curl -X POST -H 'Content-Type: application/json' -d @replace_metadata.json http://127.0.0.1:8181/v1/query` ## Expected behavior normal functioning ## Environment * GHC version used: 9.2.4 (fork with backported fix: https://gitlab.haskell.org/ghc/ghc/-/pipelines/56889 ; I'll need to circle back about whether I see this on stock 9.2.4 as well) Optional: * Operating System: linux * System Architecture: debian bullseye