ext-interp tests fail on AArch64

added ARM RTS linking runtime crash labels and removed needs triage label

changed milestone to %8.10.1

assigned to @bgamari

changed the description

I managed to get a backtrace by inserting a threadDelay into iserv's main and attaching a debugger after starting GHC:

$ inplace/bin/ghc-stage2 -c -XTemplateHaskell -package template-haskell -fexternal-interpreter testsuite/tests/th/T10603.hs

I found this:

Thread 1 "ghc-iserv.bin" received signal SIGSEGV, Segmentation fault.
fillGot (oc=oc@entry=0x30aea360) at rts/linker/elf_got.c:106
106	                *(void**)symbol->got_addr = symbol->addr;
(gdb) bt
#0  fillGot (oc=oc@entry=0x30aea360) at rts/linker/elf_got.c:106
#1  0x0000000001cf8b70 in ocResolve_ELF (oc=oc@entry=0x30aea360) at rts/linker/Elf.c:1792
#2  0x0000000001cdd110 in ocTryLoad (oc=0x30aea360) at rts/Linker.c:1608
#3  0x0000000001cdd3d4 in resolveObjs_ () at rts/Linker.c:1654
#4  resolveObjs () at rts/Linker.c:1673
#5  0x0000000000f5da28 in ghcizm8zi9zi0zi20190604_GHCiziObjLink_resolveObjs1_info$def ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) list
101	                }
102	                if(0x0 == symbol->got_addr) {
103	                    errorBelch("Not good either!");
104	                    return EXIT_FAILURE;
105	                }
106	                *(void**)symbol->got_addr = symbol->addr;
107	            }
108	        }
109	    }
110	    return EXIT_SUCCESS;
(gdb) print symbol[0]
$3 = {
  name = 0xffff7ebf5905 "ghczmprim_GHCziClasses_zdp11ZLzvz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUz2cUzvZR_info", addr = 0xffff8415f8a8, got_addr = 0xffff84054000, 
  elf_sym = 0xffff7eb3e610}
(gdb) print symbol[0].elf_sym[0]
$4 = {st_name = 460813, st_info = 17 '\021', st_other = 0 '\000', st_shndx = 1, st_value = 112808, st_size = 84}

Looking at st_info it seems that this is an STB_GLOBAL STT_OBJECT symbol.

Strangely it seems I can read from symbol->got_addr:

(gdb) x/8 symbol->got_addr 
0xffff84054000:	0x0	0x0
0xffff84054010:	0x0	0x0
0xffff84054020:	0x0	0x0
0xffff84054030:	0x0	0x0

The disassembly suggests that we are precisely where the backtrace says we are:

Disassembly

Dump of assembler code for function fillGot:
   0x0000000001d125e8 <+0>:	stp	x19, x20, [sp, #-48]!
   0x0000000001d125ec <+4>:	stp	x21, x22, [sp, #16]
   0x0000000001d125f0 <+8>:	str	x30, [sp, #32]
   0x0000000001d125f4 <+12>:	ldr	x0, [x0, #64]
   0x0000000001d125f8 <+16>:	ldr	x21, [x0, #32]
   0x0000000001d125fc <+20>:	cbz	x21, 0x1d12680 
   0x0000000001d12600 <+24>:	ldr	x5, [x21, #16]
   0x0000000001d12604 <+28>:	mov	x19, #0x0                   	// #0
   0x0000000001d12608 <+32>:	cbnz	x5, 0x1d12618 
   0x0000000001d1260c <+36>:	b	0x1d12678 
   0x0000000001d12610 <+40>:	cmp	x5, x19
   0x0000000001d12614 <+44>:	b.ls	0x1d12678   // b.plast
   0x0000000001d12618 <+48>:	ldr	x4, [x21, #8]
   0x0000000001d1261c <+52>:	lsl	x3, x19, #5
   0x0000000001d12620 <+56>:	add	x19, x19, #0x1
   0x0000000001d12624 <+60>:	add	x20, x4, x3
   0x0000000001d12628 <+64>:	ldr	x0, [x20, #24]
   0x0000000001d1262c <+68>:	ldrb	w2, [x0, #4]
   0x0000000001d12630 <+72>:	lsr	w1, w2, #4
   0x0000000001d12634 <+76>:	sub	w0, w1, #0x1
   0x0000000001d12638 <+80>:	and	w0, w0, #0xff
   0x0000000001d1263c <+84>:	cmp	w0, #0x1
   0x0000000001d12640 <+88>:	b.hi	0x1d12610   // b.pmore
   0x0000000001d12644 <+92>:	tst	x2, #0xf
   0x0000000001d12648 <+96>:	ldr	x0, [x20, #8]
   0x0000000001d1264c <+100>:	cset	w22, eq  // eq = none
   0x0000000001d12650 <+104>:	cmp	w1, #0x2
   0x0000000001d12654 <+108>:	csinc	w22, w22, wzr, ne  // ne = any
   0x0000000001d12658 <+112>:	cbz	w22, 0x1d12688 
   0x0000000001d1265c <+116>:	cbz	x0, 0x1d126b4 
   0x0000000001d12660 <+120>:	ldr	x1, [x20, #16]
   0x0000000001d12664 <+124>:	cbz	x1, 0x1d126d8 
=> 0x0000000001d12668 <+128>:	str	x0, [x1]
   0x0000000001d1266c <+132>:	ldr	x5, [x21, #16]
   0x0000000001d12670 <+136>:	cmp	x5, x19
   0x0000000001d12674 <+140>:	b.hi	0x1d12618   // b.pmore
   0x0000000001d12678 <+144>:	ldr	x21, [x21, #32]
   0x0000000001d1267c <+148>:	cbnz	x21, 0x1d12600 
   0x0000000001d12680 <+152>:	mov	w22, #0x0                   	// #0
   0x0000000001d12684 <+156>:	b	0x1d126a0 
   0x0000000001d12688 <+160>:	cbnz	x0, 0x1d12660 
   0x0000000001d1268c <+164>:	ldr	x1, [x20]
   0x0000000001d12690 <+168>:	adrp	x0, 0x1d5d000
   0x0000000001d12694 <+172>:	mov	w22, #0x1                   	// #1
   0x0000000001d12698 <+176>:	add	x0, x0, #0x3d8
   0x0000000001d1269c <+180>:	bl	0x1cea830 
   0x0000000001d126a0 <+184>:	mov	w0, w22
   0x0000000001d126a4 <+188>:	ldr	x30, [sp, #32]
   0x0000000001d126a8 <+192>:	ldp	x21, x22, [sp, #16]
   0x0000000001d126ac <+196>:	ldp	x19, x20, [sp], #48
   0x0000000001d126b0 <+200>:	ret
   0x0000000001d126b4 <+204>:	ldr	x0, [x4, x3]
   0x0000000001d126b8 <+208>:	bl	0x1cdd1b8 
   0x0000000001d126bc <+212>:	str	x0, [x20, #8]
   0x0000000001d126c0 <+216>:	cbnz	x0, 0x1d12660 
   0x0000000001d126c4 <+220>:	ldr	x1, [x20]
   0x0000000001d126c8 <+224>:	adrp	x0, 0x1d5d000
   0x0000000001d126cc <+228>:	add	x0, x0, #0x3b8
   0x0000000001d126d0 <+232>:	bl	0x1cea830 
   0x0000000001d126d4 <+236>:	b	0x1d126a0 
   0x0000000001d126d8 <+240>:	mov	w22, #0x1                   	// #1
   0x0000000001d126dc <+244>:	adrp	x0, 0x1d5d000
   0x0000000001d126e0 <+248>:	add	x0, x0, #0x410
   0x0000000001d126e4 <+252>:	bl	0x1cea830 
   0x0000000001d126e8 <+256>:	mov	w0, w22
   0x0000000001d126ec <+260>:	ldr	x30, [sp, #32]
   0x0000000001d126f0 <+264>:	ldp	x21, x22, [sp, #16]
   0x0000000001d126f4 <+268>:	ldp	x19, x20, [sp], #48
   0x0000000001d126f8 <+272>:	ret
End of assembler dump.
(gdb)

Moreover, looking at $x1 again confirms that we are where we think we are:

(gdb) print/a $x1
$9 = 0xffff84054000

I'm rather puzzled; it seems I can both read and write to symbol->got_addr. Why is it crashing?

So I'm still puzzled by why I can access the memory in gdb but I think I found the culprit: elf_got.c:makeGot calls mprotect to remap the GOT region as PROT_READ after initializing the GOT. Commenting this out allows the program to get a bit further.

Now the program is instead failing with this assertion failure:

Thread 1 "ghc-iserv.bin" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x0000ffff94ffa8b4 in __GI_abort () at abort.c:79
#2  0x0000ffff94ff2b44 in __assert_fail_base (fmt=0xffff950ee0c0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x1d9c538 "isInt64(32, addend)", 
    file=file@entry=0x1d9c518 "rts/linker/elf_reloc_aarch64.c", line=line@entry=93, function=function@entry=0x1d9c810 <__PRETTY_FUNCTION__.9614> "encodeAddendAarch64") at assert.c:92
#3  0x0000ffff94ff2bc4 in __GI___assert_fail (assertion=0x1d9c538 "isInt64(32, addend)", file=0x1d9c518 "rts/linker/elf_reloc_aarch64.c", line=93, 
    function=0x1d9c810 <__PRETTY_FUNCTION__.9614> "encodeAddendAarch64") at assert.c:101
#4  0x0000000001d3dc88 in encodeAddendAarch64 (section=0x2bba4608, rel=0xffff92bc6be8, addend=-281473104080896) at rts/linker/elf_reloc_aarch64.c:93
#5  0x0000000001d3e728 in relocateObjectCodeAarch64 (oc=0x2bba4410) at rts/linker/elf_reloc_aarch64.c:328
#6  0x0000000001d1bf18 in relocateObjectCode (oc=0x2bba4410) at rts/linker/elf_reloc.c:9
#7  0x0000000001d1a9c0 in ocResolve_ELF (oc=0x2bba4410) at rts/linker/Elf.c:1801
#8  0x0000000001cd8908 in ocTryLoad (oc=0x2bba4410) at rts/Linker.c:1608
#9  0x0000000001cd89e0 in resolveObjs_ () at rts/Linker.c:1654
#10 0x0000000001cd8a74 in resolveObjs () at rts/Linker.c:1673
#11 0x0000000000f57fd8 in ghcizm8zi9zi0zi20190604_GHCiziObjLink_resolveObjs1_info$def ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) up 4
(gdb) print/x addend
$2 = 0xffff00006f9e1000

This is in the COMPAT_R_AARCH64_ADR_PREL_PG_HI21 case of elf_reloc_aarch64.c:encodeAddendAarch64.

I have opened #16779 (closed) to track this.

I suspect what is happening here is that some bit of code (the RTS?) is getting mapped in low memory whereas the RTS is loading static libraries in high memory (e.g. 0x0000ffff00000000). Consequently we can't fit the final relocation result in 32-bits.

On x86-64 mmap accepts a MAP_32BIT flag which hints that the mapping should be placed in the lower 2GB; it appears that the motivation for this flag is to solve issues very similar to what I describe above. Unfortunately, this is apparently not provided on AArch64.

Indeed symbol in relocateObjectCodeAarch64 is

(gdb) print symbol[0]
$3 = {name = 0xffff98a8a46c "stg_upd_frame_info", addr = 0x1d24a60 <stg_upd_frame_info$def>, got_addr = 0xffff9801da88, elf_sym = 0xffff9889d5a8}

For what it's worth adding -dynamic to the GHC command-line (causing GHC to use the dynamically-linked iserv interpreter) runs successfully. It seems that DYNAMIC_BY_DEFAULT is currently forced off in config.mk.in for reasons that aren't at all clear and probably no longer relevant. We should fix this.

CCing @angerman.

mentioned in issue #16779 (closed)

mentioned in issue #16784 (closed)

The second issue was that the PLT was mapped too far from the call-sites due to a naive use of mmap instead of mmapForLinker. Tracked as #16784 (closed).

I still suspect that the GOT should also be using mmapForLinker, but fixing the PLT seems to have fixed most of the issues.

mentioned in merge request !1139 (closed)

mentioned in issue #16796 (closed)

So essentially this just means our linker is not very good. Maybe it's really time to turn that thing into a library.

My thought was to simply avoid the problem entirely and use DYNAMIC_BY_DEFAULT, at least until our linker is better. I have tested and results are much better with DYNAMIC_BY_DEFAULT.

removed milestone

marked this issue as related to #16087

mentioned in merge request !5786 (closed)

ext-interp tests fail on AArch64

Child items ...

Activity

ext-interp tests fail on AArch64

Relates to

Activity