rts: Don't madvise if mmap failed

On 32-bit Linux `outofmem` did not fail with the expected out-of-memory
error message, instead failing with,

    outofmem: internal error: getMBlock: mmap: Invalid argument

This happened because, `my_mmap` would attempt to `madvise` even if the
`mmap` call failed. So while `mmap` returns `ENOMEM` we nevertheless try
to `madvise`, which clobbers `errno`, giving us the unexpected `EINVAL`
error. Consequently we don't detect this to be an out-of-memory error.

This should fix #15060.

Test Plan: `make test TEST=outofmem` on i386

Reviewers: simonmar, erikd

Reviewed By: simonmar

Subscribers: rwbarton, thomie, carter

GHC Trac Issues: #15060

Differential Revision: https://phabricator.haskell.org/D4704

rts/posix/OSMem.c

@@ -192,16 +192,18 @@ my_mmap (void *addr, W_ size, int operation)
         }
     }
 
-    if (operation & MEM_COMMIT) {
-        madvise(ret, size, MADV_WILLNEED);
+    if (ret != (void *)-1) {
+        if (operation & MEM_COMMIT) {
+            madvise(ret, size, MADV_WILLNEED);
 #if defined(MADV_DODUMP)
-        madvise(ret, size, MADV_DODUMP);
+            madvise(ret, size, MADV_DODUMP);
 #endif
-    } else {
-        madvise(ret, size, MADV_DONTNEED);
+        } else {
+            madvise(ret, size, MADV_DONTNEED);
 #if defined(MADV_DONTDUMP)
-        madvise(ret, size, MADV_DONTDUMP);
+            madvise(ret, size, MADV_DONTDUMP);
 #endif
+        }
     }
 
 #else