GitLab

This reverts commit bf10456e.

SMALL_MUT_ARR_PTRS_FROZEN0 -> SMALL_MUT_ARR_PTRS_FROZEN_DIRTY
SMALL_MUT_ARR_PTRS_FROZEN  -> SMALL_MUT_ARR_PTRS_FROZEN_CLEAN
MUT_ARR_PTRS_FROZEN0       -> MUT_ARR_PTRS_FROZEN_DIRTY
MUT_ARR_PTRS_FROZEN        -> MUT_ARR_PTRS_FROZEN_CLEAN

Naming is now consistent with other CLEAR/DIRTY objects (MVAR, MUT_VAR,
MUT_ARR_PTRS).

(alternatively we could rename MVAR_DIRTY/MVAR_CLEAN etc. to MVAR0/MVAR)

Removed a few comments in Scav.c about FROZEN0 being on the mut_list
because it's now clear from the closure type.

Reviewers: bgamari, simonmar, erikd

Reviewed By: simonmar

Subscribers: rwbarton, thomie, carter

Differential Revision: https://phabricator.haskell.org/D4784

This feature has some very serious correctness issues (#14310),
introduces a great deal of complexity, and hasn't seen wide usage.
Consequently we are removing it, as proposed in Proposal #77 [1]. This
is heavily based on a patch from fryguybob.

Updates stm submodule.

[1] https://github.com/ghc-proposals/ghc-proposals/pull/77

Test Plan: Validate

Reviewers: erikd, simonmar, hvr

Reviewed By: simonmar

Subscribers: rwbarton, thomie, carter

GHC Trac Issues: #14310

Differential Revision: https://phabricator.haskell.org/D4760

Unfortunately, this optimisation is infeasible on MachO platforms (e.g.
Darwin) due to an object format limitation. Specifically, linking fails
with errors of the form:

     error: unsupported relocation with subtraction expression, symbol
     '_integerzmgmp_GHCziIntegerziType_quotInteger_closure' can not be
     undefined in a subtraction expression

Apparently MachO does not permit relocations' subtraction expressions to
refer to undefined symbols. As far as I can tell this means that it is
essentially impossible to express an offset between symbols living in
different compilation units. This means that we lively can't use this
optimisation on MachO platforms.

Test Plan: Validate on Darwin

Reviewers: simonmar, erikd

Subscribers: rwbarton, thomie, carter, angerman

GHC Trac Issues: #15169

Differential Revision: https://phabricator.haskell.org/D4715

This pulls parts of Joachim Breitner's ghc-heap-view library inside GHC.
The bits added are the C hooks into the RTS and a basic Haskell wrapper
to these C hooks. The main reason for these to be added to GHC proper
is that the code needs to be kept in sync with the closure types
defined by the RTS. It is expected that the version of HeapView shipped
with GHC will always work with that version of GHC and that extra
functionality can be layered on top with a library like ghc-heap-view
distributed via Hackage.

Test Plan: validate

Reviewers: simonmar, hvr, nomeata, austin, Phyx, bgamari, erikd

Reviewed By: bgamari

Subscribers: carter, patrickdoc, tmcgilchrist, rwbarton, thomie

Differential Revision: https://phabricator.haskell.org/D3055

Summary:
See the new note.
This should fix cb5c2fe8 enough
to unbreak Windows and OS X builds.

Test Plan: manual testing with patched gdb

Reviewers: bgamari, simonmar, erikd

Reviewed By: bgamari

Subscribers: rwbarton, thomie, carter

Differential Revision: https://phabricator.haskell.org/D4694

Summary:
The idea here is to save a little code size and some work in the GC,
by collapsing FUN_STATIC closures and their SRTs.

This is (4) in a series; see D4632 for more details.

There's a tradeoff here: more complexity in the compiler in exchange
for a modest code size reduction (probably around 0.5%).

Results:
* GHC binary itself (statically linked) is 1% smaller
* -0.2% binary sizes in nofib (-0.5% module sizes)

Full nofib results comparing D4634 with this: P177 (ignore runtimes,
these aren't stable on my laptop)

Test Plan: validate, nofib

Reviewers: bgamari, niteria, simonpj, erikd

Subscribers: thomie, carter

Differential Revision: https://phabricator.haskell.org/D4637

Summary:
An info table with an SRT normally looks like this:

    StgWord64 srt_offset
    StgClosureInfo layout
    StgWord32 layout
    StgWord32 has_srt

But we only need 32 bits for srt_offset on x86_64, because the small
memory model requires that code segments are at most 2GB. So we can
optimise this to

    StgClosureInfo layout
    StgWord32 layout
    StgWord32 srt_offset

saving a word.  We can tell whether the info table has an SRT or not,
because zero is not a valid srt_offset, so zero still indicates that
there's no SRT.

Test Plan:
* validate
* For results, see D4632.

Reviewers: bgamari, niteria, osa1, erikd

Subscribers: thomie, carter

Differential Revision: https://phabricator.haskell.org/D4634

Summary:
- Previously we would hvae a single big table of pointers per module,
  with a set of bitmaps to reference entries within it. The new
  representation is identical to a static constructor, which is much
  simpler for the GC to traverse, and we get to remove the complicated
  bitmap-traversal code from the GC.

- Rewrite all the code to generate SRTs in CmmBuildInfoTables, and
  document it much better (see Note [SRTs]). This has been something
  I've wanted to do since we moved to the new code generator, I
  finally had the opportunity to finish it while on a transatlantic
  flight recently :)

There are a series of 4 diffs:

1. D4632 (this one), which does the bulk of the changes

2. D4633 which adds support for smaller `CmmLabelDiffOff` constants

3. D4634 which takes advantage of D4632 and D4633 to save a word in
   info tables that have an SRT on x86_64. This is where most of the
   binary size improvement comes from.

4. D4637 which makes a further optimisation to merge some SRTs with
   static FUN closures.  This adds some complexity and the benefits
   are fairly modest, so it's not clear yet whether we should do this.

Results (after (3), on x86_64)

- GHC itself (staticaly linked) is 5.2% smaller

- -1.7% binary sizes in nofib, -2.9% module sizes. Full nofib results: P176

- I measured the overhead of traversing all the static objects in a
  major GC in GHC itself by doing `replicateM_ 1000 performGC` as the
  first thing in `Main.main`.  The new version was 5-10% faster, but
  the results did vary quite a bit.

- I'm not sure if there's a compile-time difference, the results are
  too unreliable.

Test Plan: validate

Reviewers: bgamari, michalt, niteria, simonpj, erikd, osa1

Subscribers: thomie, carter

Differential Revision: https://phabricator.haskell.org/D4632

GCC 8 now generates warnings for incompatible function pointer casts
[-Werror=cast-function-type]. Apparently there are a few of those in rts
code, which makes `./validate` unhappy (since we compile with `-Werror`)

This commit tries to fix these issues by changing the functions to have
the correct type (and, if necessary, moving the casts into those
functions).

For instance, hash/comparison function are declared (`Hash.h`) to take
`StgWord` but we want to use `StgWord64[2]` in `StaticPtrTable.c`.
Instead of casting the function pointers, we can cast the `StgWord`
parameter to `StgWord*`. I think this should be ok since `StgWord`
should be the same size as a pointer.
Signed-off-by: Michal Terepeta <michal.terepeta@gmail.com>

Test Plan: ./validate

Reviewers: bgamari, erikd, simonmar

Reviewed By: bgamari

Subscribers: rwbarton, thomie, carter

Differential Revision: https://phabricator.haskell.org/D4673

This reverts commit cb5c2fe8.

It appears to have broken OSX and Windows builds.

See the new note.

Test Plan:
manual testing with patched gdb

Reviewers: bgamari, simonmar, erikd

Subscribers: rwbarton, thomie, carter

Differential Revision: https://phabricator.haskell.org/D4666

Test Plan: Passes validate

Reviewers: simonmar, bgamari, erikd

Subscribers: thomie, carter

GHC Trac Issues: #10296

Differential Revision: https://phabricator.haskell.org/D4627

Reviewers: bgamari, simonmar, erikd

Reviewed By: bgamari, simonmar

Subscribers: thomie, carter

Differential Revision: https://phabricator.haskell.org/D4539

[skip ci]

Summary:
get/setAllocationCounter didn't take into account allocations in the
current block. This was known at the time, but it turns out to be
important to have more accuracy when using these in a fine-grained
way.

Test Plan:
New unit test to test incrementally larger allocaitons.  Before I got
results like this:

```
+0
+0
+0
+0
+0
+4096
+0
+0
+0
+0
+0
+4064
+0
+0
+4088
+4056
+0
+0
+0
+4088
+4096
+4056
+4096
```

Notice how the results aren't always monotonically increasing.  After
this patch:

```
+344
+416
+488
+560
+632
+704
+776
+848
+920
+992
+1064
+1136
+1208
+1280
+1352
+1424
+1496
+1568
+1640
+1712
+1784
+1856
+1928
+2000
+2072
+2144
```

Reviewers: hvr, erikd, simonmar, jrtc27, trommler

Reviewed By: simonmar

Subscribers: trommler, jrtc27, rwbarton, thomie, carter

Differential Revision: https://phabricator.haskell.org/D4363

The existing internal counters:
* gc_alloc_block_sync
* whitehole_spin
* gen[g].sync
* gen[1].sync

are now not shown in the -s report unless --internal-counters is also passed.

If --internal-counters is passed we now show the counters above, reformatted, as
well as several other counters. In particular, we now count the yieldThread()
calls that SpinLocks do as well as their spins.

The added counters are:
* gc_spin (spin and yield)
* mut_spin (spin and yield)
* whitehole_threadPaused (spin only)
* whitehole_executeMessage (spin only)
* whitehole_lockClosure (spin only)
* waitForGcThreadsd (spin and yield)

As well as the following, which are not SpinLock-like things:
* any_work
* do_work
* scav_find_work

See the Note for descriptions of what these counters are.

We add busy_wait_nops in these loops along with the counter increment where it
was absent.

Old internal counters output:
```
gc_alloc_block_sync: 0
whitehole_gc_spin: 0
gen[0].sync: 0
gen[1].sync: 0
```

New internal counters output:
```
Internal Counters:
                                           Spins        Yields
    gc_alloc_block_sync                      323             0
    gc_spin                              9016713           752
    mut_spin                            57360944         47716
    whitehole_gc                               0           n/a
    whitehole_threadPaused                     0           n/a
    whitehole_executeMessage                   0           n/a
    whitehole_lockClosure                      0             0
    waitForGcThreads                           2           415
    gen[0].sync                                6             0
    gen[1].sync                                1             0

    any_work                                2017
    no_work                                 2014
    scav_find_work                          1004
```

Test Plan:
./validate

Check it builds with #define PROF_SPIN removed from includes/rts/Config.h

Reviewers: bgamari, erikd, simonmar, hvr

Reviewed By: simonmar

Subscribers: rwbarton, thomie, carter

GHC Trac Issues: #3553, #9221

Differential Revision: https://phabricator.haskell.org/D4302

Summary: See definition of PRINTF above the change

Reviewers: bgamari, erikd, simonmar, Phyx

Reviewed By: Phyx

Subscribers: Phyx, rwbarton, thomie, carter

Differential Revision: https://phabricator.haskell.org/D4392

Test Plan: Validate

Reviewers: erikd, simonmar

Reviewed By: simonmar

Subscribers: rwbarton, thomie, carter

Differential Revision: https://phabricator.haskell.org/D4374

This prevents the register being picked up as a scratch register.
Otherwise the allocator would be free to use it before a call. This
fixes #14619.

Test Plan: ci, repro case on #14619

Reviewers: bgamari, Phyx, erikd, simonmar, RyanGlScott, simonpj

Reviewed By: Phyx, RyanGlScott, simonpj

Subscribers: simonpj, RyanGlScott, Phyx, rwbarton, thomie, carter

GHC Trac Issues: #14619

Differential Revision: https://phabricator.haskell.org/D4348

This reverts commit a1a689dd.

Summary:
get/setAllocationCounter didn't take into account allocations in the
current block. This was known at the time, but it turns out to be
important to have more accuracy when using these in a fine-grained
way.

Test Plan:
New unit test to test incrementally larger allocaitons.  Before I got
results like this:

```
+0
+0
+0
+0
+0
+4096
+0
+0
+0
+0
+0
+4064
+0
+0
+4088
+4056
+0
+0
+0
+4088
+4096
+4056
+4096
```

Notice how the results aren't always monotonically increasing.  After
this patch:

```
+344
+416
+488
+560
+632
+704
+776
+848
+920
+992
+1064
+1136
+1208
+1280
+1352
+1424
+1496
+1568
+1640
+1712
+1784
+1856
+1928
+2000
+2072
+2144
```

Reviewers: niteria, bgamari, hvr, erikd

Subscribers: rwbarton, thomie, carter

Differential Revision: https://phabricator.haskell.org/D4288

Test Plan: ./validate

Reviewers: bgamari, erikd, simonmar

Reviewed By: bgamari

Subscribers: rwbarton, thomie

GHC Trac Issues: #14513

Differential Revision: https://phabricator.haskell.org/D4226

Reviewers: erikd, simonmar

Reviewed By: simonmar

Subscribers: rwbarton, thomie

Differential Revision: https://phabricator.haskell.org/D4194

Summary:
GC sync is the time between a GC being intiated and all the mutator
threads finally stopping so that the GC can start. Problems that cause
the GC sync to be delayed are hard to find and can cause dramatic
slowdowns for heavily parallel programs.

The new flag --long-gc-sync=<time> helps by emitting a warning and
calling a user-overridable hook when the GC sync time exceeds the
specified threshold. A debugger can be used to set a breakpoint when
this happens and inspect the stacks of threads to find the culprit.

Test Plan:
```
$ ./inplace/bin/ghc-stage2 +RTS --long-gc-sync=0.0000001 -S
    Alloc    Copied     Live     GC     GC      TOT      TOT  Page Flts
    bytes     bytes     bytes   user   elap     user     elap
  1135856     51144    153736  0.000  0.000    0.002    0.002    0    0  (Gen:  0)
  1034760     94704    188752  0.000  0.000    0.002    0.002    0    0  (Gen:  0)
  1038888    134832    228888  0.009  0.009    0.011    0.011    0    0  (Gen:  1)
  1025288     90128    235184  0.000  0.000    0.012    0.012    0    0  (Gen:  0)
  1049088    130080    333984  0.000  0.000    0.013    0.013    0    0  (Gen:  0)
Warning: waited 0us for GC sync
  1034424     73360    331976  0.000  0.000    0.013    0.013    0    0  (Gen:  0)
```

Also tested on a real production problem.

Reviewers: niteria, bgamari, erikd

Subscribers: rwbarton, thomie

Differential Revision: https://phabricator.haskell.org/D4193

This is another step for fixing #13825 and is based on D38 by Simon
Marlow.

The change allows storing multiple constructor fields within the same
word. This currently applies only to `Float`s, e.g.,
```
data Foo = Foo {-# UNPACK #-} !Float {-# UNPACK #-} !Float
```
on 64-bit arch, will now store both fields within the same constructor
word. For `WordX/IntX` we'll need to introduce new primop types.

Main changes:

- We now use sizes in bytes when we compute the offsets for
  constructor fields in `StgCmmLayout` and introduce padding if
  necessary (word-sized fields are still word-aligned)

- `ByteCodeGen` had to be updated to correctly construct the data
  types. This required some new bytecode instructions to allow pushing
  things that are not full words onto the stack (and updating
  `Interpreter.c`). Note that we only use the packed stuff when
  constructing data types (i.e., for `PACK`), in all other cases the
  behavior should not change.

- `RtClosureInspect` was changed to handle the new layout when
  extracting subterms. This seems to be used by things like `:print`.
  I've also added a test for this.

- I deviated slightly from Simon's approach and use `PrimRep` instead
  of `ArgRep` for computing the size of fields.  This seemed more
  natural and in the future we'll probably want to introduce new
  primitive types (e.g., `Int8#`) and `PrimRep` seems like a better
  place to do that (where we already have `Int64Rep` for example).
  `ArgRep` on the other hand seems to be more focused on calling
  functions.
Signed-off-by: Michal Terepeta <michal.terepeta@gmail.com>

Test Plan: ./validate

Reviewers: bgamari, simonmar, austin, hvr, goldfire, erikd

Reviewed By: bgamari

Subscribers: maoe, rwbarton, thomie

GHC Trac Issues: #13825

Differential Revision: https://phabricator.haskell.org/D3809

Summary:
This patch adds the ability to generate stack traces on crashes for Windows.
When running in the interpreter this attempts to use symbol information from
the interpreter and information we know about the loaded object files to
resolve addresses to symbols.

When running compiled it doesn't have this information and then defaults
to using symbol information from PDB files. Which for now means only
files compiled with ICC or MSVC will show traces compiled.

But I have a future patch that may address this shortcoming.

Also since I don't know how to walk a pure haskell stack, I can for now
only show the last entry. I'm hoping to figure out how Apply.cmm works to
be able to walk the stalk and give more entries for pure haskell code.

In GHCi

```
$ echo main | inplace/bin/ghc-stage2.exe --interactive ./testsuite/tests/rts/derefnull.hs
GHCi, version 8.3.20170830: http://www.haskell.org/ghc/  :? for help
Ok, 1 module loaded.
Prelude Main>
Access violation in generated code when reading 0x0

 Attempting to reconstruct a stack trace...

   Frame        Code address
 * 0x77cde10    0xc370229 E:\..\base\dist-install\build\HSbase-4.10.0.0.o+0x190031
                 (base_ForeignziStorable_zdfStorableInt4_info+0x3f)
```

and compiled

```
Access violation in generated code when reading 0x0

 Attempting to reconstruct a stack trace...

   Frame        Code address
 * 0xf0dbd0     0x40bb01 E:\..\rts\derefnull.run\derefnull.exe+0xbb01
```

Test Plan: ./validate

Reviewers: austin, hvr, bgamari, erikd, simonmar

Reviewed By: bgamari

Subscribers: rwbarton, thomie

Differential Revision: https://phabricator.haskell.org/D3913

It's often hard to debug things like segfaults on Windows,
mostly because gdb isn't always of use and users don't know
how to effectively use it.

This patch provides a way to create a crash drump by passing

`+RTS --generate-crash-dumps` as an option. If any unhandled
exception is triggered a dump is made that contains enough
information to be able to diagnose things successfully.

Currently the created dumps are a bit big because I include
all registers, code and threads information.

This looks like

```
$ testsuite/tests/rts/derefnull.run/derefnull.exe +RTS
--generate-crash-dumps

Access violation in generated code when reading 0000000000000000
Crash dump created. Dump written to:
        E:\msys64\tmp\ghc-20170901-220250-11216-16628.dmp
```

Test Plan: ./validate

Reviewers: austin, hvr, bgamari, erikd, simonmar

Reviewed By: bgamari, simonmar

Subscribers: rwbarton, thomie

Differential Revision: https://phabricator.haskell.org/D3912

Exception handling on Windows is unfortunately a bit complicated.
But essentially the VEH Handlers we currently have are running too
early.

This was a problem as it ran so early it also swallowed C++ exceptions
and other software exceptions which the system could have very well
recovered from.

So instead we use a sequence of chains to for the exception handlers to
run as late as possible. You really can't get any later than this.

Please read the comment in the patch for more details.

I'm also providing a switch to allow people to turn off the exception
handling entirely. In case it does present a problem with their code.

(Reverted and recommitted to fix authorship information)

Test Plan: ./validate

Reviewers: austin, hvr, bgamari, erikd, simonmar

Reviewed By: bgamari

Subscribers: rwbarton, thomie

GHC Trac Issues: #13911, #12110

Differential Revision: https://phabricator.haskell.org/D3911

Reverting to fix authorship of commit.

This reverts commit 1825cbdb.

Test Plan: Validate, add tests

Reviewers: simonmar, austin, erikd

Reviewed By: simonmar

Subscribers: rwbarton, thomie

Differential Revision: https://phabricator.haskell.org/D4021

Exception handling on Windows is unfortunately a bit complicated.
But essentially the VEH Handlers we currently have are running too
early.

This was a problem as it ran so early it also swallowed C++ exceptions
and other software exceptions which the system could have very well
recovered from.

So instead we use a sequence of chains to for the exception handlers to
run as late as possible. You really can't get any later than this.

Please read the comment in the patch for more details.

I'm also providing a switch to allow people to turn off the exception
handling entirely. In case it does present a problem with their code.

Test Plan: ./validate

Reviewers: austin, hvr, bgamari, erikd, simonmar

Reviewed By: bgamari

Subscribers: rwbarton, thomie

GHC Trac Issues: #13911, #12110

Differential Revision: https://phabricator.haskell.org/D3911

This is done by reusing the existing cross-platform
`getProcessElapsedTime()` function, which already provides nanosecond
monotonic clocks, and fallback for platforms that don't have those.

To do this, `getProcessElapsedTime()` had to be moved from a private RTS
symbol into the public interface.

Accuracy is improved in 2 ways:

* Use of the monotonic clock where available
* Measuring the total time spent waiting instead of a sum
  of intervals (between which there are small gaps)

Reviewers: bgamari, austin, hvr, erikd, simonmar

Reviewed By: bgamari

Subscribers: rwbarton, thomie

Differential Revision: https://phabricator.haskell.org/D3953

orzo in `#ghc` reported seeing a crash due to the retainer profiler encountering
a BLOCKING_QUEUE closure, which isRetainer didn't know about. I performed an
audit to make sure that all of the valid closure types were listed; they
weren't. This is my guess of how they should appear.

Test Plan: Validate

Reviewers: simonmar, austin, erikd

Reviewed By: simonmar

Subscribers: rwbarton, thomie

GHC Trac Issues: #14235

Differential Revision: https://phabricator.haskell.org/D3967

This adds a function to the RTS linker API which lets the
user check the status of dynamically linked objects.

It was initially proposed by @afarmer in D2068.
It's useful for testing the linker and also for detecting retention
problems in production.

It takes a path, because it's easier to use path as key instead of producing
some stable handle.

It returns an enum instead of bool, because I see no reason for destroying
information. All the complexity is already out in the open, so there's
nothing to save the users from.

Test Plan: ./validate

Reviewers: simonmar, Phyx, bgamari, austin, erikd

Reviewed By: Phyx, bgamari

Subscribers: rwbarton, afarmer, thomie

Differential Revision: https://phabricator.haskell.org/D3963

This sprintf is safe thanks to the guarantees on the format strings that
we pass to it.  Well, almost. The GR_FILENAME_FMT_GUM format would not
have satisfied them if it was still used.

If someone makes a mistake that's a potential privilege escalation,
so I think it's reasonable to switch to snprintf to protect against
that remote possibility.

Test Plan: it builds, CI

Reviewers: simonmar, bgamari, austin, erikd

Reviewed By: bgamari

Subscribers: rwbarton, thomie

Differential Revision: https://phabricator.haskell.org/D3944

This trailing comma snuck in in a recent patch. There is nothing wrong with the
comma; it's perfectly valid C99, yet nevertheless Mac OS X's dtrace utility
chokes on it with,

    dtrace: failed to compile script rts/RtsProbes.d:
            "includes/rts/EventLogFormat.h", line 245: syntax error near "}"
    make[1]: *** [rts/dist/build/RtsProbes.h] Error 1