GitLab

Summary:
This commit makes various improvements and addresses some issues with
Compact Regions (aka Compact Normal Forms).

This was the most important thing I wanted to fix.  Compaction
previously prevented GC from running until it was complete, which
would be a problem in a multicore setting.  Now, we compact using a
hand-written Cmm routine that can be interrupted at any point.  When a
GC is triggered during a sharing-enabled compaction, the GC has to
traverse and update the hash table, so this hash table is now stored
in the StgCompactNFData object.

Previously, compaction consisted of a deepseq using the NFData class,
followed by a traversal in C code to copy the data.  This is now done
in a single pass with hand-written Cmm (see rts/Compact.cmm). We no
longer use the NFData instances, instead the Cmm routine evaluates
components directly as it compacts.

The new compaction is about 50% faster than the old one with no
sharing, and a little faster on average with sharing (the cost of the
hash table dominates when we're doing sharing).

Static objects that don't (transitively) refer to any CAFs don't need
to be copied into the compact region.  In particular this means we
often avoid copying Char values and small Int values, because these
are static closures in the runtime.

Each Compact# object can support a single compactAdd# operation at any
given time, so the Data.Compact library now enforces mutual exclusion
using an MVar stored in the Compact object.

We now get exceptions rather than killing everything with a barf()
when we encounter an object that cannot be compacted (a function, or a
mutable object).  We now also detect pinned objects, which can't be
compacted either.

The Data.Compact API has been refactored and cleaned up.  A new
compactSize operation returns the size (in bytes) of the compact
object.

Most of the documentation is in the Haddock docs for the compact
library, which I've expanded and improved here.

Various comments in the code have been improved, especially the main
Note [Compact Normal Forms] in rts/sm/CNF.c.

I've added a few tests, and expanded a few of the tests that were
there.  We now also run the tests with GHCi, and in a new test way
that enables sanity checking (+RTS -DS).

There's a benchmark in libraries/compact/tests/compact_bench.hs for
measuring compaction speed and comparing sharing vs. no sharing.

The field totalDataW in StgCompactNFData was unnecessary.

Test Plan:
* new unit tests
* validate
* tested manually that we can compact Data.Aeson data

Reviewers: gcampax, bgamari, ezyang, austin, niteria, hvr, erikd

Subscribers: thomie, simonpj

Differential Revision: https://phabricator.haskell.org/D2751

GHC Trac Issues: #12455

This brings in initial support for compact regions, as described in the
ICFP 2015 paper "Efficient Communication and Collection with Compact
Normal Forms" (Edward Z. Yang et.al.) and implemented by Giovanni
Campagna.

Some things may change before the 8.2 release, but I (Simon M.) wanted
to get the main patch committed so that we can iterate.

What documentation there is is in the Data.Compact module in the new
compact package.  We'll need to extend and polish the documentation
before the release.

Test Plan:
validate
(new test cases included)

Reviewers: ezyang, simonmar, hvr, bgamari, austin

Subscribers: vikraman, Yuras, RyanGlScott, qnikst, mboes, facundominguez, rrnewton, thomie, erikd

Differential Revision: https://phabricator.haskell.org/D1264

GHC Trac Issues: #11493

Adds a primitive operation to determine whether a particular
`MutableByteArray#` is backed by a pinned buffer.

Test Plan: Validate with included testcase

Reviewers: austin, simonmar

Reviewed By: austin, simonmar

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D2217

GHC Trac Issues: #12059

This patch fixes Cmm generation required to produce histograms when
compiling with -ticky flag, strips dead code from rts/Ticky.c and
reworks it to use a shared constant in both C and Haskell code.

Fixes #8308.

Test Plan: T8308

Reviewers: jstolarek, simonpj, austin

Reviewed By: simonpj

Subscribers: mpickering, simonpj, bgamari, mlen, thomie, jstolarek

Differential Revision: https://phabricator.haskell.org/D931

GHC Trac Issues: #8308

Missing a close paren.

xcrun --find seems like the appropriate choice here. Thanks to Brandon
Allbery for suggesting this.

Thanks to George Colpitts for the suggestion.

Add a sanity check ensuring that nm emits valid hexadecimal output, as
required by POSIX. See #11744 for motivation.

Reviewers: austin, bgamari

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D2113

GHC Trac Issues: #11744

Various people (myself included) have complained about the lack of
useful descriptions for the various packages included in GHC's source
tree. Fix this.

Test Plan: Validate

Reviewers: austin, thomie

Reviewed By: thomie

Subscribers: angerman, ezyang

Differential Revision: https://phabricator.haskell.org/D1736

On IBM AIX `nm` doesn't support reporting symbol sizes, so we need to
resort to `objdump` instead, which has a peculiar output format on AIX.

depends on D1499

Reviewers: austin, bgamari

Reviewed By: bgamari

Subscribers: kgardas, thomie

Differential Revision: https://phabricator.haskell.org/D1500

`derivedConstants` currently uses `System.Info.os` for decisions (which
doesn't necessarily reflect the build-target), as well as hardcoding
"/usr/bin/objdump" for openbsd.

This patch auto-detects `objdump` similiar to how `nm` is detected via
Autoconf as well as passing the target-os into `derivedConstants` via
commandline.

Reviewers: austin, kgardas, erikd, bgamari

Reviewed By: kgardas, erikd, bgamari

Subscribers: kgardas, thomie, erikd

Differential Revision: https://phabricator.haskell.org/D1499

This is needed for #10374 (but doesn't fix it yet).

Also rename DeriveConstants.hs to Main.hs, because the build
system has trouble with Main modules not called Main.hs.

Differential Revision: https://phabricator.haskell.org/D1380

Rename StgArrWords to StgArrBytes (see Trac #8552)

Reviewed By: austin

Differential Revision: https://phabricator.haskell.org/D1233

GHC Trac Issues: #8552

Summary:
We were not treating the offset as a signed field in this rare case,
so it would blow up if the offset was negative.

Test Plan: Looked at the assembly

Reviewers: austin, bgamari, rwbarton

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D1042

Summary:
clearNursery resets all the bd->free pointers of nursery blocks to
make the blocks empty.  In profiles we've seen clearNursery taking
significant amounts of time particularly with large -N and -A values.

This patch moves the work of clearNursery to the point at which we
actually need the new block, thereby introducing an invariant that
blocks to the right of the CurrentNursery pointer still need their
bd->free pointer reset.  This should make things faster overall,
because we don't need to clear blocks that we don't use.

Test Plan: validate

Reviewers: AndreasVoellmy, ezyang, austin

Subscribers: thomie, carter, ezyang, simonmar

Differential Revision: https://phabricator.haskell.org/D318

This reverts commit f0fcc41d.

New changes: now works on 32-bit platforms too.  I added some basic
support for 64-bit subtraction and comparison operations to the x86
NCG.

Summary: OpenBSD's nm doesn't support the -P option and there appears to be no other way to get the desired information from it.

Reviewers: kgardas, #ghc, austin

Reviewed By: kgardas, #ghc, austin

Subscribers: austin, ggreif

Projects: #ghc

Differential Revision: https://phabricator.haskell.org/D332

GHC Trac Issues: #9549

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

This was generated by applying `-ddump-minimal-imports` and addresses
the current compile failure (see #9016) with GHC HEAD due to the new
`die` being exported by `System.Exit`
Signed-off-by: Herbert Valerio Riedel <hvr@gnu.org>

Fixes #8783.

In order to avoid querying the nm version that does not work on Mac OS X
we use the "nm -P" output that is supposed to produce (more portable)
POSIX output and works on all tested OSes (MinGW, Mac OS X, Solaris and
Linux using GNU nm) although slightly different (as documented). The "nm
-P" output is actually only needed to recognize the output of a non-GNU
Solaris nm (all other OSes produce sane outut using "nm" only).
Signed-off-by: Austin Seipp <austin@well-typed.com>

Problems were found on 32-bit platforms, I'll commit again when I have a fix.

This reverts the following commits:
   54b31f74
   b0534f78

This tracks the amount of memory allocation by each thread in a
counter stored in the TSO.  Optionally, when the counter drops below
zero (it counts down), the thread can be sent an asynchronous
exception: AllocationLimitExceeded.  When this happens, given a small
additional limit so that it can handle the exception.  See
documentation in GHC.Conc for more details.

Allocation limits are similar to timeouts, but

  - timeouts use real time, not CPU time.  Allocation limits do not
    count anything while the thread is blocked or in foreign code.

  - timeouts don't re-trigger if the thread catches the exception,
    allocation limits do.

  - timeouts can catch non-allocating loops, if you use
    -fno-omit-yields.  This doesn't work for allocation limits.

I couldn't measure any impact on benchmarks with these changes, even
for nofib/smp.

These array types are smaller than Array# and MutableArray# and are
faster when the array size is small, as they don't have the overhead
of a card table. Having no card table reduces the closure size with 2
words in the typical small array case and leads to less work when
updating or GC:ing the array.

Reduces both the runtime and memory allocation by 8.8% on my insert
benchmark for the HashMap type in the unordered-containers package,
which makes use of lots of small arrays. With tuned GC settings
(i.e. `+RTS -A6M`) the runtime reduction is 15%.

Fixes #8923.

It causes a failure on Windows right now.

This reverts commit 045b2803.

The patch provided by Christian Maeder <Christian.Maeder@dfki.de>
Signed-off-by: Karel Gardas <karel.gardas@centrum.cz>
Signed-off-by: Austin Seipp <austin@well-typed.com>

This results in a 57% runtime decrease when allocating an array of 128
bytes on a 64-bit machine.

Fixes #8876.

This reverts commit 2f5db98e.

We were using SSE is some places and XMM in others. Better to keep a consistent
naming scheme.

The commit replaces mkWeakForeignEnv# with addCFinalizerToWeak#.
This new primop mutates an existing Weak# object and adds a new
C finalizer to it.

This change removes an invariant in MarkWeak.c, namely that the relative
order of Weak# objects in the list needs to be preserved across GC. This
makes it easier to split the list into per-generation structures.

The patch also removes a race condition between two threads calling
finalizeWeak# on the same WEAK object at that same time.

  * the new StgCmmArgRep module breaks a dependency cycle; I also
    untabified it, but made no real changes

  * updated the documentation in the wiki and change the user guide to
    point there

  * moved the allocation enters for ticky and CCS to after the heap check

    * I left LDV where it was, which was before the heap check at least
      once, since I have no idea what it is

  * standardized all (active?) ticky alloc totals to bytes

  * in order to avoid double counting StgCmmLayout.adjustHpBackwards
    no longer bumps ALLOC_HEAP_ctr

  * I resurrected the SLOW_CALL counters

    * the new module StgCmmArgRep breaks cyclic dependency between
      Layout and Ticky (which the SLOW_CALL counters cause)

    * renamed them SLOW_CALL_fast_<pattern> and VERY_SLOW_CALL

  * added ALLOC_RTS_ctr and _tot ticky counters

    * eg allocation by Storage.c:allocate or a BUILD_PAP in stg_ap_*_info

    * resurrected ticky counters for ALLOC_THK, ALLOC_PAP, and
      ALLOC_PRIM

    * added -ticky and -DTICKY_TICKY in ways.mk for debug ways

  * added a ticky counter for total LNE entries

  * new flags for ticky: -ticky-allocd -ticky-dyn-thunk -ticky-LNE

    * all off by default

    * -ticky-allocd: tracks allocation *of* closure in addition to
       allocation *by* that closure

    * -ticky-dyn-thunk tracks dynamic thunks as if they were functions

    * -ticky-LNE tracks LNEs as if they were functions

  * updated the ticky report format, including making the argument
    categories (more?) accurate again

  * the printed name for things in the report include the unique of
    their ticky parent as well as if they are not top-level

We now define _PROGNAME, and _PROG is automatically defined with
$(exeext). This will shortly automatically use the right exeext
depending on what stage it is being compiled with (exeext may be
different for different stages when cross-compiling).

To improve performance of StablePtr.