GitLab

Because garbage collector calls `retainerProfile()` and `heapCensus()`,
GC times normally include some of PROF times too. To fix this we have
these lines:

    // heapCensus() is called by the GC, so RP and HC time are
    // included in the GC stats.  We therefore subtract them to
    // obtain the actual GC cpu time.
    stats.gc_cpu_ns      -=  prof_cpu;
    stats.gc_elapsed_ns  -=  prof_elapsed;

These variables are later used for calculating GC time excluding the
final GC (which should be attributed to EXIT).

    exit_gc_elapsed      = stats.gc_elapsed_ns - start_exit_gc_elapsed;

The problem is if we subtract PROF times from `gc_elapsed_ns` and then
subtract `start_exit_gc_elapsed` from the result, we end up subtracting
PROF times twice, because `start_exit_gc_elapsed` also includes PROF
times.

We now subtract PROF times from GC after the calculations for EXIT and
MUT times. The existing assertion that checks

    INIT + MUT + GC + EXIT = TOTAL

now holds. When we subtract PROF numbers from GC, and a new assertion

    INIT + MUT + GC + PROF + EXIT = TOTAL

also holds.

Fixes #15897. New assertions added in this commit also revealed #16102,
which is also fixed by this commit.

 * Mark arith011 as broken with integer-simple

   As noted in #16091, arith011 fails when run against integer-simple with a
   "divide by zero" exception. This suggests that integer-gmp and integer-simple
   are handling division by zero differently.

 * This also fixes broken_without_gmp; the lack of types made the previous
   failure silent, sadly. Improves situation of #16043.

 * Mark several tests implicitly depending upon integer-gmp as broken
   with integer-simple. These expect to see Core coming from integer-gmp,
   which breaks with integer-simple.

 * Increase runtime timeout multiplier of T11627a with integer-simple

   I previously saw that T11627a timed out in all profiling ways when run against
   integer-simple. I suspect this is due to integer-simple's rather verbose heap
   representation. Let's see whether increasing the runtime timeout helps.

   Fixes test for #11627.

This is all in service of fixing #16043.

The retainer profiler no longer uses the C stack for its mark stack (#14758).
Consequently even the small C stack provided on Darwin should be sufficient to
run this test. See #11627

As documented in #15382, this is known to fail in prof_hc_hb on i386.
Concerningly, I have also seen this test non-deterministically fail in
prof_hc_hb on amd64. We should really investigate this.

CONSTR_NOCAF was introduced with 55d535da as a replacement for
CONSTR_STATIC and CONSTR_NOCAF_STATIC, however, as explained in Note
[static constructors], we copy CONSTR_NOCAFs (which can also be seen in
evacuate) during GC, and they can become dead, like other CONSTR_X_Ys.
processHeapClosureForDead is updated to reflect this.

Test Plan: Validates on x86_64. Existing failures on i386.

Reviewers: simonmar, bgamari, erikd

Reviewed By: simonmar, bgamari

Subscribers: rwbarton, thomie, carter

GHC Trac Issues: #7836, #15063, #15087, #15165

Differential Revision: https://phabricator.haskell.org/D4928

Darwin tends to give us a very small stack which the retainer profiler tends to
overflow. Strangely, this manifested on CircleCI yet not Harbormaster.

See #15287 and #11627.

Due to #15063.

This finally gets us to a green ./validate --slow on linux for a ghc
checkout from the beginning of this week, see

  https://circleci.com/gh/ghc/ghc/4739

This is hopefully the final (or second to final) patch to
address #14890.

Test Plan: ./validate --slow

Reviewers: bgamari, hvr, simonmar

Reviewed By: bgamari

Subscribers: rwbarton, thomie, carter

GHC Trac Issues: #14890

Differential Revision: https://phabricator.haskell.org/D4712

This was broken by D3746 and/or D3809, but unfortunately we didn't
notice because CI at the time wasn't building the profiling way.

Test Plan:
```
cd testsuite/test/profiling/should_run
make WAY=ghci-ext-prof
```

Reviewers: bgamari, michalt, hvr, erikd

Subscribers: rwbarton, thomie, carter

GHC Trac Issues: #14705

Differential Revision: https://phabricator.haskell.org/D4437

CircleCI seems to suggest that the cost center ordering is wrong in the prof
way. I'm beginning to wonder whether there is some non-determinism here. If only
I know what this test was supposed to be testing.

These two tests have been failing on CircleCI.

- T12962: just fix function locations
- scc003: reorder cost centres, cost centre `f.(...)` renamed to `f.x'`

Reviewers: bgamari

Subscribers: rwbarton, thomie, carter

GHC Trac Issues: #14704

Differential Revision: https://phabricator.haskell.org/D4351

Subscribers: rwbarton, thomie, duog

GHC Trac Issues: #14257

Differential Revision: https://phabricator.haskell.org/D4201

Instead of excluding `isAnyInlinePragma`, just exclude
`isInlinePragma`. This makes GHC behave as documented;
the user's guide only indicates that GHC does not automatically
add SCCs to `INLINE` bindings.

Fixes #12962.

Reviewers: austin, bgamari

Reviewed By: bgamari

Subscribers: DemiMarie, osa1, Mikolaj, simonpj, rwbarton, thomie

GHC Trac Issues: #12962

Differential Revision: https://phabricator.haskell.org/D3550

The script I used is included as testsuite/driver/kill_extra_files.py,
though at this point it is for mostly historical interest.

Some of the tests in libraries/hpc relied on extra_files.py, so this
commit includes an update to that submodule.

One test in libraries/process also relies on extra_files.py, but we
cannot update that submodule so easily, so for now we special-case it
in the test driver.

The `clean_cmd` and `extra_clean` setup functions don't do anything.
Remove them from .T files.

Created using https://github.com/thomie/refactor-ghc-testsuite. This
diff is a test for the .T-file parser/processor/pretty-printer in that
repository.

    find . -name '*.T' -exec ~/refactor-ghc-testsuite/Main "{}" \;

Tests containing inline comments or multiline strings are not modified.

Preparation for #12223.

Test Plan: Harbormaster

Reviewers: austin, hvr, simonmar, mpickering, bgamari

Reviewed By: mpickering

Subscribers: mpickering

Differential Revision: https://phabricator.haskell.org/D3000

GHC Trac Issues: #12223

* In stg_ap_0_fast, if we're evaluating a thunk, the thunk might
  evaluate to a function in which case we may have to adjust its CCS.

* The interpreter has its own implementation of stg_ap_0_fast, so we
  have to do the same shenanigans with creating empty PAPs and copying
  PAPs there.

* GHCi creates Cost Centres as children of CCS_MAIN, which enterFunCCS()
  wrongly assumed to imply that they were CAFs.  Now we use the is_caf
  flag for this, which we have to correctly initialise when we create a
  Cost Centre in GHCi.

We don't have any other tests for this, except one Template Haskell
test.  This would have caught the bug I just fixed in D2868, at least
when validating with profiling on.

Test Plan: Ran tests

Reviewers: niteria, austin, erikd, bgamari

Reviewed By: bgamari

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D2869

GHC Trac Issues: #5654

This fixes some cases of wrong stacks being generated by the profiler.
For background and details on the fix see
`Note [Evaluating functions with profiling]` in `rts/Apply.cmm`.

This does have an impact on allocations for some programs when
profiling.  nofib results:

```
   k-nucleotide          +0.0%     +8.8%    +11.0%    +11.0%      0.0%
         puzzle          +0.0%    +12.5%     0.244     0.246      0.0%
      typecheck           0.0%     +8.7%    +16.1%    +16.2%      0.0%
------------------------------------------------------------------------
--------
            Min          -0.0%     -0.0%    -34.4%    -35.5%    -25.0%
            Max          +0.0%    +12.5%    +48.9%    +49.4%    +10.6%
 Geometric Mean          +0.0%     +0.6%     +2.0%     +1.8%     -0.3%

```

But runtimes don't seem to be affected much, and the examples I looked
at were completely legitimate.  For example, in puzzle we have this:

```
position :: ItemType -> StateType ->  BankType
position Bono = bonoPos
position Edge = edgePos
position Larry = larryPos
position Adam = adamPos
```

where the identifiers on the rhs are all record selectors.  Previously
the profiler gave a stack that looked like

```
  position
  bonoPos
  ...
```

i.e. `bonoPos` was at the same level of the call stack as `position`,
but now it looks like

```
  position
   bonoPos
   ...
```

I used the normaliser from the testsuite to diff the profiling output
from other nofib programs and they all looked better.

Test Plan:
* the broken test passes
* validate
* compiled and ran all of nofib, measured perf, diff'd several .prof
files

Reviewers: niteria, erikd, austin, scpmw, bgamari

Reviewed By: bgamari

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D2804

GHC Trac Issues: #5654, #10007

Looks like e3f341f3 broke it, but we
wouldn't have noticed because the test isn't run in validate.

I think the change is OK.  We don't get a tick for x' any more, because
it is converted directly to a case instead of a let-binding, but we
still get a tick for the "pattern binding" (!x').  This could be better
- f.(...) isn't all that intuitive - but it's not strictly wrong.

Test Plan: `validate --slow`

Reviewers: austin

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D2536

GHC Trac Issues: #12019

Not having SCCs at the top level is becoming annoying real quick. For
simplest cases, it's possible to do this transformation:

    f x y = ...
    =>
    f = {-# SCC f #-} \x y -> ...

However, it doesn't work when there's a `where` clause:

    f x y = <t is in scope>
      where t = ...
    =>
    f = {-# SCC f #-} \x y -> <t is out of scope>
      where t = ...

Or when we have a "equation style" definition:

    f (C1 ...) = ...
    f (C2 ...) = ...
    f (C3 ...) = ...
    ...

(usual solution is to rename `f` to `f'` and define a new `f` with a
`SCC`)

This patch implements support for SCC annotations in declaration
contexts. This is now a valid program:

    f x y = ...
      where
        g z = ...
        {-# SCC g #-}
    {-# SCC f #-}

Test Plan: This passes slow validate (no new failures added).

Reviewers: goldfire, mpickering, austin, bgamari, simonmar

Reviewed By: bgamari, simonmar

Subscribers: simonmar, thomie, mpickering

Differential Revision: https://phabricator.haskell.org/D2407

* T7837 is still broken for prof_ways (#9406)
* T11627b is broken on Windows for WAY=prof_hc_hb (#12236)
* T8089 is also broken for WAY=profasm on Windows

This fixes the problem with duplicate cost-centre names that was
reported a couple of times before. When a module implements a typeclass
multiple times for different types, methods of different implementations
get same cost-centre names and are reported like this:

    COST CENTRE MODULE            %time %alloc

    CAF         GHC.IO.Handle.FD    0.0   32.8
    CAF         GHC.Read            0.0    1.0
    CAF         GHC.IO.Encoding     0.0    1.8
    showsPrec   Main                0.0    1.2
    readPrec    Main                0.0   19.4
    readPrec    Main                0.0   20.5
    main        Main                0.0   20.2

                                            individual      inherited
    COST CENTRE  MODULE  no.     entries  %time %alloc   %time %alloc

    MAIN         MAIN     53          0    0.0    0.2     0.0  100.0
     CAF         Main    105          0    0.0    0.3     0.0   62.5
      readPrec   Main    109          1    0.0    0.6     0.0    0.6
      readPrec   Main    107          1    0.0    0.6     0.0    0.6
      main       Main    106          1    0.0   20.2     0.0   61.0
       ==        Main    114          1    0.0    0.0     0.0    0.0
       ==        Main    113          1    0.0    0.0     0.0    0.0
       showsPrec Main    112          2    0.0    1.2     0.0    1.2
       showsPrec Main    111          2    0.0    0.9     0.0    0.9
       readPrec  Main    110          0    0.0   18.8     0.0   18.8
       readPrec  Main    108          0    0.0   19.9     0.0   19.9

It's not possible to tell from the report which `==` took how long. This
patch adds one more column at the cost of making outputs wider. The
report now looks like this:

    COST CENTRE MODULE           SRC                       %time %alloc

    CAF         GHC.IO.Handle.FD <entire-module>             0.0   32.9
    CAF         GHC.IO.Encoding  <entire-module>             0.0    1.8
    CAF         GHC.Read         <entire-module>             0.0    1.0
    showsPrec   Main             Main_1.hs:7:19-22           0.0    1.2
    readPrec    Main             Main_1.hs:7:13-16           0.0   19.5
    readPrec    Main             Main_1.hs:4:13-16           0.0   20.5
    main        Main             Main_1.hs:(10,1)-(20,20)    0.0   20.2

                                                                       individual      inherited
    COST CENTRE  MODULE        SRC                      no. entries  %time %alloc   %time %alloc

    MAIN         MAIN          <built-in>                53      0    0.0    0.2     0.0  100.0
     CAF         Main          <entire-module>          105      0    0.0    0.3     0.0   62.5
      readPrec   Main          Main_1.hs:7:13-16        109      1    0.0    0.6     0.0    0.6
      readPrec   Main          Main_1.hs:4:13-16        107      1    0.0    0.6     0.0    0.6
      main       Main          Main_1.hs:(10,1)-(20,20) 106      1    0.0   20.2     0.0   61.0
       ==        Main          Main_1.hs:7:25-26        114      1    0.0    0.0     0.0    0.0
       ==        Main          Main_1.hs:4:25-26        113      1    0.0    0.0     0.0    0.0
       showsPrec Main          Main_1.hs:7:19-22        112      2    0.0    1.2     0.0    1.2
       showsPrec Main          Main_1.hs:4:19-22        111      2    0.0    0.9     0.0    0.9
       readPrec  Main          Main_1.hs:7:13-16        110      0    0.0   18.8     0.0   18.8
       readPrec  Main          Main_1.hs:4:13-16        108      0    0.0   19.9     0.0   19.9
     CAF         Text.Read.Lex <entire-module>          102      0    0.0    0.5     0.0    0.5

To fix failing test cases because of different orderings of cost centres
(e.g. optimized and non-optimized build printing in different order),
with this patch we also start sorting cost centres before printing. The
order depends on 1) entries (more entered cost centres come first) 2)
names (using strcmp() on cost centre names).

Reviewers: simonmar, austin, erikd, bgamari

Reviewed By: simonmar, bgamari

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D2282

GHC Trac Issues: #11543, #8473, #7105

Change help message so it doesn't specify -auto-all.
Make old profiling flags deprecated as they are no longer
documented.
Update Makefile and documentation accordingly.
Update release notes for ghc 8.2

Test Plan:
./verify; `ghc --help` shouldn't specify the -auto-all
flag. Furthermore `ghc -fprof -auto-all` should emit a warning

Reviewed By: thomie, austin

Differential Revision: https://phabricator.haskell.org/D2257

GHC Trac Issues: #12084

Update submodule nofib

Test Plan: validate

Reviewers: carter, austin, simonmar, bgamari

Reviewed By: simonmar, bgamari

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D2154

GHC Trac Issues: #11978

The heap census now handles large ARR_WORDS objects which have
been shrunk by shrinkMutableByteArray# or resizeMutableByteArray#.

Test Plan: ./validate && make test WAY=profasm

Reviewers: hvr, bgamari, austin, thomie

Reviewed By: thomie

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D2005

GHC Trac Issues: #11627

These tests fail not only for WAY=prof, but also for WAY=profllvm.

Refactoring only. I compared before and after with 'make slow', and it
still runs each test with the same 'ways' as before.