GitLab

Without this change RTS typically doesn't flush some important
events until the process terminates or it doesn't write them at
all in case it terminates abnormally.

Here is a list of such events:

* EVENT_WALL_CLOCK_TIME
* EVENT_OS_PROCESS_PID
* EVENT_OS_PROCESS_PPID
* EVENT_RTS_IDENTIFIER
* EVENT_PROGRAM_ARGS
* EVENT_PROGRAM_ENV

An additional stat is tracked per gc: par_balanced_copied This is the
the number of bytes copied by each gc thread under the balanced lmit,
which is simply (copied_bytes / num_gc_threads).  The stat is added to
all the appropriate GC structures, so is visible in the eventlog and in
GHC.Stats.

A note is added explaining how work balance is computed.

Remove some end of line whitespace

Test Plan:
./validate
experiment with the program attached to the ticket
examine code changes carefully

Reviewers: simonmar, austin, hvr, bgamari, erikd

Reviewed By: simonmar

Subscribers: Phyx, rwbarton, thomie

GHC Trac Issues: #13830

Differential Revision: https://phabricator.haskell.org/D3658

Our new CPP linter enforces this.

Currently eventlog data is always written to a file `progname.eventlog`.
This patch introduces the `flushEventLog` field in `RtsConfig` which
allows to customize the writing of eventlog data.

One possible scenario is the ongoing live-profile-monitor effort by
@NCrashed which slurps all eventlog data through `fluchEventLog`.

`flushEventLog` takes a buffer with eventlog data and its size and
returns `false` (0) in case eventlog data could not be procesed.

Reviewers: simonmar, austin, erikd, bgamari

Reviewed By: simonmar, bgamari

Subscribers: qnikst, thomie, NCrashed

Differential Revision: https://phabricator.haskell.org/D2934

Test Plan: Validate on lots of platforms

Reviewers: erikd, simonmar, austin

Reviewed By: erikd, simonmar

Subscribers: michalt, thomie

Differential Revision: https://phabricator.haskell.org/D2699

Summary:
I was getting annoyed by cap/capset messages when using +RTS -DS, which
doesn't cause any other trace messages to be emitted.  This makes it
possible to add --with-rtsopts=-DS when running tests, and not have all
the tests fail due to spurious trace messages.

Test Plan: validate

Reviewers: duncan, bgamari, ezyang, austin, erikd

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D2438

Test Plan: Try it

Reviewers: hvr, simonmar, austin, erikd

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D1722

GHC Trac Issues: #11094

Summary:
Specifcally we want the MinGW compiler to use ISO print format
specfifiers.

Test Plan: Validate on Linux, OS X and Windows

Reviewers: Phyx, austin, bgamari, simonmar

Reviewed By: bgamari, simonmar

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D2192

The `nat` type was an alias for `unsigned int` with a comment saying
it was at least 32 bits. We keep the typedef in case client code is
using it but mark it as deprecated.

Test Plan: Validated on Linux, OS X and Windows

Reviewers: simonmar, austin, thomie, hvr, bgamari, hsyl20

Differential Revision: https://phabricator.haskell.org/D2166

Build system support for Cygwin was removed in b6be81b8.

Test Plan:
 - Validate on x86_64/linux
 - Cross-compile rts/RtsSymbols.c and rts/Linker.c to Windows using the
   i686-w64-mingw32-gcc and x86_64-w64-mingw32-gcc cross compilers.

Reviewers: hvr, austin, bgamari

Reviewed By: bgamari

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D1371

This has been unnecessary for quite some time due to the create/delete
capability events.

Summary:
Don't call postLogMsg to post a user msg, because it truncates messages to
512 bytes.

Rename traceCap_stderr and trace_stderr to vtraceCap_stderr and trace_stderr,
to signal that they take a va_list (similar to vdebugBelch vs debugBelch).

See #3874 for the original reason behind traceFormatUserMsg.

See the commit msg in #9395 (d360d440) for a discussion about using
null-terminated strings vs strings with an explicit length.

Test Plan:
Run `cabal install ghc-events` and inspect the result of `ghc-events show` on
an eventlog file created with `ghc -eventlog Test.hs` and `./Test +RTS -l`,
where Test.hs contains:

```
import Debug.Trace
main = traceEvent (replicate 510 'a' ++ "bcd") $ return ()
```

Depends on D655.

Reviewers: austin

Reviewed By: austin

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D656

GHC Trac Issues: #8309

Summary: See also ab9711d8.

Reviewers: austin

Reviewed By: austin

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D655

This reverts commit 39b5c1cb.

Signed-off-by: Austin Seipp <austin@well-typed.com>

This will hopefully help ensure some basic consistency in the forward by
overriding buffer variables. In particular, it sets the wrap length, the
offset to 4, and turns off tabs.
Signed-off-by: Austin Seipp <austin@well-typed.com>

We add the invariant to the MVar blocked threads queue that
threads blocked on an atomic read are always at the front of
the queue.  This invariant is easy to maintain, since takers
are only ever added to the end of the queue.
Signed-off-by: Edward Z. Yang <ezyang@mit.edu>

In time-based profiling visualisations (e.g. heap profiles and ThreadScope)
it would be useful to be able to mark particular points in the execution and
have those points in time marked in the visualisation.

The traceMarker# primop currently emits an event into the eventlog. In
principle it could be extended to do something in the heap profiling too.

lnat was originally "long unsigned int" but we were using it when we
wanted a 64-bit type on a 64-bit machine.  This broke on Windows x64,
where long == int == 32 bits.  Using types of unspecified size is bad,
but what we really wanted was a type with N bits on an N-bit machine.
StgWord is exactly that.

lnat was mentioned in some APIs that clients might be using
(e.g. StackOverflowHook()), so we leave it defined but with a comment
to say that it's deprecated.

Based on initial patches by Mikolaj Konarski <mikolaj@well-typed.com>

These new eventlog events are to let profiling tools keep track of all
the OS threads that belong to an RTS capability at any moment in time.
In the RTS, OS threads correspond to the Task abstraction, so that is
what we track. There are events for tasks being created, migrated
between capabilities and deleted. In particular the task creation event
also records the kernel thread id which lets us match up the OS thread
with data collected by others tools (in the initial use case with
Linux's perf tool, but in principle also with DTrace).

Mostly this meant getting pointer<->int conversions to use the right
sizes. lnat is now size_t, rather than unsigned long, as that seems a
better match for how it's used.

Quoting design rationale by dcoutts: The event indicates that we're doing
a stop-the-world GC and all other HECs should be between their GC_START
and GC_END events at that moment. We don't want to use GC_STATS_GHC
for that, because GC_STATS_GHC is for extra GHC-specific info,
not something we have to rely on to be able to match the GC pauses
across HECs to a particular global GC.

There was a discrepancy between GC times reported in +RTS -s
and the timestamps of GC_START and GC_END events on the cap,
on which +RTS -s stats for the given GC are based.
This is fixed by posting the events with exactly the same timestamp
as generated for the stat calculation. The calls posting the events
are moved too, so that the events are emitted close to the time instant
they claim to be emitted at. The GC_STATS_GHC was moved, too, ensuring
it's emitted before the moved GC_END on all caps, which simplifies tools code.

They cover much the same info as is available via the GHC.Stats module
or via the '+RTS -s' textual output, but via the eventlog and with a
better sampling frequency.

We have three new generic heap info events and two very GHC-specific
ones. (The hope is the general ones are usable by other implementations
that use the same eventlog system, or indeed not so sensitive to changes
in GHC itself.)

The general ones are:

 * total heap mem allocated since prog start, on a per-HEC basis
 * current size of the heap (MBlocks reserved from OS for the heap)
 * current size of live data in the heap

Currently these are all emitted by GHC at GC time (live data only at
major GC).

The GHC specific ones are:

 * an event giving various static heap paramaters:
   * number of generations (usually 2)
   * max size if any
   * nursary size
   * MBlock and block sizes
 * a event emitted on each GC containing:
   * GC generation (usually just 0,1)
   * total bytes copied
   * bytes lost to heap slop and fragmentation
   * the number of threads in the parallel GC (1 for serial)
   * the maximum number of bytes copied by any par GC thread
   * the total number of bytes copied by all par GC threads
     (these last three can be used to calculate an estimate of the
      work balance in parallel GCs)

Now that we can adjust the number of capabilities on the fly, we need
this reflected in the eventlog. Previously the eventlog had a single
startup event that declared a static number of capabilities. Obviously
that's no good anymore.

For compatability we're keeping the EVENT_STARTUP but adding new
EVENT_CAP_CREATE/DELETE. The EVENT_CAP_DELETE is actually just the old
EVENT_SHUTDOWN but renamed and extended (using the existing mechanism
to extend eventlog events in a compatible way). So we now emit both
EVENT_STARTUP and EVENT_CAP_CREATE. One day we will drop EVENT_STARTUP.

Since reducing the number of capabilities at runtime does not really
delete them, it just disables them, then we also have new events for
disable/enable.

The old EVENT_SHUTDOWN was in the scheduler class of events. The new
EVENT_CAP_* events are in the unconditional class, along with the
EVENT_CAPSET_* ones. Knowing when capabilities are created and deleted
is crucial to making sense of eventlogs, you always want those events.
In any case, they're extremely low volume.

At present the number of capabilities can only be *increased*, not
decreased.  The latter presents a few more challenges!

The existing GHC.Conc.labelThread will now also emit the the thread
label into the eventlog. Profiling tools like ThreadScope could then
use the thread labels rather than thread numbers.

Enables people to turn them on/off. Defaults to on.

Eventlog timestamps are elapsed times (in nanoseconds) relative to the
process start. To be able to merge eventlogs from multiple processes we
need to be able to align their timelines. If they share a clock domain
(or a user judges that their clocks are sufficiently closely
synchronised) then it is sufficient to know how the eventlog timestamps
match up with the clock.

The EVENT_WALL_CLOCK_TIME contains the clock time with (up to)
nanosecond precision. It is otherwise an ordinary event and so contains
the usual timestamp for the same moment in time. It therefore enables
us to match up all the eventlog timestamps with clock time.

Replaces the existing EVENT_RUN/STEAL_SPARK events with 7 new events
covering all stages of the spark lifcycle:
  create, dud, overflow, run, steal, fizzle, gc

The sampled spark events are still available. There are now two event
classes for sparks, the sampled and the fully accurate. They can be
enabled/disabled independently. By default +RTS -l includes the sampled
but not full detail spark events. Use +RTS -lf-p to enable the detailed
'f' and disable the sampled 'p' spark.

Includes work by Mikolaj <mikolaj.konarski@gmail.com>