GitLab

Based on a patch from David Terei.

Some parts are a little ugly (e.g. defining things that only ASSERTs
use only when DEBUG is defined), so we might want to tweak things a
little.

I've also turned off -Werror for didn't-inline warnings, as we now
get a few such warnings.

Previously the code generator generated small code fragments labelled
with __stginit_M for each module M, and these performed whatever
initialisation was necessary for that module and recursively invoked
the initialisation functions for imported modules.  This appraoch had
drawbacks:

 - FFI users had to call hs_add_root() to ensure the correct
   initialisation routines were called.  This is a non-standard,
   and ugly, API.

 - unless we were using -split-objs, the __stginit dependencies would
   entail linking the whole transitive closure of modules imported,
   whether they were actually used or not.  In an extreme case (#4387,
   #4417), a module from GHC might be imported for use in Template
   Haskell or an annotation, and that would force the whole of GHC to
   be needlessly linked into the final executable.

So now instead we do our initialisation with C functions marked with
__attribute__((constructor)), which are automatically invoked at
program startup time (or DSO load-time).  The C initialisers are
emitted into the stub.c file.  This means that every time we compile
with -prof or -hpc, we now get a stub file, but thanks to #3687 that
is now invisible to the user.

There are some refactorings in the RTS (particularly for HPC) to
handle the fact that initialisers now get run earlier than they did
before.

The __stginit symbols are still generated, and the hs_add_root()
function still exists (but does nothing), for backwards compatibility.

This has no effect with static libraries, but when the RTS is in a
shared library it does two things:

 - it prevents the function from being exposed by the shared library

 - internal calls to the function can use the faster non-PLT calls,
   because the function cannot be overriden at link time.

Now that constructor info tables contain the name of the constructor,
we can generate useful heap profiles without requiring the whole
program and libraries to be compiled with -prof.  So now, "+RTS -hT"
generates a heap profile for any program, dividing the profile by
constructor.  It wouldn't be hard to add support for grouping
constructors by module, or to restrict the profile to certain
constructors/modules/packages.

This means that for the first time we can get heap profiles for GHCi,
which was previously impossible because the byte-code
interpreter and linker don't work with -prof.

Most of the other users of the fptools build system have migrated to
Cabal, and with the move to darcs we can now flatten the source tree
without losing history, so here goes.

The main change is that the ghc/ subdir is gone, and most of what it
contained is now at the top level.  The build system now makes no
pretense at being multi-project, it is just the GHC build system.

No doubt this will break many things, and there will be a period of
instability while we fix the dependencies.  A straightforward build
should work, but I haven't yet fixed binary/source distributions.
Changes to the Building Guide will follow, too.

Big re-hash of the threaded/SMP runtime

This is a significant reworking of the threaded and SMP parts of
the runtime.  There are two overall goals here:

  - To push down the scheduler lock, reducing contention and allowing
    more parts of the system to run without locks.  In particular,
    the scheduler does not require a lock any more in the common case.

  - To improve affinity, so that running Haskell threads stick to the
    same OS threads as much as possible.

At this point we have the basic structure working, but there are some
pieces missing.  I believe it's reasonably stable - the important
parts of the testsuite pass in all the (normal,threaded,SMP) ways.

In more detail:

  - Each capability now has a run queue, instead of one global run
    queue.  The Capability and Task APIs have been completely
    rewritten; see Capability.h and Task.h for the details.

  - Each capability has its own pool of worker Tasks.  Hence, Haskell
    threads on a Capability's run queue will run on the same worker
    Task(s).  As long as the OS is doing something reasonable, this
    should mean they usually stick to the same CPU.  Another way to
    look at this is that we're assuming each Capability is associated
    with a fixed CPU.

  - What used to be StgMainThread is now part of the Task structure.
    Every OS thread in the runtime has an associated Task, and it
    can ask for its current Task at any time with myTask().

  - removed RTS_SUPPORTS_THREADS symbol, use THREADED_RTS instead
    (it is now defined for SMP too).

  - The RtsAPI has had to change; we must explicitly pass a Capability
    around now.  The previous interface assumed some global state.
    SchedAPI has also changed a lot.

  - The OSThreads API now supports thread-local storage, used to
    implement myTask(), although it could be done more efficiently
    using gcc's __thread extension when available.

  - I've moved some POSIX-specific stuff into the posix subdirectory,
    moving in the direction of separating out platform-specific
    implementations.

  - lots of lock-debugging and assertions in the runtime.  In particular,
    when DEBUG is on, we catch multiple ACQUIRE_LOCK()s, and there is
    also an ASSERT_LOCK_HELD() call.

What's missing so far:

  - I have almost certainly broken the Win32 build, will fix soon.

  - any kind of thread migration or load balancing.  This is high up
    the agenda, though.

  - various performance tweaks to do

  - throwTo and forkProcess still do not work in SMP mode

gcc 4.0.0 fix: don't declare static total_prof_ticks as extern

Remove uses of stderr from .cmm code.

We can't reliably refer to stdin/stdout/stderr from .cmm, because the
C standard doesn't specify whether stderr is a variable or a macro.

Cleanup: all (well, most) messages from the RTS now go through the
functions in RtsUtils: barf(), debugBelch() and errorBelch().  The
latter two were previously called belch() and prog_belch()
respectively.  See the comments for the right usage of these message
functions.

One reason for doing this is so that we can avoid spurious uses of
stdout/stderr by Haskell apps on platforms where we shouldn't be using
them (eg. non-console apps on Windows).

Various fixes and removal of dead code.

Merge backend-hacking-branch onto HEAD.  Yay!

cosmetic only

- Add a new type of restriction: -hC, which restricts to closures
  whose CCS contains the specified CCs *anywhere* (not just at the
  top).

- Complain if the user tries to request both retainer and biographical
  profiling.  We don't support both simultaneously, because they use
  the same header word in the closure.

- Allow for the fact that the heap might contain some closures which
  don't have a valid retainer set during the heap census.  The only
  known closures of this kind so far are DEAD_WEAK closures.

- Some cruft-removal and renaming of functions to follow conventions.

Retainer Profiling / Lag-drag-void profiling.

This is mostly work by Sungwoo Park, who spent a summer internship at
MSR Cambridge this year implementing these two types of heap profiling
in GHC.

Relative to Sungwoo's original work, I've made some improvements to
the code:

   - it's now possible to apply constraints to retainer and LDV profiles
     in the same way as we do for other types of heap profile (eg.
     +RTS -hc{foo,bar} -hR -RTS gives you a retainer profiling considering
     only closures with cost centres 'foo' and 'bar').

   - the heap-profile timer implementation is cleaned up.

   - heap profiling no longer has to be run in a two-space heap.

   - general cleanup of the code and application of the SDM C coding
     style guidelines.

Profiling will be a little slower and require more space than before,
mainly because closures have an extra header word to support either
retainer profiling or LDV profiling (you can't do both at the same
time).

We've used the new profiling tools on GHC itself, with moderate
success.  Fixes for some space leaks in GHC to follow...

- generate the heap profiling into <prog>.hp, as advertised
- clean up RtsFlags a bit

misc profiling cleanups

- generalise the per-module initialisation stubs so that we use it
  in normal (non-profiled) code too.  The initialisation stubs are
  now called '__init_<module>' rather than '_reg<module>'.

- Register foreign exported functions as stable pointers in the
  initialisation code for the module.  This fixes the foreign export
  problems reported by several people.

- remove the concept of "module groups" from the profiling subsystem.

- change the profiling semantics slightly; it should be unnecessary
  to use '-caf-all' to get reasonable profiles now.

Remove duplicate prof_file variable.  Heap profiling should work again.

Fitst cut at XML-style profiling logs.  Time/allocation profiling only
for now, and the new flag '+RTS -px' enables the new log file format.

Corrected time stamp calculation for .hp files.

Fix for DEBUG && !PROFILING

Heap Profiling
--------------

This commit adds simple hp2ps-style heap profiling.  Just the '-h'
option is supported so far.

Heap profiles will show the top two cost centres on each cost centre
stack.

Time/Alloc profiles have been extended to aggregate the costs on a
per-cost-centre basis and show the table of highest consuming cost
centres before the full table of cost centre stacks.

Profiles work best if '-caf-all' is used, especially when using
one of the '-auto' flags.

Support for thread{WaitRead,WaitWrite,Delay}.  These should behave
identically to the 3.02 implementations.

We now have the virtual timer on during all program runs, which ticks
at 50Hz by default.  This is used to implement threadDelay, so you
won't get any better granularity than the tick frequency
unfortunately.  It remains to be seen whether using the virtual timer
will have a measurable impact on performance for non-threadDelaying
programs.

All operations in the I/O subsystem should now be non-blocking with
respect to other running Haskell threads.  It remains to be seen
whether this will have a measurable performance impact on
non-concurrent programs (probably not).

Copyright police.

Move 4.01 onto the main trunk.