GitLab

Patch mostly from Lennart Augustsson in #803, with additions to
Task.c by me.

A simple interface for generating trace messages with timestamps and
thread IDs attached to them.  Most debugging output goes through this
interface now, so it is straightforward to get timestamped debugging
traces with +RTS -vt.  Also, we plan to use this to generate
parallelism profiles from the trace output.

    
In the threaded RTS on *nix platforms:
    
 - we now start the IO manager thread eagerly at startup time
   (previously was started on demand).

 - we now ask the IO manager thread to stop at shutdown
    
 - In Timer.c:handle_tick, if it looks like we might be in a
   deadlock, instead of calling prodOneCapability() which was known to be
   wrong, we now send a byte down the IO manager's pipe to wake it up.
  
This also avoids a case of double-acquisition of a mutex, which
happened if prodOneCapability() was called while the current thread
was holding a mutex.

Most of the other users of the fptools build system have migrated to
Cabal, and with the move to darcs we can now flatten the source tree
without losing history, so here goes.

The main change is that the ghc/ subdir is gone, and most of what it
contained is now at the top level.  The build system now makes no
pretense at being multi-project, it is just the GHC build system.

No doubt this will break many things, and there will be a period of
instability while we fix the dependencies.  A straightforward build
should work, but I haven't yet fixed binary/source distributions.
Changes to the Building Guide will follow, too.

The recent patch to free memory in hs_exit() on Win32 unfortunately broke
profiling, because it freed the memory slightly too early.

Now, the threaded RTS also includes SMP support.  The -smp flag is a
synonym for -threaded.  The performance implications of this are small
to negligible, and it results in a code cleanup and reduces the number
of combinations we have to test.

Re-use temporary storage in the STM implementation

Big re-hash of the threaded/SMP runtime

This is a significant reworking of the threaded and SMP parts of
the runtime.  There are two overall goals here:

  - To push down the scheduler lock, reducing contention and allowing
    more parts of the system to run without locks.  In particular,
    the scheduler does not require a lock any more in the common case.

  - To improve affinity, so that running Haskell threads stick to the
    same OS threads as much as possible.

At this point we have the basic structure working, but there are some
pieces missing.  I believe it's reasonably stable - the important
parts of the testsuite pass in all the (normal,threaded,SMP) ways.

In more detail:

  - Each capability now has a run queue, instead of one global run
    queue.  The Capability and Task APIs have been completely
    rewritten; see Capability.h and Task.h for the details.

  - Each capability has its own pool of worker Tasks.  Hence, Haskell
    threads on a Capability's run queue will run on the same worker
    Task(s).  As long as the OS is doing something reasonable, this
    should mean they usually stick to the same CPU.  Another way to
    look at this is that we're assuming each Capability is associated
    with a fixed CPU.

  - What used to be StgMainThread is now part of the Task structure.
    Every OS thread in the runtime has an associated Task, and it
    can ask for its current Task at any time with myTask().

  - removed RTS_SUPPORTS_THREADS symbol, use THREADED_RTS instead
    (it is now defined for SMP too).

  - The RtsAPI has had to change; we must explicitly pass a Capability
    around now.  The previous interface assumed some global state.
    SchedAPI has also changed a lot.

  - The OSThreads API now supports thread-local storage, used to
    implement myTask(), although it could be done more efficiently
    using gcc's __thread extension when available.

  - I've moved some POSIX-specific stuff into the posix subdirectory,
    moving in the direction of separating out platform-specific
    implementations.

  - lots of lock-debugging and assertions in the runtime.  In particular,
    when DEBUG is on, we catch multiple ACQUIRE_LOCK()s, and there is
    also an ASSERT_LOCK_HELD() call.

What's missing so far:

  - I have almost certainly broken the Win32 build, will fix soon.

  - any kind of thread migration or load balancing.  This is high up
    the agenda, though.

  - various performance tweaks to do

  - throwTo and forkProcess still do not work in SMP mode

Add comment

Initialise foreign exports from GNU C __attribute__((constructor)) functions
in the stub C file, rather than from __stginit_ functions.
For non-profiling ways, leave out __stginit_ alltogether.

Some multi-processor hackery, including

  - Don't hang blocked threads off BLACKHOLEs any more, instead keep
    them all on a separate queue which is checked periodically for
    threads to wake up.

    This is good because (a) we don't have to worry about locking the
    closure in SMP mode when we want to block on it, and (b) it means
    the standard update code doesn't need to wake up any threads or
    check for a BLACKHOLE_BQ, simplifying the update code.

    The downside is that if there are lots of threads blocked on
    BLACKHOLEs, we might have to do a lot of repeated list traversal.
    We don't expect this to be common, though.  conc023 goes slower
    with this change, but we expect most programs to benefit from the
    shorter update code.

  - Fixing up the Capability code to handle multiple capabilities (SMP
    mode), and related changes to get the SMP mode at least building.

Remove locale setting, it is no longer required

Make setProgArgv() set the RTS's idea of the prog name too.

Rationalise the BUILD,HOST,TARGET defines.

Recall that:

  - build is the platform we're building on
  - host is the platform we're running on
  - target is the platform we're generating code for

The change is that now we take these definitions as applying from the
point of view of the particular source code being built, rather than
the point of view of the whole build tree.

For example, in RTS and library code, we were previously testing the
TARGET platform.  But under the new rule, the platform on which this
code is going to run is the HOST platform.  TARGET only makes sense in
the compiler sources.

In practical terms, this means that the values of BUILD, HOST & TARGET
may vary depending on which part of the build tree we are in.

Actual changes:

 - new file: includes/ghcplatform.h contains platform defines for
   the RTS and library code.

 - new file: includes/ghcautoconf.h contains the autoconf settings
   only (HAVE_BLAH).  This is so that we can get hold of these
   settings independently of the platform defines when necessary
   (eg. in GHC).

 - ghcconfig.h now #includes both ghcplatform.h and ghcautoconf.h.

 - MachRegs.h, which is included into both the compiler and the RTS,
   now has to cope with the fact that it might need to test either
   _TARGET_ or _HOST_ depending on the context.

 - the compiler's Makefile now generates
     stage{1,2,3}/ghc_boot_platform.h
   which contains platform defines for the compiler.  These differ
   depending on the stage, of course: in stage2, the HOST is the
   TARGET of stage1.  This was wrong before.

 - The compiler doesn't get platform info from Config.hs any more.
   Previously it did (sometimes), but unless we want to generate
   a new Config.hs for each stage we can't do this.

 - GHC now helpfully defines *_{BUILD,HOST}_{OS,ARCH} automatically
   in CPP'd Haskell source.

 - ghcplatform.h defines *_TARGET_* for backwards compatibility
   (ghcplatform.h is included by ghcconfig.h, which is included by
   config.h, so code which still #includes config.h will get the TARGET
   settings as before).

 - The Users's Guide is updated to mention *_HOST_* rather than
   *_TARGET_*.

 - coding-style.html in the commentary now contains a section on
   platform defines.  There are further doc updates to come.

Thanks to Wolfgang Thaller for pointing me in the right direction.

Don't try to run finalizers at program exit.  This turned out to be
hard if not impossible to do in general, so now we don't attempt it at
all.

The Main.main wrapper, previously called runIO and now called
runMainIO, flushes stdout and stderr before exiting.  This should
catch most cases where programs rely on Handles being flushed at
program exit, but note that now if you simply drop a Handle in your
program, there's no guarantee it'll be flushed on exit.  If the
punters complain enough, I suppose we could implement a global
Handle table and flush them all at exit... I'd rather not do this if
possible, though.  Better to teach people to close their Handles
properly.

Support for atomic memory transactions and associated regression tests conc041-048

Limit ThreadLabel API use to DEBUG only.

Warning police. Why is x86_init_fpu disabled, BTW?

Removed the annoying "Id" CVS keywords, they're a real PITA when it
comes to merging...

Cleanup: all (well, most) messages from the RTS now go through the
functions in RtsUtils: barf(), debugBelch() and errorBelch().  The
latter two were previously called belch() and prog_belch()
respectively.  See the comments for the right usage of these message
functions.

One reason for doing this is so that we can avoid spurious uses of
stdout/stderr by Haskell apps on platforms where we shouldn't be using
them (eg. non-console apps on Windows).

Perform initialisation of adjustor thunk layer at startup.

Merge backend-hacking-branch onto HEAD.  Yay!

Restrict the setlocale() to LC_CTYPE (was LC_ALL), because printf and friends
are LC_NUMERIC-sensitive. This fixes the generation of heap profiles for e.g.
de_DE locales where a comma separates the fractional part of a number, not a
dot. (There was a similar bug in Hugs regarding atof, BTW, which was fixed last
December.)

Merge to STABLE
(I leave this to the Grand Marshal of Releases, Mr. SimonM, because I'm not 100%
sure of all the consequences.)

hs_exit(): call exitScheduler() *after* finalizeWeakPointersNow(),
because the latter needs to create/run threads.

Warning police

To fix dynamic linking issues with Hugs, access saved_termios via
getter/setter.

If we change the terminal settings as a result of hSetBuffering or
hSetEcho, then restore them again in hs_exit().  This is just good
citizenship on Unixy platforms.

We *don't* just automatically save the terminal settings and restore
them at the end, because that would prevent implementing stty-like
programs in Haskell.  This scheme is a compromise that hopefully DTRT
in most cases.

setupRtsFlags(): don't overwrite argv[0] with its basename:
   - argv[] may not point to writeable memory
   - System.Environment.getProgName strips off the 'dirname' portion
     anyway.
   - Not possible to get at the untransformed argv[0] from
     Haskell code, should such a need arise.

Uses of prog_argv[0] within the RTS has now been replaced with prog_name,
which is the basename of prog_argv[0].

setlocale(LC_ALL,""), to enable the locale support in libc.

Tidy up code that supports user/Haskell signal handlers.

Signals.h now defines RTS_USER_SIGNALS when this is supported,
which is then used elsewhere.

Clean up code&interfaces that deals with timers and asynchrony:

- Timer.{c,h} now defines the platform-independent interface
  to the timing services needed by the RTS. Itimer.{c,h} +
  win32/Ticker.{c,h} defines the OS-specific services that
  creates/destroys a timer.
- For win32 plats, drop the long-standing use of the 'multimedia'
  API timers and implement the ticking service ourselves. Simpler
  and more flexible.
- Select.c is now solely for platforms that use select() to handle
  non-blocking I/O & thread delays. win32/AwaitEvent.c provides
  the same API on the Win32 side.
- support threadDelay on win32 platforms via worker threads.

Not yet compiled up on non-win32 platforms; will do once checked in.

Asynchronous / non-blocking I/O for Win32 platforms.

This commit introduces a Concurrent Haskell friendly view of I/O on
Win32 platforms. Through the use of a pool of worker Win32 threads, CH
threads may issue asynchronous I/O requests without blocking the
progress of other CH threads. The issuing CH thread is blocked until
the request has been serviced though.

GHC.Conc exports the primops that take care of issuing the
asynchronous I/O requests, which the IO implementation now takes
advantage of. By default, all Handles are non-blocking/asynchronous,
but should performance become an issue, having a per-Handle flag for
turning off non-blocking could easily be imagined&introduced.

[Incidentally, this thread pool-based implementation could easily be
extended to also allow Haskell code to delegate the execution of
arbitrary pieces of (potentially blocking) external code to another OS
thread. Given how relatively gnarly the locking story has turned out
to be with the 'threaded' RTS, that may not be such a bad idea.]

Quick fix for profiling after I broke it in rev. 1.68.

The profiling system needs some initialisation after module
registration (hs_add_root()).  Since module registration now happens
later, the profiling system was getting initialised too early before
any cost centers had been registered.  This is a quick fix; we still
can't handle multiple hs_add_root()s with profiling.

- re-instate setProgArgv, it is used in System.Environment (bah, could
  have sworn I grepped for it and found nothing...)

- Remove init_stack symbol from the Linker's symbol table; this is
  now static.

Flesh out support for hs_init() and hs_exit() according to the latest
FFI spec.

For GHC, I also added:

  hs_add_root( void (*fn)(void) );

which is used to specify the root module.  This *must* be called prior
to invoking any Haskell functions.

The previous way of doing things still works:

  startupHaskell( argc, argv, root );

but the right way to do this is now

  hs_init( &argc, &argv );
  hs_add_root( root );

It is possible to invoke hs_add_root() multiple times with different
roots.

- setProgArgv() has been removed; it was unused and looks like it was
  there to support STG Hugs.

Merge the eval-apply-branch on to the HEAD
------------------------------------------

This is a change to GHC's evaluation model in order to ultimately make
GHC more portable and to reduce complexity in some areas.

At some point we'll update the commentary to describe the new state of
the RTS.  Pending that, the highlights of this change are:

  - No more Su.  The Su register is gone, update frames are one
    word smaller.

  - Slow-entry points and arg checks are gone.  Unknown function calls
    are handled by automatically-generated RTS entry points (AutoApply.hc,
    generated by the program in utils/genapply).

  - The stack layout is stricter: there are no "pending arguments" on
    the stack any more, the stack is always strictly a sequence of
    stack frames.

    This means that there's no need for LOOKS_LIKE_GHC_INFO() or
    LOOKS_LIKE_STATIC_CLOSURE() any more, and GHC doesn't need to know
    how to find the boundary between the text and data segments (BIG WIN!).

  - A couple of nasty hacks in the mangler caused by the neet to
    identify closure ptrs vs. info tables have gone away.

  - Info tables are a bit more complicated.  See InfoTables.h for the
    details.

  - As a side effect, GHCi can now deal with polymorphic seq.  Some bugs
    in GHCi which affected primitives and unboxed tuples are now
    fixed.

  - Binary sizes are reduced by about 7% on x86.  Performance is roughly
    similar, some programs get faster while some get slower.  I've seen
    GHCi perform worse on some examples, but haven't investigated
    further yet (GHCi performance *should* be about the same or better
    in theory).

  - Internally the code generator is rather better organised.  I've moved
    info-table generation from the NCG into the main codeGen where it is
    shared with the C back-end; info tables are now emitted as arrays
    of words in both back-ends.  The NCG is one step closer to being able
    to support profiling.

This has all been fairly thoroughly tested, but no doubt I've messed
up the commit in some way.

Global and common variable sweep: staticize many variables that don't
need to be globally visible.

Remove most #includes of system headers from Stg.h, and instead
#include any required headers directly in each RTS source file.

The idea is to (a) reduce namespace pollution from system headers that
we don't need, (c) be clearer about dependencies on system things in
the RTS, and (c) improve via-C compilation times (maybe).

In practice though, HsBase.h #includes everything anyway, so the
difference from the point of view of .hc source is minimal.  However,
this makes it easier to move to zero-includes if we wanted to (see
discussion on the FFI list; I'm still not sure that's possible but
at least this is a step in the right direction).

- Make TSO "stable" again: The thread label was changing the size of the
   TSO if you were building a debugging-RTS, leading to binary
   incompatibility. Now we map TSOs to strings using Hash.c.

- API change for labelThread: Label arbitrary threads.