GitLab

There are still some fixes required to get the threaded RTS compilable
with the NCG, and apparently there are problems on 32-bit archs too.

as a precautionary measure.  It is definitely required for GC.c,
but it may well become necessary for other files in the future due to
our (mis-)use of the C "type system".

This patch makes throwTo work with -threaded, and also refactors large
parts of the concurrency support in the RTS to clean things up.  We
have some new files:

  RaiseAsync.{c,h}	asynchronous exception support
  Threads.{c,h}         general threading-related utils

Some of the contents of these new files used to be in Schedule.c,
which is smaller and cleaner as a result of the split.

Asynchronous exception support in the presence of multiple running
Haskell threads is rather tricky.  In fact, to my annoyance there are
still one or two bugs to track down, but the majority of the tests run
now.

Somewhere along the 6.5 branch, gcc started compiling the rts such that
it triggers the stack smash handler, causing stage2 to by kill'd
immediately. This turns off the stack protector, which will do for now.

Most of the other users of the fptools build system have migrated to
Cabal, and with the move to darcs we can now flatten the source tree
without losing history, so here goes.

The main change is that the ghc/ subdir is gone, and most of what it
contained is now at the top level.  The build system now makes no
pretense at being multi-project, it is just the GHC build system.

No doubt this will break many things, and there will be a period of
instability while we fix the dependencies.  A straightforward build
should work, but I haven't yet fixed binary/source distributions.
Changes to the Building Guide will follow, too.

Itimer.h has moved into posix/

Big re-hash of the threaded/SMP runtime

This is a significant reworking of the threaded and SMP parts of
the runtime.  There are two overall goals here:

  - To push down the scheduler lock, reducing contention and allowing
    more parts of the system to run without locks.  In particular,
    the scheduler does not require a lock any more in the common case.

  - To improve affinity, so that running Haskell threads stick to the
    same OS threads as much as possible.

At this point we have the basic structure working, but there are some
pieces missing.  I believe it's reasonably stable - the important
parts of the testsuite pass in all the (normal,threaded,SMP) ways.

In more detail:

  - Each capability now has a run queue, instead of one global run
    queue.  The Capability and Task APIs have been completely
    rewritten; see Capability.h and Task.h for the details.

  - Each capability has its own pool of worker Tasks.  Hence, Haskell
    threads on a Capability's run queue will run on the same worker
    Task(s).  As long as the OS is doing something reasonable, this
    should mean they usually stick to the same CPU.  Another way to
    look at this is that we're assuming each Capability is associated
    with a fixed CPU.

  - What used to be StgMainThread is now part of the Task structure.
    Every OS thread in the runtime has an associated Task, and it
    can ask for its current Task at any time with myTask().

  - removed RTS_SUPPORTS_THREADS symbol, use THREADED_RTS instead
    (it is now defined for SMP too).

  - The RtsAPI has had to change; we must explicitly pass a Capability
    around now.  The previous interface assumed some global state.
    SchedAPI has also changed a lot.

  - The OSThreads API now supports thread-local storage, used to
    implement myTask(), although it could be done more efficiently
    using gcc's __thread extension when available.

  - I've moved some POSIX-specific stuff into the posix subdirectory,
    moving in the direction of separating out platform-specific
    implementations.

  - lots of lock-debugging and assertions in the runtime.  In particular,
    when DEBUG is on, we catch multiple ACQUIRE_LOCK()s, and there is
    also an ASSERT_LOCK_HELD() call.

What's missing so far:

  - I have almost certainly broken the Win32 build, will fix soon.

  - any kind of thread migration or load balancing.  This is high up
    the agenda, though.

  - various performance tweaks to do

  - throwTo and forkProcess still do not work in SMP mode

Darwin/x86:

Maintain 16-byte alignment for foreign import wrapper calls.

comments about -fno-strict-aliasing

gcc has gone to town on the garbage collector and killed it: use
-optc-fno-strict-aliasing as a workaround for now.

Compile GC.c with -O3 (unrolls some important loops)

Two SMP-related changes:

  - New storage manager interface:

    bdescr *allocateLocal(StgRegTable *reg, nat words)

    which allocates from the current thread's nursery (being careful
    not to clash with the heap pointer).  It can do this without
    taking any locks; the lock only has to be taken if a block needs
    to be allocated.  allocateLocal() is now used instead of allocate()
    in a few PrimOps.

    This removes locks from most Integer operations, cutting down
    the overhead for SMP a bit more.

    To make this work, we have to be able to grab the current thread's
    Capability out of thin air (i.e. when called from GMP), so the
    Capability subsystem needs to keep a hash from thread IDs to
    Capabilities.

  - Small MVar optimisation: instead of taking the global
    storage-manager lock, do our own locking of MVars with a bit of
    inline assembly (x86 only for now).

Hold the sm_mutex around access to the mutable list.

The SMP RTS now seems quite stable, I've run my simple test program
with 64 threads without crashes.

- Now that labels are always prefixed with '&' in .hc code, we have to
  fix some sloppiness in the RTS .cmm code.  Fortunately it's not too
  painful.

- SMP: acquire/release the storage manager lock around
  atomicModifyMutVar#.  This is a hack: atomicModifyMutVar# isn't
  atomic under SMP otherwise, but the SM lock is a large sledgehammer.
  I think I'll apply the sledgehammer to the MVar primitives too, for
  the time being.

Mac OS X:
Kill HaskellSupport.framework.
Instead, look for GMP.framework (a framework-version of libgmp), else look
for a normal -lgmp as usual.
The other part of HaskellSupport.framework, dlcompat, is no longer needed
(as of Mac OS X 10.3, it's included in libSystem).
It's enough to just use the normal configure tests for -ldl.

MERGE TO STABLE

Optimisation: the update code only needs to check whether the closure
being updated is a BLACKHOLE_BQ if it has been blackholed; which is
only the case after lazy blackholing has taken place.

So to avoid this unnecessary cost, we have two kinds of update frame:
one which checks for BLACKHOLE_BQ and one that doesn't, and the lazy
blackholing algorithm converts the latter kind into the former as it
replaces thunks with blackholes.

Add AdjustorAsm.S to EXCLUDED_SRCS unless we're building for
powerpc/powerpc64.  Apparently Solaris ld chokes on the empty object
file otherwise.

Suppress .cmm suffix rules when $(BootingFromHc)

Squash gcc 3.4.x warnings.

Fix the unreg test in the rule for building AutoApply$(_way).cmm

Support for atomic memory transactions and associated regression tests conc041-048

Had enough of 'make boot' completely rebuilding gmp/ each time around.

Synched with latest GHC #include changes: Add '.' to the #include path manually.

all target: if building GMP, only descend into gmp/ when way=""

Windows fixes.

Merge backend-hacking-branch onto HEAD.  Yay!

Hack to generate correct dependencies for the RTS.  We now #define
PROFILING, THREADED_RTS and DEBUG when generating dependencies so that
we get (hopefully) a superset of all the deps.

Allow configuration of RTS ways by tweaking the GhcRTSWays variable.
Add default settings to config.mk.in, and add a comment to explain
which values make sense here.

Also, I disabled the debug_p way by default; it's of limited
usefulness to the average punter.

Fix a test

Experimental support for RTS-only "ways"

HEADS UP!  This changes the way that the threaded RTS is used, and
also the use of debugging RTSs:

  - We always build threaded and debugging variants of the RTS now.
    The --enable-threaded-rts configure option is ignored (and will
    be removed at some point).

  - New option:  -debug     enables the debugging RTS

  - New option:  -threaded  enables the threaded RTS.  When the threaded
    RTS is stable enough, we might make it the default.

The new options just cause a different variant of the RTS to be linked
in, and they cause one or two extra options to be enabled too.  The
implementation is via the usual ways machinery in the compiler, except
that these ways are labelled as RTS-only, and so don't require
rebuilding all the libraries too.

All of this means we can ship threaded and debugging RTSs with GHC, so
that users don't need to fetch and build a GHC source tree to use
them.

I'd like to get this functionality into 6.2.1 if possible, so please
test (I'm willing to stretch the definition of "interface change" to
accomodate this, since having a threaded RTS available without having
to build GHC will be a big win for the Visual Studio project).

Bound Threads
=============

Introduce a way to use foreign libraries that rely on thread local state
from multiple threads (mainly affects the threaded RTS).

See the file threads.tex in CVS at haskell-report/ffi/threads.tex
(not entirely finished yet) for a definition of this extension. A less formal
description is also found in the documentation of Control.Concurrent.

The changes mostly affect the THREADED_RTS (./configure --enable-threaded-rts),
except for saving & restoring errno on a per-TSO basis, which is also necessary
for the non-threaded RTS (a bugfix).

Detailed list of changes
------------------------

- errno is saved in the TSO object and restored when necessary:
ghc/includes/TSO.h, ghc/rts/Interpreter.c, ghc/rts/Schedule.c

- rts_mainLazyIO is no longer needed, main is no special case anymore
ghc/includes/RtsAPI.h, ghc/rts/RtsAPI.c, ghc/rts/Main.c, ghc/rts/Weak.c

- passCapability: a new function that releases the capability and "passes"
  it to a specific OS thread:
ghc/rts/Capability.h ghc/rts/Capability.c

- waitThread(), scheduleWaitThread() and schedule() get an optional
  Capability *initialCapability passed as an argument:
ghc/includes/SchedAPI.h, ghc/rts/Schedule.c, ghc/rts/RtsAPI.c

- Bound Thread scheduling (that's what this is all about):
ghc/rts/Schedule.h, ghc/rts/Schedule.c

- new Primop isCurrentThreadBound#:
ghc/compiler/prelude/primops.txt.pp, ghc/includes/PrimOps.h, ghc/rts/PrimOps.hc,
ghc/rts/Schedule.h, ghc/rts/Schedule.c

- a simple function, rtsSupportsBoundThreads, that returns true if THREADED_RTS
  is defined:
ghc/rts/Schedule.h, ghc/rts/Schedule.c

- a new implementation of forkProcess (the old implementation stays in place
  for the non-threaded case). Partially broken; works for the standard
  fork-and-exec case, but not for much else. A proper forkProcess is
  really next to impossible to implement:
ghc/rts/Schedule.c

- Library support for bound threads:
    Control.Concurrent.
      rtsSupportsBoundThreads, isCurrentThreadBound, forkOS,
      runInBoundThread, runInUnboundThread
libraries/base/Control/Concurrent.hs, libraries/base/Makefile,
libraries/base/include/HsBase.h, libraries/base/cbits/forkOS.c (new file)

Remove special mingw32 case for configuring GMP, since it appears to
be wrong.

More BootingFromHc fixes

- no need for hacky delayed setting of UseGhcForCc
- fix AutoApply.hc rule when BootingFromHc=YES

Make 'depend' treated GTK includes as missing when building with FrontPanel.

Support for interop'ing with .NET via FFI declarations along the
lines of what Hugs98.NET offers, see

 http://haskell.org/pipermail/cvs-hugs/2003-March/001723.html

for FFI decl details.

To enable, configure with --enable-dotnet + have a look
in ghc/rts/dotnet/Makefile for details of what tools are needed to
build the .NET interop layer (tools from VS.NET / Framework SDK.)

The commit doesn't include some library additions + wider-scale
testing is required before this extension can be regarded as available
for general use. 'foreign import dotnet' is currently only supported
by the C backend.

Clean AutoApply.hc too.

win32 tidyup: exclude Select.c, Itimer.c and Signals.c from SRCS