Remove bogus code: the mutable list does not need to be rebuilt during
compaction.  It just needs to be updated, which is already happening.

(fixes crashes in compacting collection; I think a different fix is
required for STABLE, though)

Some multi-processor hackery, including

  - Don't hang blocked threads off BLACKHOLEs any more, instead keep
    them all on a separate queue which is checked periodically for
    threads to wake up.

    This is good because (a) we don't have to worry about locking the
    closure in SMP mode when we want to block on it, and (b) it means
    the standard update code doesn't need to wake up any threads or
    check for a BLACKHOLE_BQ, simplifying the update code.

    The downside is that if there are lots of threads blocked on
    BLACKHOLEs, we might have to do a lot of repeated list traversal.
    We don't expect this to be common, though.  conc023 goes slower
    with this change, but we expect most programs to benefit from the
    shorter update code.

  - Fixing up the Capability code to handle multiple capabilities (SMP
    mode), and related changes to get the SMP mode at least building.

Main x86_64 hacking: we have a problem on this arch where binutils
can't generate 64-bit relative relocations (R_X86_64_PC64), which many
of our info-table fields are.  So far we've been hacking around it by
putting everything in the text section, but I've decided to adopt
another approach: we'll use explicit 32-bit offset fields on this
platform instead.  This is safe in the default "small" memory model
where all symbols are guaranteed to be in the lower 2Gb of the address
space.

NCG changes coming; mangler changes are probably required too.

Catch up with mutable list changes

GC changes: instead of threading old-generation mutable lists
through objects in the heap, keep it in a separate flat array.

This has some advantages:

  - the IND_OLDGEN object is now only 2 words, so the minimum
    size of a THUNK is now 2 words instead of 3.  This saves
    some amount of allocation (about 2% on average according to
    my measurements), and is more friendly to the cache by
    squashing objects together more.

  - keeping the mutable list separate from the IND object
    will be necessary for our multiprocessor implementation.

  - removing the mut_link field makes the layout of some objects
    more uniform, leading to less complexity and special cases.

  - I also unified the two mutable lists (mut_once_list and mut_list)
    into a single mutable list, which lead to more simplifications
    in the GC.

Fix build error during unregistered builds, remove warning building GCCompact

Support for atomic memory transactions and associated regression tests conc041-048

Position Independent Code and Dynamic Linking Support, Part 1

This commit allows generation of position independent code (PIC) that fully supports dynamic linking on Mac OS X and PowerPC Linux.
Other platforms are not yet supported, and there is no support for actually linking or using dynamic libraries - so if you use the -fPIC or -dynamic code generation flags, you have to type your (platform-specific) linker command lines yourself.


nativeGen/PositionIndependentCode.hs:
New file. Look here for some more comments on how this works.

cmm/CLabel.hs:
Add support for DynamicLinkerLabels and PIC base labels - for use inside the NCG.
needsCDecl: Case alternative labels now need C decls, see the codeGen/CgInfoTbls.hs below for details

cmm/Cmm.hs:
Add CmmPicBaseReg (used in NCG),
and CmmLabelDiffOff (used in NCG and for offsets in info tables)

cmm/CmmParse.y:
support offsets in info tables

cmm/PprC.hs:
support CmmLabelDiffOff
Case alternative labels now need C decls (see the codeGen/CgInfoTbls.hs for details), so we need to pprDataExterns for info tables.

cmm/PprCmm.hs:
support CmmLabelDiffOff

codeGen/CgInfoTbls.hs:
no longer store absolute addresses in info tables, instead, we store offsets.
Also, for vectored return points, emit the alternatives _after_ the vector table. This is to work around a limitation in Apple's as, which refuses to handle label differences where one label is at the end of a section. Emitting alternatives after vector info tables makes sure this never happens in GHC generated code. Case alternatives now require prototypes in hc code, though (see changes in PprC.hs, CLabel.hs).

main/CmdLineOpts.lhs:
Add a new option, -fPIC.

main/DriverFlags.hs:
Pass the correct options for PIC to gcc, depending on the platform. Only for powerpc for now.

nativeGen/AsmCodeGen.hs:
Many changes...
Mac OS X-specific management of import stubs is no longer, it's now part of a general mechanism to handle such things for all platforms that need it (Darwin [both ppc and x86], Linux on ppc, and some platforms we don't support).
Move cmmToCmm into its own monad which can accumulate a list of imported symbols. Make it call cmmMakeDynamicReference at the right places.

nativeGen/MachCodeGen.hs:
nativeGen/MachInstrs.hs:
nativeGen/MachRegs.lhs:
nativeGen/PprMach.hs:
nativeGen/RegAllocInfo.hs:
Too many changes to enumerate here, PowerPC specific.

nativeGen/NCGMonad.hs:
NatM still tracks imported symbols, as more labels can be created during code generation (float literals, jump tables; on some platforms all data access has to go through the dynamic linking mechanism).

driver/mangler/ghc-asm.lprl:
Mangle absolute addresses in info tables to offsets.
Correctly pass through GCC-generated PIC for Mac OS X and powerpc linux.

includes/Cmm.h:
includes/InfoTables.h:
includes/Storage.h:
includes/mkDerivedConstants.c:
rts/GC.c:
rts/GCCompact.c:
rts/HeapStackCheck.cmm:
rts/Printer.c:
rts/RetainerProfile.c:
rts/Sanity.c:
Adapt to the fact that info tables now contain offsets.

rts/Linker.c:
Mac-specific: change machoInitSymbolsWithoutUnderscore to support PIC.

Removed the annoying "Id" CVS keywords, they're a real PITA when it
comes to merging...

Cleanup: all (well, most) messages from the RTS now go through the
functions in RtsUtils: barf(), debugBelch() and errorBelch().  The
latter two were previously called belch() and prog_belch()
respectively.  See the comments for the right usage of these message
functions.

One reason for doing this is so that we can avoid spurious uses of
stdout/stderr by Haskell apps on platforms where we shouldn't be using
them (eg. non-console apps on Windows).

Merge backend-hacking-branch onto HEAD.  Yay!

Tweaks to have RTS (C) sources compile with MSVC. Apart from wibbles
related to the handling of 'inline', changed Schedule.h:POP_RUN_QUEUE()
not to use expression-level statement blocks.

thread_stack(RET_BCO): Grab the pointer to the BCO *before* threading
it.  Fixes crashes when using compacting GC with GHCi.

I noticed while debugging this that compacting GC is horrendously slow
now, which might have something to do with the new eval/apply PAP
layouts.  That's something to investigate later.

Fix an obscure bug: the most general kind of heap check,
HEAP_CHECK_GEN(), is supposed to save the contents of *every* register
known to the STG machine (used in cases where we either can't figure
out which ones are live, or doing so would be too much hassle).  The
problem is that it wasn't saving the L1 register.

A slight complication arose in that saving the L1 register pushed the
size of the frame over the 16 words allowed for the size of the bitmap
stored in the frame, so I changed the layout of the frame a bit.
Describing all the registers using a single bitmap is overkill when
only 8 of them can actually be pointers, so now the bitmap is only 8
bits long and we always skip over a fixed number of non-ptr words to
account for all the non-ptr regs.  This is all described in StgMacros.h.

warning elimination

Fix some bugs in compacting GC.

Bug 1: When threading the fields of an AP or PAP, we were grabbing the
info table of the function without unthreading it first.

Bug 2: eval_thunk_selector() might accidentally find itself in
to-space when going through indirections in a compacted generation.
We must check for this case and bale out if necessary.

Bug 3: This is somewhat more nasty.  When we have an AP or PAP that
points to a BCO, the layout info for the AP/PAP is in the BCO's
instruction array, which is two objects deep from the AP/PAP itself.
The trouble is, during compacting GC, we can only safely look one
object deep from the current object, because pointers from objects any
deeper might have been already updated to point to their final
destinations.

The solution is to put the arity and bitmap info for a BCO into the
BCO object itself.  This means BCOs become variable-length, which is a
slight annoyance, but it also means that looking up the arity/bitmap
is quicker.  There is a slight reduction in complexity in the byte
code generator due to not having to stuff the bitmap at the front of
the instruction stream.

Merge the eval-apply-branch on to the HEAD
------------------------------------------

This is a change to GHC's evaluation model in order to ultimately make
GHC more portable and to reduce complexity in some areas.

At some point we'll update the commentary to describe the new state of
the RTS.  Pending that, the highlights of this change are:

  - No more Su.  The Su register is gone, update frames are one
    word smaller.

  - Slow-entry points and arg checks are gone.  Unknown function calls
    are handled by automatically-generated RTS entry points (AutoApply.hc,
    generated by the program in utils/genapply).

  - The stack layout is stricter: there are no "pending arguments" on
    the stack any more, the stack is always strictly a sequence of
    stack frames.

    This means that there's no need for LOOKS_LIKE_GHC_INFO() or
    LOOKS_LIKE_STATIC_CLOSURE() any more, and GHC doesn't need to know
    how to find the boundary between the text and data segments (BIG WIN!).

  - A couple of nasty hacks in the mangler caused by the neet to
    identify closure ptrs vs. info tables have gone away.

  - Info tables are a bit more complicated.  See InfoTables.h for the
    details.

  - As a side effect, GHCi can now deal with polymorphic seq.  Some bugs
    in GHCi which affected primitives and unboxed tuples are now
    fixed.

  - Binary sizes are reduced by about 7% on x86.  Performance is roughly
    similar, some programs get faster while some get slower.  I've seen
    GHCi perform worse on some examples, but haven't investigated
    further yet (GHCi performance *should* be about the same or better
    in theory).

  - Internally the code generator is rather better organised.  I've moved
    info-table generation from the NCG into the main codeGen where it is
    shared with the C back-end; info tables are now emitted as arrays
    of words in both back-ends.  The NCG is one step closer to being able
    to support profiling.

This has all been fairly thoroughly tested, but no doubt I've messed
up the commit in some way.

Main threads are now not kept alive artificially, so it is possible
for a main thread to be sent the BlockedOnDeadMVar exception.  Main
threads are no longer GC roots.

This involved cleaning up the weak pointer processing somewhat, and
separating the processing of real weak pointers from the processing of
the all_threads list (which can be thought of as "weaker pointers": a
finalizer can keep a blocked thread alive, but not vice-versa).  The
new story is described in a detailed comment in GC.c.

One interesting consequence is that it's much harder to get a Deadlock
exception now - many deadlock situations involving main threads will
turn into BlockedOnDeadMVar situations instead.  For example, if there
are a group of threads in a circular deadlock, then they will all be
sent BlockedOnDeadMVar simultaneously, whereas before if one of them
was the main thread it would be sent Deadlock.  It's really hard to
get Deadlock now - you have to somehow keep an MVar independently
reachable, eg. by using a StablePtr.

eliminate warnings

merge from stable revs:
  1.121.4.1 +7 -6      fptools/ghc/rts/GC.c
  1.9.4.1   +4 -1      fptools/ghc/rts/GCCompact.c
  1.17.4.1  +4 -3      fptools/ghc/rts/StoragePriv.h

  SimonM's fixes to deal with GHCi and CAFs properly in the compacting
  collector.

Change the story about POSIX headers in C compilation.

Until now, all C code in the RTS and library cbits has by default been
compiled with settings for POSIXness enabled, that is:
   #define _POSIX_SOURCE   1
   #define _POSIX_C_SOURCE 199309L
   #define _ISOC9X_SOURCE
If you wanted to negate this, you'd have to define NON_POSIX_SOURCE
before including headers.

This scheme has some bad effects:

* It means that ccall-unfoldings exported via interfaces from a
  module compiled with -DNON_POSIX_SOURCE may not compile when
  imported into a module which does not -DNON_POSIX_SOURCE.

* It overlaps with the feature tests we do with autoconf.

* It seems to have caused borkage in the Solaris builds for some
  considerable period of time.

The New Way is:

* The default changes to not-being-in-Posix mode.

* If you want to force a C file into Posix mode, #include as
  the **first** include the new file ghc/includes/PosixSource.h.
  Most of the RTS C sources have this include now.

* NON_POSIX_SOURCE is almost totally expunged.  Unfortunately
  we have to retain some vestiges of it in ghc/compiler so that
  modules compiled via C on Solaris using older compilers don't
  break.

Thread the mut_once_list properly, once, before we start compacting.
Previously we tried to do it during the first threading phase, but we
missing some pointers (namely IND_STATIC objects which are only
reachable via the mut_once list).

oops, forgot to thread the link and global_link fields of a large
TSO.  Fixes a potential crash with the compacting collector.

Some 64-bit fixes for the new GC code.

Small perf. improvement: use the next bit in the bitmap to indicate
whether the current object needs to be pushed into the next block
during compaction.  This saves having to do the double traversal of
the threaded info pointer list during the second pass in most cases.

Specialise thread_static() a little bit: improves performance for
threading the static object list.

More tweaks.  Getting usable now.

Make compacting GC faster, stage 1.

A cunning trick from Simon P.J. to avoid needing to call the
potentially expensive LOOKS_LIKE_GHC_INFO() a lot.

Add a compacting garbage collector.

It isn't enabled by default, as there are still a couple of problems:
there's a fallback case I haven't implemented yet which means it will
occasionally bomb out, and speed-wise it's quite a bit slower than the
copying collector (about 1.8x slower).

Until I can make it go faster, it'll only be useful when you're
actually running low on real memory.

'+RTS -c' to enable it.

Oh, and I cleaned up a few things in the RTS while I was there, and
fixed one or two possibly real bugs in the existing GC.