GitLab

Nothing from gmp is used in the rts anymore.

Since we introduced pointer tagging, we no longer always enter a
closure to evaluate it.  However, the biographical profiler relies on
closures being entered in order to mark them as "used", so we were
getting spurious amounts of data attributed to VOID.  It turns out
there are various places that need to be fixed, and I think at least
one of them was also wrong before pointer tagging (CgCon.cgReturnDataCon).

This merge does not turn on the new codegen (which only compiles
a select few programs at this point),
but it does introduce some changes to the old code generator.

The high bits:
1. The Rep Swamp patch is finally here.
   The highlight is that the representation of types at the
   machine level has changed.
   Consequently, this patch contains updates across several back ends.
2. The new Stg -> Cmm path is here, although it appears to have a
   fair number of bugs lurking.
3. Many improvements along the CmmCPSZ path, including:
   o stack layout
   o some code for infotables, half of which is right and half wrong
   o proc-point splitting

Fixes a long-standing bug that could in some cases cause sub-optimal
scheduling behaviour.

For some reason this causes build failures for me in my 32-bit chroot,

For some reason the C-- version of recordMutable wasn't verifying that
the object was in an old generation before attempting to add it to the
mutable list, and this broke maessen_hashtab.  This version of
recordMutable is only used in unsafeThaw#.

  
This patch implements pointer tagging as per our ICFP'07 paper "Faster
laziness using dynamic pointer tagging".  It improves performance by
10-15% for most workloads, including GHC itself.

The original patches were by Alexey Rodriguez Yakushev
<mrchebas@gmail.com>, with additions and improvements by me.  I've
re-recorded the development as a single patch.

The basic idea is this: we use the low 2 bits of a pointer to a heap
object (3 bits on a 64-bit architecture) to encode some information
about the object pointed to.  For a constructor, we encode the "tag"
of the constructor (e.g. True vs. False), for a function closure its
arity.  This enables some decisions to be made without dereferencing
the pointer, which speeds up some common operations.  In particular it
enables us to avoid costly indirect jumps in many cases.

More information in the commentary:

http://hackage.haskell.org/trac/ghc/wiki/Commentary/Rts/HaskellExecution/PointerTagging

We needed to turn some inline C functions and C macros into either
real C functions or C-- macros.

The following changes restore ticky-ticky profiling to functionality
from its formerly bit-rotted state. Sort of. (It got bit-rotted as part
of the switch to the C-- back-end.)

The way that ticky-ticky is supposed to work is documented in Section 5.7
of the GHC manual (though the manual doesn't mention that it hasn't worked
since sometime around 6.0, alas). Changes from this are as follows (which
I'll document on the wiki):

* In the past, you had to build all of the libraries with way=t in order to
use ticky-ticky, because it entailed a different closure layout. No longer.
You still need to do make way=t in rts/ in order to build the ticky RTS,
but you should now be able to mix ticky and non-ticky modules.

* Some of the counters that worked in the past aren't implemented yet.
I was originally just trying to get entry counts to work, so those should
be correct. The list of counters was never documented in the first place,
so I hope it's not too much of a disaster that some don't appear anymore.
Someday, someone (perhaps me) should document all the counters and what 
they do. For now, all of the counters are either accurate (or at least as
accurate as they always were), zero, or missing from the ticky profiling
report altogether.

This hasn't been particularly well-tested, but these changes shouldn't
affect anything except when compiling with -fticky-ticky (famous last
words...)

Implementation details:

I got rid of StgTicky.h, which in the past had the macros and declarations 
for all of the ticky counters. Now, those macros are defined in Cmm.h.
StgTicky.h was still there for inclusion in C code. Now, any remaining C
code simply cannot call the ticky macros -- or rather, they do call those
macros, but from the perspective of C code, they're defined as no-ops. 
(This shouldn't be too big a problem.)

I added a new file TickyCounter.h that has all the declarations for ticky
counters, as well as dummy macros for use in C code. Someday, these 
declarations should really be automatically generated, since they need
to be kept consistent with the macros defined in Cmm.h.

Other changes include getting rid of the header that was getting added to
closures before, and getting rid of various code having to do with eager
blackholing and permanent indirections (the changes under compiler/ 
and rts/Updates.*).

Fixed version of an old patch by Simon Marlow. His description read:
 Also, now an arbitrarily short context switch interval may now be
 specified, as we increase the RTS ticker's resolution to match the
 requested context switch interval.  This also applies to +RTS -i (heap
 profiling) and +RTS -I (the idle GC timer).  +RTS -V is actually only
 required for increasing the resolution of the profile timer.

HpAlloc was not being set when returning to the scheduler via MAYBE_GC(),
which at the least was just wrong (the scheduler might allocate a large
block more than once), and at worst could lead to crashes if HpAlloc contains
garbage.

Fixes at least one threaded2 test on Windows.

So that we can build the RTS with the NCG.

Most of the other users of the fptools build system have migrated to
Cabal, and with the move to darcs we can now flatten the source tree
without losing history, so here goes.

The main change is that the ghc/ subdir is gone, and most of what it
contained is now at the top level.  The build system now makes no
pretense at being multi-project, it is just the GHC build system.

No doubt this will break many things, and there will be a period of
instability while we fix the dependencies.  A straightforward build
should work, but I haven't yet fixed binary/source distributions.
Changes to the Building Guide will follow, too.

This fixes another instance of a subtle SMP bug (see patch "really
nasty bug in SMP").

We had to bite the bullet here and add an extra word to every thunk,
to enable running ordinary libraries on SMP.  Otherwise, we would have
needed to ship an extra set of libraries with GHC 6.6 in addition to
the two sets we already ship (normal + profiled), and all Cabal
packages would have to be compiled for SMP too.  We decided it best
just to take the hit now, making SMP easily accessible to everyone in
GHC 6.6.

Incedentally, although this increases allocation by around 12% on
average, the performance hit is around 5%, and much less if your inner
loop doesn't use any laziness.

MAYBE_GC: we should check alloc_blocks in addition to CurrentNursery,
since some allocateLocal calls don't allocate from the nursery.

Big re-hash of the threaded/SMP runtime

This is a significant reworking of the threaded and SMP parts of
the runtime.  There are two overall goals here:

  - To push down the scheduler lock, reducing contention and allowing
    more parts of the system to run without locks.  In particular,
    the scheduler does not require a lock any more in the common case.

  - To improve affinity, so that running Haskell threads stick to the
    same OS threads as much as possible.

At this point we have the basic structure working, but there are some
pieces missing.  I believe it's reasonably stable - the important
parts of the testsuite pass in all the (normal,threaded,SMP) ways.

In more detail:

  - Each capability now has a run queue, instead of one global run
    queue.  The Capability and Task APIs have been completely
    rewritten; see Capability.h and Task.h for the details.

  - Each capability has its own pool of worker Tasks.  Hence, Haskell
    threads on a Capability's run queue will run on the same worker
    Task(s).  As long as the OS is doing something reasonable, this
    should mean they usually stick to the same CPU.  Another way to
    look at this is that we're assuming each Capability is associated
    with a fixed CPU.

  - What used to be StgMainThread is now part of the Task structure.
    Every OS thread in the runtime has an associated Task, and it
    can ask for its current Task at any time with myTask().

  - removed RTS_SUPPORTS_THREADS symbol, use THREADED_RTS instead
    (it is now defined for SMP too).

  - The RtsAPI has had to change; we must explicitly pass a Capability
    around now.  The previous interface assumed some global state.
    SchedAPI has also changed a lot.

  - The OSThreads API now supports thread-local storage, used to
    implement myTask(), although it could be done more efficiently
    using gcc's __thread extension when available.

  - I've moved some POSIX-specific stuff into the posix subdirectory,
    moving in the direction of separating out platform-specific
    implementations.

  - lots of lock-debugging and assertions in the runtime.  In particular,
    when DEBUG is on, we catch multiple ACQUIRE_LOCK()s, and there is
    also an ASSERT_LOCK_HELD() call.

What's missing so far:

  - I have almost certainly broken the Win32 build, will fix soon.

  - any kind of thread migration or load balancing.  This is high up
    the agenda, though.

  - various performance tweaks to do

  - throwTo and forkProcess still do not work in SMP mode

Two SMP-related changes:

  - New storage manager interface:

    bdescr *allocateLocal(StgRegTable *reg, nat words)

    which allocates from the current thread's nursery (being careful
    not to clash with the heap pointer).  It can do this without
    taking any locks; the lock only has to be taken if a block needs
    to be allocated.  allocateLocal() is now used instead of allocate()
    in a few PrimOps.

    This removes locks from most Integer operations, cutting down
    the overhead for SMP a bit more.

    To make this work, we have to be able to grab the current thread's
    Capability out of thin air (i.e. when called from GMP), so the
    Capability subsystem needs to keep a hash from thread IDs to
    Capabilities.

  - Small MVar optimisation: instead of taking the global
    storage-manager lock, do our own locking of MVars with a bit of
    inline assembly (x86 only for now).

StgFunInfoExtra_slow_apply(): convert the slow_apply_offset to a W_
before arithmetic.

- Now that labels are always prefixed with '&' in .hc code, we have to
  fix some sloppiness in the RTS .cmm code.  Fortunately it's not too
  painful.

- SMP: acquire/release the storage manager lock around
  atomicModifyMutVar#.  This is a hack: atomicModifyMutVar# isn't
  atomic under SMP otherwise, but the SM lock is a large sledgehammer.
  I think I'll apply the sledgehammer to the MVar primitives too, for
  the time being.

SMP: the rest of the changes to support safe thunk entry & updates.  I
thought the compiler changes were independent, but I ended up breaking
the HEAD, so I'll have to commit the rest.  non-SMP compilation should
not be affected.

Track size change of alloc_blocks and alloc_blocks_lim.

(They are of type nat, which used to be the same size as W_, but now is
the same size as CInt).

* Some preprocessors don't like the C99/C++ '//' comments after a
  directive, so use '/* */' instead. For consistency, a lot of '//' in
  the include files were converted, too.

* UnDOSified libraries/base/cbits/runProcess.c.

* My favourite sport: Killed $Id$s.

assertion failures should go through the RtsMessages layer, so they
get a pop-up box in a Windows app.

64-bit fixes.

Don't assume that sizeof(int) == sizeof(StgInt).
This assumption creeped in in many places since 6.2.

Position Independent Code and Dynamic Linking Support, Part 1

This commit allows generation of position independent code (PIC) that fully supports dynamic linking on Mac OS X and PowerPC Linux.
Other platforms are not yet supported, and there is no support for actually linking or using dynamic libraries - so if you use the -fPIC or -dynamic code generation flags, you have to type your (platform-specific) linker command lines yourself.


nativeGen/PositionIndependentCode.hs:
New file. Look here for some more comments on how this works.

cmm/CLabel.hs:
Add support for DynamicLinkerLabels and PIC base labels - for use inside the NCG.
needsCDecl: Case alternative labels now need C decls, see the codeGen/CgInfoTbls.hs below for details

cmm/Cmm.hs:
Add CmmPicBaseReg (used in NCG),
and CmmLabelDiffOff (used in NCG and for offsets in info tables)

cmm/CmmParse.y:
support offsets in info tables

cmm/PprC.hs:
support CmmLabelDiffOff
Case alternative labels now need C decls (see the codeGen/CgInfoTbls.hs for details), so we need to pprDataExterns for info tables.

cmm/PprCmm.hs:
support CmmLabelDiffOff

codeGen/CgInfoTbls.hs:
no longer store absolute addresses in info tables, instead, we store offsets.
Also, for vectored return points, emit the alternatives _after_ the vector table. This is to work around a limitation in Apple's as, which refuses to handle label differences where one label is at the end of a section. Emitting alternatives after vector info tables makes sure this never happens in GHC generated code. Case alternatives now require prototypes in hc code, though (see changes in PprC.hs, CLabel.hs).

main/CmdLineOpts.lhs:
Add a new option, -fPIC.

main/DriverFlags.hs:
Pass the correct options for PIC to gcc, depending on the platform. Only for powerpc for now.

nativeGen/AsmCodeGen.hs:
Many changes...
Mac OS X-specific management of import stubs is no longer, it's now part of a general mechanism to handle such things for all platforms that need it (Darwin [both ppc and x86], Linux on ppc, and some platforms we don't support).
Move cmmToCmm into its own monad which can accumulate a list of imported symbols. Make it call cmmMakeDynamicReference at the right places.

nativeGen/MachCodeGen.hs:
nativeGen/MachInstrs.hs:
nativeGen/MachRegs.lhs:
nativeGen/PprMach.hs:
nativeGen/RegAllocInfo.hs:
Too many changes to enumerate here, PowerPC specific.

nativeGen/NCGMonad.hs:
NatM still tracks imported symbols, as more labels can be created during code generation (float literals, jump tables; on some platforms all data access has to go through the dynamic linking mechanism).

driver/mangler/ghc-asm.lprl:
Mangle absolute addresses in info tables to offsets.
Correctly pass through GCC-generated PIC for Mac OS X and powerpc linux.

includes/Cmm.h:
includes/InfoTables.h:
includes/Storage.h:
includes/mkDerivedConstants.c:
rts/GC.c:
rts/GCCompact.c:
rts/HeapStackCheck.cmm:
rts/Printer.c:
rts/RetainerProfile.c:
rts/Sanity.c:
Adapt to the fact that info tables now contain offsets.

rts/Linker.c:
Mac-specific: change machoInitSymbolsWithoutUnderscore to support PIC.

Merge backend-hacking-branch onto HEAD.  Yay!