GitLab

The first phase of this tidyup is focussed on the header files, and in
particular making sure we are exposinng publicly exactly what we need
to, and no more.

 - Rts.h now includes everything that the RTS exposes publicly,
   rather than a random subset of it.

 - Most of the public header files have moved into subdirectories, and
   many of them have been renamed.  But clients should not need to
   include any of the other headers directly, just #include the main
   public headers: Rts.h, HsFFI.h, RtsAPI.h.

 - All the headers needed for via-C compilation have moved into the
   stg subdirectory, which is self-contained.  Most of the headers for
   the rest of the RTS APIs have moved into the rts subdirectory.

 - I left MachDeps.h where it is, because it is so widely used in
   Haskell code.
 
 - I left a deprecated stub for RtsFlags.h in place.  The flag
   structures are now exposed by Rts.h.

 - Various internal APIs are no longer exposed by public header files.

 - Various bits of dead code and declarations have been removed

 - More gcc warnings are turned on, and the RTS code is more
   warning-clean.

 - More source files #include "PosixSource.h", and hence only use
   standard POSIX (1003.1c-1995) interfaces.

There is a lot more tidying up still to do, this is just the first
pass.  I also intend to standardise the names for external RTS APIs
(e.g use the rts_ prefix consistently), and declare the internal APIs
as hidden for shared libraries.

Among others:
- Fixed Stg->C-- translation of let-no-escapes -- it's important to use the
  right continuation...
- Fixed infinite recursion in X86 backend (shortcutJump mishandled infinite loops)
- Fixed yet another wrong calling convention -- primops take args only in vanilla regs,
  but they may return results on the stack!
- Removed StackInfo from LGraph and Block -- now in LastCall and CmmZ
- Updated avail-variable and liveness code

o Fixed bug that emitted the copy-in code for closure entry
  in the wrong place -- at the initialization of the closure.
o Refactored some of the closure entry code.
o Added code to check that no LocalRegs are live-in to a procedure
   -- trip up some buggy programs earlier
o Fixed environment bindings for thunks
   -- we weren't (re)binding the free variables in a thunk
o Fixed a bug in proc-point splitting that dropped some updates
  to the entry block in a procedure.
o Fixed improper calls to code that generates CmmLit's for strings
o New invariant on cg_loc in CgIdInfo: the expression is always tagged
o Code to load free vars on entry to a thunk was (wrongly) placed before
  the heap check.
o Some of the StgCmm code was redundantly passing around Id's
  along with CgIdInfo's; no more.
o Initialize the LocalReg's that point to a closure before allocating and
  initializing the closure itself -- otherwise, we have problems with
  recursive closure bindings
o BlockEnv and BlockSet types are now abstract.
o Update frames:
  - push arguments in Old call area
  - keep track of the return sp in the FCode monad
  - keep the return sp in every call, tail call, and return
      (because it might be different at different call sites,
       e.g. tail calls to the gc after a heap check are performed
            before pushing the update frame)
  - set the sp appropriately on returns and tail calls
o Reduce call, tail call, and return to a single LastCall node
o Added slow entry code, using different calling conventions on entry and tail call
o More fixes to the calling convention code.
  The tricky stuff is all about the closure environment: it must be passed in R1,
  but in non-closures, there is no such argument, so we can't treat all arguments
  the same way: the closure environment is special. Maybe the right step forward
  would be to define a different calling convention for closure arguments.
o Let-no-escapes need to be emitted out-of-line -- otherwise, we drop code.
o Respect RTS requirement of word alignment for pointers
  My stack allocation can pack sub-word values into a single word on the stack,
  but it wasn't requiring word-alignment for pointers. It does now,
  by word-aligning both pointer registers and call areas.
o CmmLint was over-aggresively ruling out non-word-aligned memory references,
  which may be kosher now that we can spill small values into a single word.
o Wrong label order on a conditional branch when compiling switches.
o void args weren't dropped in many cases.
  To help prevent this kind of mistake, I defined a NonVoid wrapper,
  which I'm applying only to Id's for now, although there are probably
  other good candidates.
o A little code refactoring: separate modules for procpoint analysis splitting, 
  stack layout, and building infotables.
o Stack limit check: insert along with the heap limit check, using a symbolic
  constant (a special CmmLit), then replace it when the stack layout is known.
o Removed last node: MidAddToContext 
o Adding block id as a literal: means that the lowering of the calling conventions
  no longer has to produce labels early, which was inhibiting common-block elimination.
  Will also make it easier for the non-procpoint-splitting path.
o Info tables: don't try to describe the update frame!
o Over aggressive use of NonVoid!!!!
  Don't drop the non-void args before setting the type of the closure!!!
o Sanity checking:
  Added a pass to stub dead dead slots on the stack
  (only ~10 lines with the dataflow framework)
o More sanity checking:
  Check that incoming pointer arguments are non-stubbed.
  Note: these checks are still subject to dead-code removal, but they should
  still be quite helpful.
o Better sanity checking: why stop at function arguments?
  Instead, in mkAssign, check that _any_ assignment to a pointer type is non-null
  -- the sooner the crash, the easier it is to debug.
  Still need to add the debugging flag to turn these checks on explicitly.
o Fixed yet another calling convention bug.
  This time, the calls to the GC were wrong. I've added a new convention
  for GC calls and invoked it where appropriate.
  We should really straighten out the calling convention stuff:
    some of the code (and documentation) is spread across the compiler,
    and there's some magical use of the node register that should really
    be handled (not avoided) by calling conventions.
o Switch bug: the arms in mkCmmLitSwitch weren't returning to a single join point.
o Environment shadowing problem in Stg->Cmm:
  When a closure f is bound at the top-level, we should not bind f to the
  node register on entry to the closure.
  Why? Because if the body of f contains a let-bound closure g that refers
  to f, we want to make sure that it refers to the static closure for f.
  Normally, this would all be fine, because when we compile a closure,
  we rebind free variables in the environment. But f doesn't look like
  a free variable because it's a static value. So, the binding for f
  remains in the environment when we compile g, inconveniently referring
  to the wrong thing.
  Now, I bind the variable in the local environment only if the closure is not
  bound at the top level. It's still okay to make assumptions about the
  node holding the closure environment; we just won't find the binding
  in the environment, so code that names the closure will now directly
  get the label of the static closure, not the node register holding a
  pointer to the static closure.
o Don't generate bogus Cmm code containing SRTs during the STG -> Cmm pass!
  The tables made reference to some labels that don't exist when we compute and
  generate the tables in the back end.
o Safe foreign calls need some special treatment (at least until we have the integrated
  codegen). In particular:
  o they need info tables
  o they are not procpoints -- the successor had better be in the same procedure
  o we cannot (yet) implement the calling conventions early, which means we have
    to carry the calling-conv info all the way to the end
o We weren't following the old convention when registering a module.
  Now, we use update frames to push any new modules that have to be registered
  and enter the youngest one on the stack.
  We also use the update frame machinery to specify that the return should pop
  the return address off the stack.
o At each safe foreign call, an infotable must be at the bottom of the stack,
  and the TSO->sp must point to it.
o More problems with void args in a direct call to a function:
  We were checking the args (minus voids) to check whether the call was saturated,
  which caused problems when the function really wasn't saturated because it
  took an extra void argument.
o Forgot to distinguish integer != from floating != during Stg->Cmm
o Updating slotEnv and areaMap to include safe foreign calls
  The dataflow analyses that produce the slotEnv and areaMap give
  results for each basic block, but we also need the results for
  a safe foreign call, which is a middle node.
  After running the dataflow analysis, we have another pass that
  updates the results to includ any safe foreign calls.
o Added a static flag for the debugging technique that inserts
  instructions to stub dead slots on the stack and crashes when
  a stubbed value is loaded into a pointer-typed LocalReg.
o C back end expects to see return continuations before their call sites.
  Sorted the flowgraphs appropriately after splitting.
o PrimOp calling conventions are special -- unlimited registers, no stack
  Yet another calling convention...
o More void value problems: if the RHS of a case arm is a void-typed variable,
  don't try to return it.
o When calling some primOp, they may allocate memory; if so, we need to
  do a heap check when we return from the call.

This merge does not turn on the new codegen (which only compiles
a select few programs at this point),
but it does introduce some changes to the old code generator.

The high bits:
1. The Rep Swamp patch is finally here.
   The highlight is that the representation of types at the
   machine level has changed.
   Consequently, this patch contains updates across several back ends.
2. The new Stg -> Cmm path is here, although it appears to have a
   fair number of bugs lurking.
3. Many improvements along the CmmCPSZ path, including:
   o stack layout
   o some code for infotables, half of which is right and half wrong
   o proc-point splitting

Modules that need it import it themselves instead.

I've renamed a number of type and data constructors within Cmm so that
the names used in the compiler may more closely reflect the C--
specification 2.1.  I've done a bit of other renaming as well.
Highlights:

  CmmFormal and CmmActual now bear a CmmKind (which for now is a
                                              MachHint as before)
  CmmFormals = [CmmFormal] and CmmActuals = [CmmActual]
  
  suitable changes have been made to both code and nonterminals in the
  Cmm parser (which is as yet untested)

  For reasons I don't understand, parts of the code generator use a
  sequence of 'formal parameters' with no C-- kinds.  For these we now
  have the types
    type CmmFormalWithoutKind   = LocalReg
    type CmmFormalsWithoutKinds = [CmmFormalWithoutKind]

  A great many appearances of (Tau, MachHint) have been simplified to
  the appropriate CmmFormal or CmmActual, though I'm sure there are
  more opportunities.

  Kind and its data constructors are now renamed to
     data GCKind = GCKindPtr | GCKindNonPtr 
  to avoid confusion with the Kind used in the type checker and with CmmKind.

Finally, in a somewhat unrelated bit (and in honor of Simon PJ, who
thought of the name), the Whalley/Davidson 'transaction limit' is now
called 'OptimizationFuel' with the net effect that there are no longer
two unrelated uses of the abbreviation 'tx'.

Changes too numerous to comment on, but here is some old history that
I saved: 


Wed Aug 15 11:07:13 BST 2007  Norman Ramsey <nr@eecs.harvard.edu>
  * type synonyms made consistent with new Cmm types

    M ./compiler/nativeGen/MachInstrs.hs -2 +2

Mon Aug 20 19:22:14 BST 2007  Norman Ramsey <nr@eecs.harvard.edu>
  * pushing return info beyond cmm into codegen

    M ./compiler/codeGen/Bitmap.hs r3
    M ./compiler/codeGen/CgBindery.lhs r3
    M ./compiler/codeGen/CgCallConv.hs r3
    M ./compiler/codeGen/CgCase.lhs r3
    M ./compiler/codeGen/CgClosure.lhs r3
    M ./compiler/codeGen/CgCon.lhs r3
    M ./compiler/codeGen/CgExpr.lhs r3
    M ./compiler/codeGen/CgForeignCall.hs -6 +7 r3
    M ./compiler/codeGen/CgHeapery.lhs r3
    M ./compiler/codeGen/CgHpc.hs +1 r3
    M ./compiler/codeGen/CgInfoTbls.hs r3
    M ./compiler/codeGen/CgLetNoEscape.lhs r3
    M ./compiler/codeGen/CgMonad.lhs r3
    M ./compiler/codeGen/CgParallel.hs r3
    M ./compiler/codeGen/CgPrimOp.hs +3 r3
    M ./compiler/codeGen/CgProf.hs r3
    M ./compiler/codeGen/CgStackery.lhs r3
    M ./compiler/codeGen/CgTailCall.lhs r3
    M ./compiler/codeGen/CgTicky.hs r3
    M ./compiler/codeGen/CgUtils.hs -1 +1 r3
    M ./compiler/codeGen/ClosureInfo.lhs r3
    M ./compiler/codeGen/CodeGen.lhs r3
    M ./compiler/codeGen/SMRep.lhs r3
    M ./compiler/nativeGen/AsmCodeGen.lhs -2 +2 r1
    M ./compiler/nativeGen/MachCodeGen.hs -3 +3 r1
    M ./compiler/nativeGen/MachInstrs.hs r1
    M ./compiler/nativeGen/MachRegs.lhs r1
    M ./compiler/nativeGen/NCGMonad.hs r1
    M ./compiler/nativeGen/PositionIndependentCode.hs r1
    M ./compiler/nativeGen/PprMach.hs r1
    M ./compiler/nativeGen/RegAllocInfo.hs r1
    M ./compiler/nativeGen/RegisterAlloc.hs r1

Mon Aug 20 20:54:41 BST 2007  Norman Ramsey <nr@eecs.harvard.edu>
  * put CmmReturnInfo into a CmmCall (and related types)

    M ./compiler/cmm/Cmm.hs -2 +1 r3
    M ./compiler/cmm/CmmBrokenBlock.hs -13 +12 r1
    M ./compiler/cmm/CmmCPS.hs -3 +3
    M ./compiler/cmm/CmmCPSGen.hs -8 +6 r1
    M ./compiler/cmm/CmmLint.hs -1 +1
    M ./compiler/cmm/CmmLive.hs -1 +1
    M ./compiler/cmm/CmmOpt.hs -3 +3
    M ./compiler/cmm/CmmParse.y -6 +6 r3
    M ./compiler/cmm/PprC.hs -3 +3
    M ./compiler/cmm/PprCmm.hs -7 +4 r2
    M ./compiler/codeGen/CgForeignCall.hs -7 +6 r2
    M ./compiler/codeGen/CgHpc.hs -1 r1
    M ./compiler/codeGen/CgPrimOp.hs -3 r1
    M ./compiler/codeGen/CgUtils.hs -1 +1 r1
    M ./compiler/nativeGen/AsmCodeGen.lhs -2 +2
    M ./compiler/nativeGen/MachCodeGen.hs -3 +3 r1

Tue Aug 21 18:09:13 BST 2007  Norman Ramsey <nr@eecs.harvard.edu>
  * add call info in nativeGen

    M ./compiler/nativeGen/AsmCodeGen.lhs r1
    M ./compiler/nativeGen/MachInstrs.hs r1
    M ./compiler/nativeGen/MachRegs.lhs r1
    M ./compiler/nativeGen/NCGMonad.hs r1
    M ./compiler/nativeGen/PositionIndependentCode.hs r1
    M ./compiler/nativeGen/PprMach.hs r1
    M ./compiler/nativeGen/RegAllocInfo.hs r1

Wed Aug 22 16:41:58 BST 2007  Norman Ramsey <nr@eecs.harvard.edu>
  * ListGraph is now a newtype, not a synonym
  The resultant bookkeepping is unenviable, but the change
  greatly simplifies our ability to make Cmm things propertly
  Outputable for both list-graph and zipper-graph representations.

    M ./compiler/cmm/Cmm.hs -5 +3
    M ./compiler/cmm/CmmCPS.hs -2 +2
    M ./compiler/cmm/CmmCPSGen.hs -1 +1
    M ./compiler/cmm/CmmContFlowOpt.hs -3 +3
    M ./compiler/cmm/CmmCvt.hs -2 +2
    M ./compiler/cmm/CmmInfo.hs -2 +3
    M ./compiler/cmm/CmmLint.hs -1 +1
    M ./compiler/cmm/CmmOpt.hs -2 +2
    M ./compiler/cmm/PprC.hs -1 +1
    M ./compiler/cmm/PprCmm.hs -5 +8
    M ./compiler/cmm/PprCmmZ.hs -7 +1
    M ./compiler/codeGen/CgMonad.lhs -1 +1
    M ./compiler/nativeGen/AsmCodeGen.lhs -15 +15
    M ./compiler/nativeGen/MachCodeGen.hs -2 +2
    M ./compiler/nativeGen/PositionIndependentCode.hs -6 +6
    M ./compiler/nativeGen/PprMach.hs -3 +2
    M ./compiler/nativeGen/RegAllocColor.hs +1
    M ./compiler/nativeGen/RegAllocLinear.hs -4 +5
    M ./compiler/nativeGen/RegCoalesce.hs -6 +6
    M ./compiler/nativeGen/RegLiveness.hs -12 +12

Thu Aug 23 13:44:49 BST 2007  Norman Ramsey <nr@eecs.harvard.edu>
  * diagnostic assistance in case fromJust fails

    M ./compiler/nativeGen/MachCodeGen.hs -2 +5

Thu Aug 23 14:07:28 BST 2007  Norman Ramsey <nr@eecs.harvard.edu>
  * give every block, even the first, a label
    With branch-chain elimination, the first block of a procedure
    might be the target of a branch.  This actually happens to 
    a dozen or more procedures in the run-time system.

    M ./compiler/nativeGen/PprMach.hs -8 +3

Fri Aug 24 17:27:04 BST 2007  Norman Ramsey <nr@eecs.harvard.edu>
  * clean up the code in PprMach

    M ./compiler/nativeGen/PprMach.hs -16 +14

Fri Aug 24 19:35:03 BST 2007  Norman Ramsey <nr@eecs.harvard.edu>
  * a bunch of impedance matching to get the compiler to build, plus 
   * the plus is diagnostics for unreachable code, which required
     moving a lot of prettyprinting code

    M ./compiler/cmm/Cmm.hs -7 +5
    M ./compiler/cmm/CmmCPSZ.hs -1 +1
    M ./compiler/cmm/CmmCvt.hs -8 +8
    M ./compiler/cmm/CmmParse.y -4 +3
    M ./compiler/cmm/MkZipCfg.hs -19 +9
    M ./compiler/cmm/PprCmmZ.hs -118 +4
    M ./compiler/cmm/ZipCfg.hs -1 +13
    M ./compiler/cmm/ZipCfgCmm.hs -10 +129
    M ./compiler/main/HscMain.lhs -4 +4
    M ./compiler/nativeGen/NCGMonad.hs -2 +2
    M ./compiler/nativeGen/RegAllocInfo.hs -3 +3

Fri Aug 31 14:38:02 BST 2007  Norman Ramsey <nr@eecs.harvard.edu>
  * fix a warning about an import

    M ./compiler/nativeGen/RegAllocColor.hs -1 +1

The type parameter to a C-- procedure now represents a control-flow
graph, not a single instruction.  The newtype ListGraph preserves the 
current representation while enabling other representations and a
sensible way of prettyprinting.  Except for a few changes in the
prettyprinter the new compiler binary should be bit-for-bit identical
to the old.

Older GHCs can't parse OPTIONS_GHC.
This also changes the URL referenced for the -w options from
WorkingConventions#Warnings to CodingStyle#Warnings for the compiler
modules.

Consistently make the type and description in the info table an offset
or a pointer, depending on whether tables are next to code or not.

Fixes read002 on x86_64 (and maybe others).  The stg_ap_f_info info
table had the wrong liveness on it, because the float had been treated
as having zero size.

* Fix code output order when printing C so things are defined before
  they are used.
* Generate _ret rather than _entry functions for INFO_TABLE_RET.
* Use "ASSIGN_BaseReg" rather than "BaseReg =".

(This required a bit of refactoring of CmmInfo.)

This eliminates one of the panics introduced by 
the previous patch:
'First pass at implementing info tables for CPS'
  
The other panic introduced by that patch still remains.
It was due to the need to convert from a
  ContinuationInfo to a CmmInfo.
  (codeGen/CgInfoTbls.hs:emitClosureCodeAndInfoTable)
  (codeGen/CgInfoTbls.hs:emitReturnTarget)