GitLab

Merge backend-hacking-branch onto HEAD.  Yay!

Add accurate source location annotations to HsSyn
-------------------------------------------------

Every syntactic entity in HsSyn is now annotated with a SrcSpan, which
details the exact beginning and end points of that entity in the
original source file.  All honest compilers should do this, and it was
about time GHC did the right thing.

The most obvious benefit is that we now have much more accurate error
messages; when running GHC inside emacs for example, the cursor will
jump to the exact location of an error, not just a line somewhere
nearby.  We haven't put a huge amount of effort into making sure all
the error messages are accurate yet, so there could be some tweaking
still needed, although the majority of messages I've seen have been
spot-on.

Error messages now contain a column number in addition to the line
number, eg.

   read001.hs:25:10: Variable not in scope: `+#'

To get the full text span info, use the new option -ferror-spans.  eg.

   read001.hs:25:10-11: Variable not in scope: `+#'

I'm not sure whether we should do this by default.  Emacs won't
understand the new error format, for one thing.

In a more elaborate editor setting (eg. Visual Studio), we can arrange
to actually highlight the subexpression containing an error.  Eventually
this information will be used so we can find elements in the abstract
syntax corresponding to text locations, for performing high-level editor
functions (eg. "tell me the type of this expression I just highlighted").

Performance of the compiler doesn't seem to be adversely affected.
Parsing is still quicker than in 6.0.1, for example.

Implementation:

This was an excrutiatingly painful change to make: both Simon P.J. and
myself have been working on it for the last three weeks or so.  The
basic changes are:

 - a new datatype SrcSpan, which represents a beginning and end position
   in a source file.

 - To reduce the pain as much as possible, we also defined:

      data Located e = L SrcSpan e

 - Every datatype in HsSyn has an equivalent Located version.  eg.

      type LHsExpr id = Located (HsExpr id)

   and pretty much everywhere we used to use HsExpr we now use
   LHsExpr.  Believe me, we thought about this long and hard, and
   all the other options were worse :-)


Additional changes/cleanups we made at the same time:

  - The abstract syntax for bindings is now less arcane.  MonoBinds
    and HsBinds with their built-in list constructors have gone away,
    replaced by HsBindGroup and HsBind (see HsSyn/HsBinds.lhs).

  - The various HsSyn type synonyms have now gone away (eg. RdrNameHsExpr,
    RenamedHsExpr, and TypecheckedHsExpr are now HsExpr RdrName,
    HsExpr Name, and HsExpr Id respectively).

  - Utilities over HsSyn are now collected in a new module HsUtils.
    More stuff still needs to be moved in here.

  - MachChar now has a real Char instead of an Int.  All GHC versions that
    can compile GHC now support 32-bit Chars, so this was a simplification.

This commit does a long-overdue tidy-up

* Remove PprType (gets rid of one more bunch of hi-boot files)

* Put pretty-printing for types in TypeRep

* Make a specialised pretty-printer for Types, rather than
  converting to IfaceTypes and printing those

CasmTarget is gone

Some random import trimming

Fix obscure bug in GHCi: when generating code for tag2enum#, we were
wrongly using the source name for the DataCons rather than the worker
name, which lead to spurious link errors.

This fixes galois_raytrace(ghci).

Change the way SRTs are represented:

Previously, the SRT associated with a function or thunk would be a
sub-list of the enclosing top-level function's SRT.  But this approach
can lead to lots of duplication: if a CAF is referenced in several
different thunks, then it may appear several times in the SRT.
Let-no-escapes compound the problem, because the occurrence of a
let-no-escape-bound variable would expand to all the CAFs referred to
by the let-no-escape.

The new way is to describe the SRT associated with a function or thunk
as a (pointer+offset,bitmap) pair, where the pointer+offset points
into some SRT table (the enclosing function's SRT), and the bitmap
indicates which entries in this table are "live" for this closure.
The bitmap is stored in the 16 bits previously used for the length
field, but this rarely overflows.  When it does overflow, we store the
bitmap externally in a new "SRT descriptor".

Now the enclosing SRT can be a set, hence eliminating the duplicates.

Also, we now have one SRT per top-level function in a recursive group,
where previously we used to have one SRT for the whole group.  This
helps keep the size of SRTs down.

Bottom line: very little difference most of the time.  GHC itself got
slightly smaller.  One bad case of a module in GHC which had a huge
SRT has gone away.

While I was in the area:

  - Several parts of the back-end require bitmaps.  Functions for
    creating bitmaps are now centralised in the Bitmap module.

  - We were trying to be independent of word-size in a couple of
    places in the back end, but we've now abandoned that strategy so I
    simplified things a bit.

Off-by-one tidyup.

ALLOC_AP, ALLOC_PAP and MKAP were all being constructed
with size arguments equal to (1+number of args/FVs) in
ByteCodeGen.schemeE, only for Interpreter.c to subtract 1
when fishing out the payloads. This commit drops the
up-and-downery.

Simplification spotted by Andy Moran

NCG support for f.e.d. stdcall -- Literal.MachLabels now optionally carry
the size (in bytes) of the stack frame it expects, if known. That just
so happens to match what stdcall labels need to be annotated with when
emitting them in the NCG..

A round of space-leak fixing.

  - re-instate zapping of the PersistentCompilerState at various
    points during the compilation cycle in HscMain.  This affects
    one-shot compilation only, since in this mode the information
    collected in the PCS is not required after creating the final
    interface file.

  - Unravel the recursive dependency between MkIface and
    CoreTidy/CoreToStg.  Previously the CafInfo for each binding was
    calculated by CoreToStg, and fed back into the IdInfo of the Ids
    generated by CoreTidy (an earlier pass).  MkIface then took this
    IdInfo and the bindings from CoreTidy to generate the interface;
    but it couldn't do this until *after* CoreToStg, because the CafInfo
    hadn't been calculated yet.  The result was that the CoreTidy
    output lived until after CoreToStg, and at the same time as the
    CorePrep and STG syntax, which is wasted space, not to mention
    the complexity and general ugliness in HscMain.

    So now we calculate CafInfo directly in CoreTidy.  The downside is
    that we have to predict what CorePrep is going to do to the
    bindings so we can tell what will turn into a CAF later, but it's
    no worse than before (it turned out that we were doing this
    prediction before in CoreToStg anyhow).

  - The typechecker lazilly typechecks unfoldings.  It turns out that
    this is a good idea from a performance perspective, but it also
    means that it must hang on to all the information it needs to
    do the typechecking.  Previously this meant holding on to the
    whole of the typechecker's environment, which includes all sorts
    of stuff which isn't necessary to typecheck unfoldings.  By paring
    down the environment captured by the lazy unfoldings, we can
    save quite a bit of space in the phases after typechecking.

Comment change only

A wibble to the constructor-naming story

-------------------------------------
  Big upheaval to the way that constructors are named
	-------------------------------------

This commit enshrines the new story for constructor names.  We could never
really get External Core to work nicely before, but now it does.

The story is laid out in detail in the Commentary
	ghc/docs/comm/the-beast/data-types.html
so I will not repeat it here.

	[Manuel: the commentary isn't being updated, apparently.]

However, the net effect is that in Core and in External Core, contructors look
like constructors, and the way things are printed is all consistent.

It is a fairly pervasive change (which is why it has been so long postponed),
but I hope the question is now finally closed.

All the libraries compile etc, and I've run many tests, but doubtless there will
be some dark corners.

- Reverse the code for workers and wrappers for nullary constructors.
  For some reason it was the wrong way around, but the effects were
  harmless since they both evaluate to the same thing.

- When passing a nullary constructor as an argument, we should pass
  the name of the worker rather than the wrapper.  Again, this is
  mostly harmless, but it enables some small simplification in
  pushAtom.

- Rearrange/cleanup pushAtom.

Changes to the way stack checks are handled in GHCi, to fix a rare bug
when a stack check fails in a BCO.

We now aggregate all stack use from case alternatives up to the
enclosing function/thunk BCO, and do a single stack check at the
beginning of that BCO.  This simplifies the stack check failure code,
because it doesn't have to cope with the case when a case alternative
needs to restart.

We still employ the trick of doing a fixed stack check before every
BCO, only inserting an actual stack check instruction in the BCO if it
needs more stack than this fixed amount.  The fixed stack check is now
only done before running a function/thunk BCO.

Wibble

---------------------------------------
	Improvements to the byte-code generator
	---------------------------------------

1. The schemeR call in coreExprToBCOs was bogusly passing a bunch of free
   variables, when the set should always be empty. As a result, compiling
   an expression with an unbound free variable (e.g. 'x + 1', where 'x' is
   entirely unbound) succeeded, expecting 'x' to be passed on the stack,
   which of course it won't be.

   This bug only shows up if an earlier stage of the compiler goes wrong,
   but fixing turns a seg-fault into a more graceful failure.

2. Make schemeE allocate non-recursive constructors directly.

3. Lots of general tidying up.  Result is 50 lines shorter than before.

Correctly describe the stack during a GHCi CCALL instruction to the
RTS.  The previous hack, temporarily truncating the stack to the
topmost valid stack frame, didn't work because stack-squeezing tends
to move the stack around before the call.

The right thing to do is correctly describe the chunk of ccall args
with an info table, which is what this change does.  We use a RET_DYN
info table with the number of non-ptrs from the CCALL instruction.

Merge the eval-apply-branch on to the HEAD
------------------------------------------

This is a change to GHC's evaluation model in order to ultimately make
GHC more portable and to reduce complexity in some areas.

At some point we'll update the commentary to describe the new state of
the RTS.  Pending that, the highlights of this change are:

  - No more Su.  The Su register is gone, update frames are one
    word smaller.

  - Slow-entry points and arg checks are gone.  Unknown function calls
    are handled by automatically-generated RTS entry points (AutoApply.hc,
    generated by the program in utils/genapply).

  - The stack layout is stricter: there are no "pending arguments" on
    the stack any more, the stack is always strictly a sequence of
    stack frames.

    This means that there's no need for LOOKS_LIKE_GHC_INFO() or
    LOOKS_LIKE_STATIC_CLOSURE() any more, and GHC doesn't need to know
    how to find the boundary between the text and data segments (BIG WIN!).

  - A couple of nasty hacks in the mangler caused by the neet to
    identify closure ptrs vs. info tables have gone away.

  - Info tables are a bit more complicated.  See InfoTables.h for the
    details.

  - As a side effect, GHCi can now deal with polymorphic seq.  Some bugs
    in GHCi which affected primitives and unboxed tuples are now
    fixed.

  - Binary sizes are reduced by about 7% on x86.  Performance is roughly
    similar, some programs get faster while some get slower.  I've seen
    GHCi perform worse on some examples, but haven't investigated
    further yet (GHCi performance *should* be about the same or better
    in theory).

  - Internally the code generator is rather better organised.  I've moved
    info-table generation from the NCG into the main codeGen where it is
    shared with the C back-end; info tables are now emitted as arrays
    of words in both back-ends.  The NCG is one step closer to being able
    to support profiling.

This has all been fairly thoroughly tested, but no doubt I've messed
up the commit in some way.

------------------
  	Improve byte-code compilation of unboxed
	 	tuple returns
		------------------

The previous byte-code for returning unboxed tuples was just wrong.  It's
a special case for the situation where we return (# s, x #), where s is
a state token, so we can just return the single result 'x' on top of the
stack.  Previously we generated an ENTER at the end, which is plain wrong.
We should RETURN.

It still doesn't work, for other tiresome reasons...but rather than fix it
we'll wait for eval-apply.  Meanwhile it's less wrong than before.

--------------------------------------
	Make Template Haskell into the HEAD
	--------------------------------------

This massive commit transfers to the HEAD all the stuff that
Simon and Tim have been doing on Template Haskell.  The
meta-haskell-branch is no more!

WARNING: make sure that you

  * Update your links if you are using link trees.
    Some modules have been added, some have gone away.

  * Do 'make clean' in all library trees.
    The interface file format has changed, and you can
    get strange panics (sadly) if GHC tries to read old interface files:
    e.g.  ghc-5.05: panic! (the `impossible' happened, GHC version 5.05):
	  Binary.get(TyClDecl): ForeignType

  * You need to recompile the rts too; Linker.c has changed


However the libraries are almost unaltered; just a tiny change in
Base, and to the exports in Prelude.


NOTE: so far as TH itself is concerned, expression splices work
fine, but declaration splices are not complete.


		---------------
		The main change
		---------------

The main structural change: renaming and typechecking have to be
interleaved, because we can't rename stuff after a declaration splice
until after we've typechecked the stuff before (and the splice
itself).

* Combine the renamer and typecheker monads into one
	(TcRnMonad, TcRnTypes)
  These two replace TcMonad and RnMonad

* Give them a single 'driver' (TcRnDriver).  This driver
  replaces TcModule.lhs and Rename.lhs

* The haskell-src library package has a module
	Language/Haskell/THSyntax
  which defines the Haskell data type seen by the TH programmer.

* New modules:
	hsSyn/Convert.hs 	converts THSyntax -> HsSyn
	deSugar/DsMeta.hs 	converts HsSyn -> THSyntax

* New module typecheck/TcSplice type-checks Template Haskell splices.

		-------------
		Linking stuff
		-------------

* ByteCodeLink has been split into
	ByteCodeLink	(which links)
	ByteCodeAsm	(which assembles)

* New module ghci/ObjLink is the object-code linker.

* compMan/CmLink is removed entirely (was out of place)
  Ditto CmTypes (which was tiny)

* Linker.c initialises the linker when it is first used (no need to call
  initLinker any more).  Template Haskell makes it harder to know when
  and whether to initialise the linker.


	-------------------------------------
	Gathering the LIE in the type checker
	-------------------------------------

* Instead of explicitly gathering constraints in the LIE
	tcExpr :: RenamedExpr -> TcM (TypecheckedExpr, LIE)
  we now dump the constraints into a mutable varabiable carried
  by the monad, so we get
	tcExpr :: RenamedExpr -> TcM TypecheckedExpr

  Much less clutter in the code, and more efficient too.
  (Originally suggested by Mark Shields.)


		-----------------
		Remove "SysNames"
		-----------------

Because the renamer and the type checker were entirely separate,
we had to carry some rather tiresome implicit binders (or "SysNames")
along inside some of the HsDecl data structures.  They were both
tiresome and fragile.

Now that the typechecker and renamer are more intimately coupled,
we can eliminate SysNames (well, mostly... default methods still
carry something similar).

		-------------
		Clean up HsPat
		-------------

One big clean up is this: instead of having two HsPat types (InPat and
OutPat), they are now combined into one.  This is more consistent with
the way that HsExpr etc is handled; there are some 'Out' constructors
for the type checker output.

So:
	HsPat.InPat	--> HsPat.Pat
	HsPat.OutPat	--> HsPat.Pat
	No 'pat' type parameter in HsExpr, HsBinds, etc

	Constructor patterns are nicer now: they use
		HsPat.HsConDetails
	for the three cases of constructor patterns:
		prefix, infix, and record-bindings

	The *same* data type HsConDetails is used in the type
	declaration of the data type (HsDecls.TyData)

Lots of associated clean-up operations here and there.  Less code.
Everything is wonderful.

Exception => Control.Exception
GlaExts => GHC.Exts

Make this compile, and maybe even work

Make the byte-code generator understand about unboxed
tuple returns.  The previous code was just wrong.

This code is better but it is still not *right*, I fear.
Don't merge till we sort this out.

Re-enable bootstrapping: More Ptr trouble, now that it's (almost) abstract...

(F)SLIT fixes, continued...

FastString cleanup, stage 1.

The FastString type is no longer a mixture of hashed strings and
literal strings, it contains hashed strings only with O(1) comparison
(except for UnicodeStr, but that will also go away in due course).  To
create a literal instance of FastString, use FSLIT("..").

By far the most common use of the old literal version of FastString
was in the pattern

	  ptext SLIT("...")

this combination still works, although it doesn't go via FastString
any more.  The next stage will be to remove the need to use this
special combination at all, using a RULE.

To convert a FastString into an SDoc, now use 'ftext' instead of
'ptext'.

I've also removed all the FAST_STRING related macros from HsVersions.h
except for SLIT and FSLIT, just use the relevant functions from
FastString instead.

Friday afternoon pet peeve removal: define (Util.notNull :: [a] -> Bool) and use it

oops, accidentally committed some untested (and non-working) cleanups
in the last commit.  This commit fixes it up again.

(fixes the ByteCodeGen panic in GHCi on the HEAD)

PrelGHC ==> GHC.Prim (in comments only)

------------------------
	Kill Type.splitRepFunTys
	------------------------

splitRepFunTys was a Bad Function that split up a function type
looking through even recursive newtypes.  Alas, it diverged if
when we had a recursive newtype with a function whose result was
the newtype itself.

I've replaced it with ordinary splitFunTys, plus a new function
Type.dropForAlls, which does what you would expect.

Binary Interface Files - stage 1
--------------------------------

This commit changes the default interface file format from text to
binary, in order to improve compilation performace.

To view an interface file, use 'ghc --show-iface Foo.hi'.

utils/Binary.hs is the basic Binary I/O library, based on the nhc98
binary I/O library but much stripped-down and working in terms of
bytes rather than bits, and with some special features for GHC: it
remembers which Module is being emitted to avoid dumping too many
qualified names, and it keeps track of a "dictionary" of FastStrings
so that we don't dump the same FastString more than once into the
binary file.  I'll make a generic version of this for the libraries at
some point.

main/BinIface.hs contains most of the Binary instances.  Some
instances are in the same module as the data type (RdrName, Name,
OccName in particular).  Most instances were generated using a
modified version of DrIFT, which I'll commit later.  However, editing
them by hand isn't hard (certainly easier than modifying
ParseIface.y).

The first thing in a binary interface is the interface version, so
nice error messages will be generated if the binary format changes and
you still have old interfaces lying around.  The version also now
includes the "way" as an extra sanity check.

Other changes
-------------

I don't like the way FastStrings contain both hashed strings (with
O(1) comparison) and literal C strings (with O(n) comparison).  So as
a first step to separating these I made serveral "literal" type
strings into hashed strings.  SLIT() still generates a literal, and
now FSLIT() generates a hashed string.  With DEBUG on, you'll get a
warning if you try to compare any SLIT()s with anything, and the
compiler will fall over if you try to dump any literal C strings into
an interface file (usually indicating a use of SLIT() which should be
FSLIT()).

mkSysLocal no longer re-encodes its FastString argument each time it
is called.

I also fixed the -pgm options so that the argument can now optionally
be separted from the option.

Bugfix: PrelNames declared Names for several comparison primops, eg.
eqCharName, eqIntName etc. but these had different uniques from the
real primop names.  I've moved these to PrimOps and defined them using
mkPrimOpIdName instead, and deleted some for which we don't have real
primops (Manuel: please check that things still work for you after
this change).

Make foreign export dynamic work in GHCi.  Main changes:

* Allow literal labels to propagate through the bytecode generator
  and eventually be linked by the runtime linker.

* Minor mods to driver plumbing so that GHCi produces the relevant
  *_stub.[ch] files, compiles them with gcc, and loads the resulting .o's

* Dereference the stable pointer in the generated C stub, rather
  than passing it to a Haskell-world helper.  This seems simpler and
  removes the need to have a H-world helper, which in turn means the
  stub .o doesn't refer to any H-world entities.  This is important
  because our linker can't deal with mutual recursion between
  BCOs and loaded objects.

Still ToDo:

* Make it thread/GC safe.  (Sigbjorn?)

* Get rid of the bits of code in DsForeign which generate the
  Haskell helper.  I had a go but it wasn't obvious how to do it,
  so have deferred.

Switch over to the new hierarchical libraries
---------------------------------------------

This commit reorganises our libraries to use the new hierarchical
module namespace extension.

The basic story is this:

   - fptools/libraries contains the new hierarchical libraries.
     Everything in here is "clean", i.e. most deprecated stuff has
     been removed.

	- fptools/libraries/base is the new base package
	  (replacing "std") and contains roughly what was previously
	  in std, lang, and concurrent, minus deprecated stuff.
	  Things that are *not allowed* in libraries/base include:
		Addr, ForeignObj, ByteArray, MutableByteArray,
		_casm_, _ccall_, ``'', PrimIO

	  For ByteArrays and MutableByteArrays we use UArray and
	  STUArray/IOUArray respectively now.

	  Modules previously called PrelFoo are now under
	  fptools/libraries/GHC.  eg. PrelBase is now GHC.Base.

	- fptools/libraries/haskell98 provides the Haskell 98 std.
	  libraries (Char, IO, Numeric etc.) as a package.  This
	  package is enabled by default.

	- fptools/libraries/network is a rearranged version of
	  the existing net package (the old package net is still
	  available; see below).

	- Other packages will migrate to fptools/libraries in
	  due course.

     NB. you need to checkout fptools/libraries as well as
     fptools/hslibs now.  The nightly build scripts will need to be
     tweaked.

   - fptools/hslibs still contains (almost) the same stuff as before.
     Where libraries have moved into the new hierarchy, the hslibs
     version contains a "stub" that just re-exports the new version.
     The idea is that code will gradually migrate from fptools/hslibs
     into fptools/libraries as it gets cleaned up, and in a version or
     two we can remove the old packages altogether.

   - I've taken the opportunity to make some changes to the build
     system, ripping out the old hslibs Makefile stuff from
     mk/target.mk; the new package building Makefile code is in
     mk/package.mk (auto-included from mk/target.mk).

     The main improvement is that packages now register themselves at
     make boot time using ghc-pkg, and the monolithic package.conf
     in ghc/driver is gone.

     I've updated the standard packages but haven't tested win32,
     graphics, xlib, object-io, or OpenGL yet.  The Makefiles in
     these packages may need some further tweaks, and they'll need
     pkg.conf.in files added.

   - Unfortunately all this rearrangement meant I had to bump the
     interface-file version and create a bunch of .hi-boot-6 files :-(

Fix a classic bug: copying a Haskell string with one of the C string
functions (in this case strncpy()) is wrong when the string contains
'\0' characters.

The symptom in this case is that Happy parsers created with -ag don't
work in GHCi, because the state tables are encoded as strings
containing lots of '\0' elements.

Allow constructors with non-ptr fields to be allocated in-line.

Generate better code for lets whose RHS is a simple fn or constructor
application.  Details are in comment in code.

Catch up with recent changes in the middle end that changed the
assumptions about constructor applications: it is not necessarily the
case any more that constructor applications are always saturated in
the output from CorePrep.

For a non-saturated constructor application there is always a curried
worker function that can be called instead.  This change updates the
byte code generator to use the curried worker when necessary.

Also: a couple of micro-optimisations/cleanups.

This is one of those commits where the commit message is hundreds of
times, in bytes, larger than the fix.  If you count pixels, it's
probably more like thousands of times larger, since the fix involves
adding a missing apostrophe.

In compiling let bindings, when filtering the free vars of each RHS
against the supplied on-stack environment p, filter against p after it
has been augmented with the binders in this let (viz, p'), rather than
before.  Without this, letrec-bound binders can never "get started" in
the environment.

This fixes the HEAD crash for [(i,1) | i <- [1]].  Stable branch
is ok since the free-var machinery works a different way there.

- Fix the free variable calculation in schemeE following some changes
  to the global-vs-local name story in earlier parts of the compiler.
  (fixes GHCi breakage on the HEAD).

- Eliminate some duplicate free variable calculations.