Commit 0960a378 authored by Austin Seipp's avatar Austin Seipp
Browse files

rm -rf ./docs/comm

The Commentary is now on the wiki and far more complete. This copy will
of course live on in the annals of history, but there's no reason to
keep it around still.
Signed-off-by: default avatarAustin Seipp <>
parent c4e9f243
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=ISO-8859-1">
<title>The GHC Commentary - Parallel Arrays</title>
<h1>The GHC Commentary - Parallel Arrays</h1>
This section describes an experimental extension by high-performance
arrays, which comprises special syntax for array types and array
comprehensions, a set of optimising program transformations, and a set
of special purpose libraries. The extension is currently only partially
implemented, but the development will be tracked here.
Parallel arrays originally got their name from the aim to provide an
architecture-independent programming model for a range of parallel
computers. However, since experiments showed that the approach is also
worthwhile for sequential array code, the emphasis has shifted to their
parallel evaluation semantics: As soon as any element in a parallel
array is demanded, all the other elements are evaluated, too. This
makes parallel arrays more strict than <a
href="">standard Haskell 98
arrays</a>, but also opens the door for a loop-based implementation
strategy that leads to significantly more efficient code.
The programming model as well as the use of the <em>flattening
transformation</em>, which is central to the approach, has its origin in
the programming language <a
<h2>More Sugar: Special Syntax for Array Comprehensions</h2>
The option <code>-XParr</code>, which is a dynamic hsc option that can
be reversed with <code>-XNoParr</code>, enables special syntax for
parallel arrays, which is not essential to using parallel arrays, but
makes for significantly more concise programs. The switch works by
making the lexical analyser (located in <a
recognise the tokens <code>[:</code> and <code>:]</code>. Given that
the additional productions in the parser (located in <a
cannot be triggered without the lexer generating the necessary tokens,
there is no need to alter the behaviour of the parser.
The following additional syntax is accepted (the non-terminals are those
from the <a href="">Haskell 98 language
atype -> '[:' type ':] (parallel array type)
aexp -> '[:' exp1 ',' ... ',' expk ':]' (explicit array, k >= 0)
| '[:' exp1 [',' exp2] '..' exp3 ':]' (arithmetic array sequence)
| '[:' exp '|' quals1 '|' ... '|' qualsn ':]' (array comprehension, n >= 1)
quals -> qual1 ',' ... ',' qualn (qualifier list, n >= 1)
apat -> '[:' pat1 ',' ... ',' patk ':]' (array pattern, k >= 0)
Moreover, the extended comprehension syntax that allows for <em>parallel
qualifiers</em> (i.e., qualifiers separated by "<code>|</code>") is also
supported in list comprehensions. In general, the similarity to the
special syntax for list is obvious. The two main differences are that
(a) arithmetic array sequences are always finite and (b)
<code>[::]</code> is not treated as a constructor in expressions and
patterns, but rather as a special case of the explicit array syntax.
The former is a simple consequence of the parallel evaluation semantics
of parallel arrays and the latter is due to arrays not being a
constructor-based data type.
As a naming convention, types and functions that are concerned with
parallel arrays usually contain the string <code>parr</code> or
<code>PArr</code> (often as a prefix), and where corresponding types or
functions for handling lists exist, the name is identical, except for
containing the substring <code>parr</code> instead of <code>list</code>
(possibly in caps).
The following implications are worth noting explicitly:
<li>As the value and pattern <code>[::]</code> is an empty explicit
parallel array (i.e., something of the form <code>ExplicitPArr ty
[]</code> in the AST). This is in contrast to lists, which use the
nil-constructor instead. In the case of parallel arrays, using a
constructor would be rather awkward, as it is not a constructor-based
type. (This becomes rather clear in the desugarer.)
<li>As a consequence, array patterns have the general form <code>[:p1,
p2, ..., pn:]</code>, where <code>n</code> >= 0. Thus, two array
patterns overlap iff they have the same length -- an important property
for the pattern matching compiler.
<h2>Prelude Support for Parallel Arrays</h2>
The Prelude module <a
defines the standard operations and their types on parallel arrays and
provides a basic implementation based on boxed arrays. The interface of
<code>PrelPArr</code> is oriented by H98's <code>PrelList</code>, but
leaving out all functions that require infinite structures and adding
frequently needed array operations, such as permutations. Parallel
arrays are quite unqiue in that they use an entirely different
representation as soon as the flattening transformation is activated,
which is described further below. In particular, <code>PrelPArr</code>
defines the type <code>[::]</code> and operations to create, process,
and inspect parallel arrays. The type as well as the names of some of
the operations are also hardwired in <a
(see the definition of <code>parrTyCon</code> in this module) and <a
This is again very much like the case of lists, where the type is
defined in <a
and similarly wired in; however, for lists the entirely
constructor-based definition is exposed to user programs, which is not
the case for parallel arrays.
<h2>Desugaring Parallel Arrays</h2>
The parallel array extension requires the desugarer to replace all
occurrences of (1) explicit parallel arrays, (2) array patterns, and (3)
array comprehensions by a suitable combination of invocations of
operations defined in <code>PrelPArr</code>.
<h4>Explicit Parallel Arrays</h4>
These are by far the simplest to remove. We simply replace every
occurrence of <code>[:<i>e<sub>1</sub></i>, ...,
<i>e<sub>n</sub></i>:]</code> by
toP [<i>e<sub>1</sub></i>, ..., <i>e<sub>n</sub></i>]
i.e., we build a list of the array elements, which is, then, converted
into a parallel array.
<h4>Parallel Array Patterns</h4>
Array patterns are much more tricky as element positions may contain
further patterns and the <a
href="../the-beast/desugar.html#patmat">pattern matching compiler</a>
requires us to flatten all those out. But before we turn to the gory
details, here first the basic idea: A flat array pattern matches exactly
iff it's length corresponds to the length of the matched array. Hence,
if we have a set of flat array patterns matching an array value
<code>a</code>, it suffices to generate a Core <code>case</code>
expression that scrutinises <code>lengthP a</code> and has one
alternative for every length of array occuring in one of the patterns.
Moreover, there needs to be a default case catching all other array
lengths. In each alternative, array indexing (i.e., the functions
<code>!:</code>) is used to bind array elements to the corresponding
pattern variables. This sounds easy enough and is essentially what the
parallel array equation of the function <a
Unfortunately, however, the pattern matching compiler expects that it
can turn (almost) any pattern into variable patterns, literals, or
constructor applications by way of the functions <a
And to make matters worse, some weird machinery in the module <a
insists on being able to reverse the process (essentially to pretty
print patterns in warnings for incomplete or overlapping patterns).
The solution to this is an (unlimited) set of <em>fake</em> constructors
for parallel arrays, courtesy of <a
In other words, any pattern of the form <code>[:<i>p<sub>1</sub></i>,
..., <i>p<sub>n</sub></i>:]</code> is transformed into
MkPArray<i>n</i> <i>p<sub>1</sub></i> ... <i>p<sub>n</sub></i>
by <code>Match.tidy1</code>, then, run through the rest of the pattern
matching compiler, and finally, picked up by
<code>DsUtils.mkCoAlgCaseMatchResult</code>, which converts it into a
<code>case</code> expression as outlined above.
As an example consider the source expression
case v of
[:x1:] -> e1
[:x2, x3, x4:] -> e2
_ -> e3</pre>
<code>Match.tidy1</code> converts it into a form that is equivalent to
case v of {
MkPArr1 x1 -> e1;
MkPArr2 x2 x3 x4 -> e2;
_ -> e3;
which <code>DsUtils.mkCoAlgCaseMatchResult</code> turns into the
following Core code:
case lengthP v of
Int# i# ->
case i# of l {
DFT -> e3
1 -> let x1 = v!:0 in e1
3 -> let x2 = v!:0; x2 = v!:1; x3 = v!:2 in e2
<h4>Parallel Array Comprehensions</h4>
The most challenging construct of the three are array comprehensions.
In principle, it would be possible to transform them in essentially the
same way as list comprehensions, but this would lead to abysmally slow
code as desugaring of list comprehensions generates code that is
optimised for sequential, constructor-based structures. In contrast,
array comprehensions need to be transformed into code that solely relies
on collective operations and avoids the creation of many small
intermediate arrays.
The transformation is implemented by the function <a
In the following, we denote this transformation function by the form
<code>&lt;&lt;<i>e</i>&gt;&gt; pa ea</code>, where <code><i>e</i></code>
is the comprehension to be compiled and the arguments <code>pa</code>
and <code>ea</code> denote a pattern and the currently processed array
expression, respectively. The invariant constraining these two
arguments is that all elements in the array produced by <code>ea</code>
will <em>successfully</em> match against <code>pa</code>.
Given a source-level comprehensions <code>[:e | qss:]</code>, we compile
it with <code>&lt;&lt;[:e | qss:]&gt;&gt; () [:():]</code> using the
<<[:e' | :]>> pa ea = mapP (\pa -> e') ea
<<[:e' | b , qs:]>> pa ea = <<[:e' | qs:]>> pa (filterP (\pa -> b) ea)
<<[:e' | p <- e, qs:]>> pa ea =
let ef = filterP (\x -> case x of {p -> True; _ -> False}) e
<<[:e' | qs:]>> (pa, p) (crossP ea ef)
<<[:e' | let ds, qs:]>> pa ea =
<<[:e' | qs:]>> (pa, (x_1, ..., x_n))
(mapP (\v@pa -> (v, let ds in (x_1, ..., x_n))) ea)
{x_1, ..., x_n} = DV (ds) -- Defined Variables
<<[:e' | qs | qss:]>> pa ea =
<<[:e' | qss:]>> (pa, (x_1, ..., x_n))
(zipP ea <<[:(x_1, ..., x_n) | qs:]>>)
{x_1, ..., x_n} = DV (qs)</pre>
We assume the denotation of <code>crossP</code> to be given by
crossP :: [:a:] -> [:b:] -> [:(a, b):]
crossP a1 a2 = let
len1 = lengthP a1
len2 = lengthP a2
x1 = concatP $ mapP (replicateP len2) a1
x2 = concatP $ replicateP len1 a2
zipP x1 x2</pre>
For a more efficient implementation of <code>crossP</code>, see
Moreover, the following optimisations are important:
<li>In the <code>p &lt;- e</code> rule, if <code>pa == ()</code>, drop
it and simplify the <code>crossP ea e</code> to <code>e</code>.
<li>We assume that fusion will optimise sequences of array processing
<li>FIXME: Do we want to have the following function?
mapFilterP :: (a -> Maybe b) -> [:a:] -> [:b:]</pre>
Even with fusion <code>(mapP (\p -&gt; e) . filterP (\p -&gt;
b))</code> may still result in redundant pattern matching
operations. (Let's wait with this until we have seen what the
Simplifier does to the generated code.)
<h2>Doing Away With Nested Arrays: The Flattening Transformation</h2>
On the quest towards an entirely unboxed representation of parallel
arrays, the flattening transformation is the essential ingredient. GHC
uses a <a
improved version</a> of the transformation whose original form was
described by Blelloch &amp; Sabot. The flattening transformation
replaces values of type <code>[:a:]</code> as well as functions
operating on these values by alternative, more efficient data structures
and functions.
The flattening machinery is activated by the option
<code>-fflatten</code>, which is a static hsc option. It consists of
two steps: function vectorisation and array specialisation. Only the
first of those is implemented so far. If selected, the transformation
is applied to a module in Core form immediately after the <a
href="../the-beast/desugar.html">desugarer,</a> before the <a
href="../the-beast/simplifier.html">Mighty Simplifier</a> gets to do its
job. After vectorisation, the Core program can be dumped using the
option <code>-ddump-vect</code>. The is a good reason for us to perform
flattening immediately after the desugarer. In <a
the so-called <em>persistent compiler state</em> is maintained, which
contains all the information about imported interface files needed to
lookup the details of imported names (e.g., during renaming and type
checking). However, all this information is zapped before the
simplifier is invoked (supposedly to reduce the space-consumption of
GHC). As flattening has to get at all kinds of identifiers from Prelude
modules, we need to do it before the relevant information in the
persistent compiler state is gone.
As flattening generally requires all libraries to be compiled for
flattening (just like profiling does), there is a <em>compiler way</em>
<code>"ndp"</code>, which can be selected using the way option code
<code>-ndp</code>. This option will automagically select
<code>-XParr</code> and <code>-fflatten</code>.
The module <a
implements the monad <code>Flatten</code> that is used during
vectorisation to keep track of various sets of bound variables and a
variable substitution map; moreover, it provides a supply of new uniques
and allows us to look up names in the persistent compiler state (i.e.,
imported identifiers).
In order to be able to look up imported identifiers in the persistent
compiler state, it is important that these identifies are included into
the free variable lists computed by the renamer. More precisely, all
names needed by flattening are included in the names produced by
<code>RnEnv.getImplicitModuleFVs</code>. To avoid putting
flattening-dependent lists of names into the renamer, the module
<code>FlattenInfo</code> exports <code>namesNeededForFlattening</code>.
[FIXME: It might be worthwhile to document in the non-Flattening part of
the Commentary that the persistent compiler state is zapped after
desugaring and how the free variables determined by the renamer imply
which names are imported.]
<!-- hhmts start -->
Last modified: Tue Feb 12 01:44:21 EST 2002
<!-- hhmts end -->
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=ISO-8859-1">
<title>The GHC Commentary - Template Haskell</title>
<h1>The GHC Commentary - Template Haskell</h1>
The Template Haskell (TH) extension to GHC adds a meta-programming
facility in which all meta-level code is executed at compile time. The
design of this extension is detailed in "Template Meta-programming for
Haskell", Tim Sheard and Simon Peyton Jones, <a
SIGPLAN 2002 Haskell Workshop,</a> 2002. However, some of the details
changed after the paper was published.
<h2>Meta Sugar</h2>
The extra syntax of TH (quasi-quote brackets, splices, and reification)
is handled in the module <a
In particular, the function <code>dsBracket</code> desugars the four
types of quasi-quote brackets (<code>[|...|]</code>,
<code>[p|...|]</code>, <code>[d|...|]</code>, and <code>[t|...|]</code>)
and <code>dsReify</code> desugars the three types of reification
operations (<code>reifyType</code>, <code>reifyDecl</code>, and
<h3>Desugaring of Quasi-Quote Brackets</h3>
A term in quasi-quote brackets needs to be translated into Core code
that, when executed, yields a <em>representation</em> of that term in
the form of the abstract syntax trees defined in <a
Within <code>DsMeta</code>, this is achieved by four functions
corresponding to the four types of quasi-quote brackets:
<code>repE</code> (for <code>[|...|]</code>), <code>repP</code> (for
<code>[p|...|]</code>), <code>repTy</code> (for <code>[t|...|]</code>),
and <code>repTopDs</code> (for <code>[d|...|]</code>). All four of
these functions receive as an argument the GHC-internal Haskell AST of
the syntactic form that they quote (i.e., arguments of type <a
Name</code>, <a href=""><code>HsPat</code></a><code>.HsPat Name</code>,
Name</code>, and <a
Name</code>, respectively).
To increase the static type safety in <code>DsMeta</code>, the functions
constructing representations do not just return plain values of type <a
<code>.CoreExpr</code>; instead, <code>DsMeta</code> introduces a
parametrised type <code>Core</code> whose dummy type parameter indicates
the source-level type of the value computed by the corresponding Core
expression. All construction of Core fragments in <code>DsMeta</code>
is performed by smart constructors whose type signatures use the dummy
type parameter to constrain the contexts in which they are applicable.
For example, a function that builds a Core expression that evaluates to
a TH type representation, which has type
<code>Language.Haskell.TH.Syntax.Type</code>, would return a value of
Core Language.Haskell.TH.Syntax.Type</pre>
<h3>Desugaring of Reification Operators</h3>
The TH paper introduces four reification operators:
<code>reifyType</code>, <code>reifyDecl</code>,
<code>reifyFixity</code>, and <code>reifyLocn</code>. Of these,
currently (= 9 Nov 2002), only the former two are implemented.
The operator <code>reifyType</code> receives the name of a function or
data constructor as its argument and yields a representation of this
entity's type in the form of a value of type
<code>TH.Syntax.Type</code>. Similarly, <code>reifyDecl</code> receives
the name of a type and yields a representation of the type's declaration
as a value of type <code>TH.Syntax.Decl</code>. The name of the reified
entity is mapped to the GHC-internal representation of the entity by
using the function <code>lookupOcc</code> on the name.
<h3>Representing Binding Forms</h3>
Care needs to be taken when constructing TH representations of Haskell
terms that include binding forms, such as lambda abstractions or let
bindings. To avoid name clashes, fresh names need to be generated for
all defined identifiers. This is achieved via the routine
<code>DsMeta.mkGenSym</code>, which, given a <code>Name</code>, produces
a <code>Name</code> / <code>Id</code> pair (of type
<code>GenSymBind</code>) that associates the given <code>Name</code>
with a Core identifier that at runtime will be bound to a string that
contains the fresh name. Notice the two-level nature of this
arrangement. It is necessary, as the Core code that constructs the
Haskell term representation may be executed multiple types at runtime
and it must be ensured that different names are generated in each run.
Such fresh bindings need to be entered into the meta environment (of
type <a
which is part of the state (of type <code>DsMonad.DsEnv</code>)
maintained in the desugarer monad (of type <code>DsMonad.DsM</code>).
This is done using the function <code>DsMeta.addBinds</code>, which
extends the current environment by a list of <code>GenSymBind</code>s
and executes a subcomputation in this extended environment. Names can
be looked up in the meta environment by way of the functions
<code>DsMeta.lookupOcc</code> and <code>DsMeta.lookupBinder</code>; more
details about the difference between these two functions can be found in
the next subsection.
NB: <code>DsMeta</code> uses <code>mkGenSym</code> only when
representing terms that may be embedded into a context where names can
be shadowed. For example, a lambda abstraction embedded into an
expression can potentially shadow names defined in the context it is
being embedded into. In contrast, this can never be the case for
top-level declarations, such as data type declarations; hence, the type
variables that a parametric data type declaration abstracts over are not
being gensym'ed. As a result, variables in defining positions are
handled differently depending on the syntactic construct in which they
<h3>Binders Versus Occurrences</h3>
Name lookups in the meta environment of the desugarer use two functions
with slightly different behaviour, namely <code>DsMeta.lookupOcc</code>
and <code>lookupBinder</code>. The module <code>DsMeta</code> contains
the following explanation as to the difference of these functions:
When we desugar [d| data T = MkT |]
we want to get
Data "T" [] [Con "MkT" []] []
and *not*
Data "Foo:T" [] [Con "Foo:MkT" []] []
That is, the new data decl should fit into whatever new module it is
asked to fit in. We do *not* clone, though; no need for this:
Data "T79" ....
But if we see this:
data T = MkT
foo = reifyDecl T
then we must desugar to
foo = Data "Foo:T" [] [Con "Foo:MkT" []] []
So in repTopDs we bring the binders into scope with mkGenSyms and addBinds,
but in dsReify we do not. And we use lookupOcc, rather than lookupBinder
in repTyClD and repC.</pre>
This implies that <code>lookupOcc</code>, when it does not find the name
in the meta environment, uses the function <code>DsMeta.globalVar</code>
to construct the <em>original name</em> of the entity (cf. the TH paper
for more details regarding original names). This name uniquely
identifies the entity in the whole program and is in scope
<em>independent</em> of whether the user name of the same entity is in
scope or not (i.e., it may be defined in a different module without
being explicitly imported) and has the form &lt;module&gt;:&lt;name&gt;.
<strong>NB:</strong> Incidentally, the current implementation of this
mechanisms facilitates breaking any abstraction barrier.
<h3>Known-key Names for Template Haskell</h3>
During the construction of representations, the desugarer needs to use a
large number of functions defined in the library
<code>Language.Haskell.TH.Syntax</code>. The names of these functions
need to be made available to the compiler in the way outlined <a
href="../the-beast/prelude.html">Primitives and the Prelude.</a>
Unfortunately, any change to <a
triggers a significant amount of recompilation. Hence, the names needed
for TH are defined in <code>DsMeta</code> instead (at the end of the
module). All library functions needed by TH are contained in the name
set <code>DsMeta.templateHaskellNames</code>.
<!-- hhmts start -->
Last modified: Wed Nov 13 18:01:48 EST 2002
<!-- hhmts end -->
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=ISO-8859-1">
<title>The GHC Commentary - Feedback</title>
<a href="">I</a> welcome any feedback on the
material and in particular would appreciated comments on which parts of
the document are incomprehensible or miss explanation -- e.g., due to
the use of GHC speak that is explained nowhere (words like infotable or
so). Moreover, I would be interested to know which areas of GHC you
would like to see covered here.
For the moment is probably best if feedback is directed to
However, if there is sufficient interest, we might consider setting up a
mailing list.
<!-- hhmts start -->
Last modified: Wed Aug 8 00:11:42 EST 2001
<!-- hhmts end -->
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=ISO-8859-1">
<title>The GHC Commentary - Outline of the Genesis</title>
<h1>The GHC Commentary - Outline of the Genesis</h1>
Building GHC happens in two stages: First you have to prepare the tree
with <code>make boot</code>; and second, you build the compiler and
associated libraries with <code>make all</code>. The <code>boot</code>
stage builds some tools used during the main build process, generates
parsers and other pre-computed source, and finally computes dependency
information. There is considerable detail on the build process in GHC's
href="">Building Guide.</a>
<h4>Debugging the Beast</h4>
If you are hacking the compiler or like to play with unstable
development versions, chances are that the compiler someday just crashes
on you. Then, it is a good idea to load the <code>core</code> into
<code>gdb</code> as usual, but unfortunately there is usually not too
much useful information.
The next step, then, is somewhat tedious. You should build a compiler
producing programs with a runtime system that has debugging turned on
and use that to build the crashing compiler. There are many sanity
checks in the RTS, which may detect inconsistency before they lead to a
crash and you may include more debugging information, which helps
<code>gdb.</code> For a RTS with debugging turned on, add the following
to <code></code> (see also the comment in