Skip to content
Snippets Groups Projects
Forked from Glasgow Haskell Compiler / GHC
5377 commits behind the upstream repository.
  • sheaf's avatar
    5c873124
    Implement -jsem: parallelism controlled by semaphores · 5c873124
    sheaf authored and Marge Bot's avatar Marge Bot committed
    See https://github.com/ghc-proposals/ghc-proposals/pull/540/ for a
    complete description for the motivation for this feature.
    
    The `-jsem` option allows a build tool to pass a semaphore to GHC which
    GHC can use in order to control how much parallelism it requests.
    
    GHC itself acts as a client in the GHC jobserver protocol.
    
    ```
    GHC Jobserver Protocol
    ~~~~~~~~~~~~~~~~~~~~~~
    
    This proposal introduces the GHC Jobserver Protocol. This protocol allows
    a server to dynamically invoke many instances of a client process,
    while restricting all of those instances to use no more than <n> capabilities.
    This is achieved by coordination over a system semaphore (either a POSIX
    semaphore [6]_  in the case of Linux and Darwin, or a Win32 semaphore [7]_
    in the case of Windows platforms).
    
    There are two kinds of participants in the GHC Jobserver protocol:
    
    - The *jobserver* creates a system semaphore with a certain number of
      available tokens.
    
      Each time the jobserver wants to spawn a new jobclient subprocess, it **must**
      first acquire a single token from the semaphore, before spawning
      the subprocess. This token **must** be released once the subprocess terminates.
    
      Once work is finished, the jobserver **must** destroy the semaphore it created.
    
    - A *jobclient* is a subprocess spawned by the jobserver or another jobclient.
    
      Each jobclient starts with one available token (its *implicit token*,
      which was acquired by the parent which spawned it), and can request more
      tokens through the Jobserver Protocol by waiting on the semaphore.
    
      Each time a jobclient wants to spawn a new jobclient subprocess, it **must**
      pass on a single token to the child jobclient. This token can either be the
      jobclient's implicit token, or another token which the jobclient acquired
      from the semaphore.
    
      Each jobclient **must** release exactly as many tokens as it has acquired from
      the semaphore (this does not include the implicit tokens).
    ```
    
    Build tools such as cabal act as jobservers in the protocol and are
    responsibile for correctly creating, cleaning up and managing the
    semaphore.
    
    Adds a new submodule (semaphore-compat) for managing and interacting
    with semaphores in a cross-platform way.
    
    Fixes #19349
    5c873124
    History
    Implement -jsem: parallelism controlled by semaphores
    sheaf authored and Marge Bot's avatar Marge Bot committed
    See https://github.com/ghc-proposals/ghc-proposals/pull/540/ for a
    complete description for the motivation for this feature.
    
    The `-jsem` option allows a build tool to pass a semaphore to GHC which
    GHC can use in order to control how much parallelism it requests.
    
    GHC itself acts as a client in the GHC jobserver protocol.
    
    ```
    GHC Jobserver Protocol
    ~~~~~~~~~~~~~~~~~~~~~~
    
    This proposal introduces the GHC Jobserver Protocol. This protocol allows
    a server to dynamically invoke many instances of a client process,
    while restricting all of those instances to use no more than <n> capabilities.
    This is achieved by coordination over a system semaphore (either a POSIX
    semaphore [6]_  in the case of Linux and Darwin, or a Win32 semaphore [7]_
    in the case of Windows platforms).
    
    There are two kinds of participants in the GHC Jobserver protocol:
    
    - The *jobserver* creates a system semaphore with a certain number of
      available tokens.
    
      Each time the jobserver wants to spawn a new jobclient subprocess, it **must**
      first acquire a single token from the semaphore, before spawning
      the subprocess. This token **must** be released once the subprocess terminates.
    
      Once work is finished, the jobserver **must** destroy the semaphore it created.
    
    - A *jobclient* is a subprocess spawned by the jobserver or another jobclient.
    
      Each jobclient starts with one available token (its *implicit token*,
      which was acquired by the parent which spawned it), and can request more
      tokens through the Jobserver Protocol by waiting on the semaphore.
    
      Each time a jobclient wants to spawn a new jobclient subprocess, it **must**
      pass on a single token to the child jobclient. This token can either be the
      jobclient's implicit token, or another token which the jobclient acquired
      from the semaphore.
    
      Each jobclient **must** release exactly as many tokens as it has acquired from
      the semaphore (this does not include the implicit tokens).
    ```
    
    Build tools such as cabal act as jobservers in the protocol and are
    responsibile for correctly creating, cleaning up and managing the
    semaphore.
    
    Adds a new submodule (semaphore-compat) for managing and interacting
    with semaphores in a cross-platform way.
    
    Fixes #19349
Code owners
Assign users and groups as approvers for specific file changes. Learn more.
packages 4.96 KiB
# Despite the name "packages", this file contains the master list of
# the *repositories* that make up GHC. It is parsed by
# * boot
# * rules/foreachLibrary.mk
#
# Some of this information is duplicated elsewhere in the build system:
#    See #3896
# In particular when adding libraries to this file, you also need to add
# the library to the SUBDIRS variable in libraries/Makefile so that they
# actually get built
#
# The repos are of several kinds:
#    - The main GHC source repo
#    - Each boot package lives in a repo
#    - Haddock and hsc2hs are applications, built on top of GHC,
#        and in turn needed to bootstrap GHC
#    - ghc-tarballs is need to build GHC
#    - nofib is an optional helpers
#
# The format of the lines in this file is:
#   localpath  tag  remotepath  upstreamurl
# where
#   * 'localpath' is where to put the repository in a checked out tree.
#   * 'remotepath' is where the repository is in the central repository.
#     It is - for submodules.
#   * 'upstreamurl' is the upstream Git repo location for packages
#     maintained outside of GHC HQ. Repositories which are hosted on
#     GitHub and GHC developers are granted push-rights for are denoted by
#     being specified with the `ssh://` scheme. Thus, `https://`
#     repo urls denote read-only access.
#   * 'tag', in combination with the variable BUILD_EXTRA_PKGS,
#     determines which packages are build by default.
#
# Lines that start with a '#' are comments.
#
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# localpath                  tag         remotepath                      upstreamurl
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ghc-tarballs                 windows     ghc-tarballs.git                -
libffi-tarballs              -           -                               -
utils/hsc2hs                 -           -                               ssh://git@github.com/haskell/hsc2hs.git
utils/haddock                -           -                               ssh://git@github.com/haskell/haddock.git
libraries/array              -           -                               -
libraries/binary             -           -                               https://github.com/kolmodin/binary.git
libraries/bytestring         -           -                               https://github.com/haskell/bytestring.git
libraries/Cabal              -           -                               https://github.com/haskell/cabal.git
libraries/ghc-compact        -           -                               -
libraries/ghc-heap           -           -                               -
libraries/containers         -           -                               https://github.com/haskell/containers.git
libraries/deepseq            -           -                               ssh://git@github.com/haskell/deepseq.git
libraries/directory          -           -                               ssh://git@github.com/haskell/directory.git
libraries/filepath           -           -                               ssh://git@github.com/haskell/filepath.git
libraries/haskeline          -           -                               https://github.com/judah/haskeline.git
libraries/hpc                -           -                               -
libraries/mtl                -           -                               https://github.com/haskell/mtl.git
libraries/parsec             -           -                               https://github.com/haskell/parsec.git
libraries/pretty             -           -                               https://github.com/haskell/pretty.git
libraries/process            -           -                               ssh://git@github.com/haskell/process.git
libraries/terminfo           -           -                               https://github.com/judah/terminfo.git
libraries/text               -           -                               https://github.com/haskell/text.git
libraries/time               -           -                               https://github.com/haskell/time.git
libraries/transformers       -           -                               https://git.haskell.org/darcs-mirrors/transformers.git
libraries/unix               -           -                               ssh://git@github.com/haskell/unix.git
libraries/Win32              -           -                               https://github.com/haskell/win32.git
libraries/xhtml              -           -                               https://github.com/haskell/xhtml.git
libraries/exceptions         -           -                               https://github.com/ekmett/exceptions.git
nofib                        nofib       -                               -
libraries/semaphore-compat   -           -                               -
libraries/stm                -           -                               ssh://git@github.com/haskell/stm.git
.                            -           ghc.git                         -