|
# Shared Libraries: distribution and build-system issues
|
|
# Shared Libraries
|
|
|
|
|
|
|
|
|
|
This page is for discussing and documenting our strategy for
|
|
This page provides an introduction to shared libraries in general and specifically how they are supported and implemented in GHC.
|
|
|
|
|
|
- How shared libraries are found
|
|
|
|
- How the build system works
|
|
|
|
- How distributions (of GHC and programs built by GHC) work
|
|
|
|
- Issues that affect Cabal
|
|
|
|
|
|
|
|
## Goals/Scenarios
|
|
More detailed topics:
|
|
|
|
|
|
|
|
- [SharedLibraries/Management](shared-libraries/management): how we organise and manage shared libs
|
|
|
|
- [SharedLibraries/PlatformSupport](shared-libraries/platform-support): status of shared lib support on various platforms
|
|
|
|
- [Commentary/PositionIndependentCode](commentary/position-independent-code): how `ghc -fPIC` works
|
|
|
|
|
|
First of all, we take it as a given that a normal GHC installation will be a good citizen on its host platform: shared libraries will go in the standard locations, and we'll use the system's normal method for finding them at link time and runtime. Windows is an exception: there is no standard location for installing shared libraries on Windows.
|
|
## What shared libs are
|
|
|
|
|
|
|
|
[ Shared libraries](http://en.wikipedia.org/wiki/Shared_libraries) (sometimes called dynamic libraries) are an alternative way of organising pre-compiled code compared to traditional static libraries. The key difference is that with shared libs, linking programs against library functions takes place when the program is run rather than when the program is built and installed.
|
|
|
|
|
|
So that we can support having multiple versions of GHC installed, shared libraries will have the GHC version number embedded, e.g. `libHSnetwork-1.1-ghc6.6.1.so`.
|
|
|
|
|
|
|
|
|
|
All modern operating systems use shared libs. For system libraries they have a particular advantage. They allow the library to be upgraded separately from the programs that use the libs. However this requires preserving an ABI. They also allow a single copy of code to be shared in memory between several programs that use it. For common system libraries this can be a significant saving.
|
|
|
|
|
|
Here is what else we'd like to do:
|
|
## The three major shared libs systems
|
|
|
|
|
|
1. Support installing GHC outside of the standard location (e.g. in a home directory), and build
|
|
|
|
binaries using that installation. Multiple such installations should be supported.
|
|
|
|
1. We need to build a distribution that supports choosing the install location at install time, for
|
|
|
|
use in (1).
|
|
|
|
1. Binaries that are built as part of the GHC build (e.g. stage2/ghc-inplace) need to run from
|
|
|
|
the build tree.
|
|
|
|
1. Cabal needs to build libraries that can be installed in the system location or elsewhere.
|
|
|
|
|
|
|
|
# Proposed strategies
|
|
There are three systems in common use:
|
|
|
|
|
|
## 1. Static linking
|
|
- **ELF** ([ Executable and Linkable Format](http://en.wikipedia.org/wiki/Executable_and_Linkable_Format)) is used on all modern Unix systems (except MacOS X), in particular it is used on Linux, Solaris and the BSDs.
|
|
|
|
- **PE** ([ Portable Executable](http://en.wikipedia.org/wiki/Portable_Executable)) format is used on Windows.
|
|
|
|
- **Mach-O** ([ Mach object](http://en.wikipedia.org/wiki/Mach-O)) is the format used on Mac OS X.
|
|
|
|
|
|
|
|
|
|
(1,2) Installations of GHC that are not in the standard locations use static linking and come with static libraries only.
|
|
On each system, the same format is used for executables, shared libraries and intermediate object files. Each system uses their own file extension for shared libraries:
|
|
|
|
|
|
|
|
<table><tr><th> System </th>
|
|
|
|
<th> executable extension </th>
|
|
|
|
<th> shared library extension
|
|
|
|
</th></tr>
|
|
|
|
<tr><th> ELF </th>
|
|
|
|
<th> (no extension) </th>
|
|
|
|
<th>`.so`</th></tr>
|
|
|
|
<tr><th> PE </th>
|
|
|
|
<th>`.exe`</th>
|
|
|
|
<th>`.dll`</th></tr>
|
|
|
|
<tr><th> Mach-O </th>
|
|
|
|
<th> (no extension) </th>
|
|
|
|
<th>`.dylib`</th></tr></table>
|
|
|
|
|
|
(3) stage2/ghc-inplace is linked statically.
|
|
|
|
|
|
|
|
|
|
Unfortunately, while static linking is relatively uncomplicated and similar between systems, shared libraries are implemented rather differently between different operating systems and pose somewhat of a management headache.
|
|
|
|
|
|
(4) Cabal packages installed outside the system locations are static only.
|
|
## Background reading
|
|
|
|
|
|
|
|
|
|
This is attractive, but there are some drawbacks:
|
|
An excellent technical introduction to ELF shared libraries is [ How To Write Shared Libraries](http://people.redhat.com/drepper/dsohowto.pdf) by Ulrich Drepper (author of glibc).
|
|
|
|
|
|
- we still need to build a distribution that uses shared libs. Presumably we have to build both
|
|
## Why we care about shared libraries
|
|
shared and static libs then.
|
|
|
|
- the testsuite needs to build binaries against the shared libs for testing, without installing GHC.
|
|
|
|
- we want the GHC binary in a shared-library installation to be dynamically linked, not statically linked.
|
|
|
|
- if there are some static-only libraries on the system, then all packages must have static versions,
|
|
|
|
because dynamic linking is all-or-nothing in GHC.
|
|
|
|
- This approach doesn't address Windows
|
|
|
|
|
|
|
|
## 2. Dynamic linking
|
|
|
|
|
|
|
|
|
|
There are several reasons we care.
|
|
|
|
|
|
The first plan was this:
|
|
|
|
|
|
|
|
[ http://www.haskell.org/pipermail/glasgow-haskell-users/2007-June/012740.html](http://www.haskell.org/pipermail/glasgow-haskell-users/2007-June/012740.html)
|
|
The greatest advantage is that it enables us to make plugins for other programs. There are loads of examples of this, think of plugins for things like vim, gimp, postgres, apache. On Windows if you want to make a COM or .NET component then it usually has to be as a shared library (a .dll file).
|
|
|
|
|
|
|
|
|
|
It has since been pointed out that `LD_LIBRARY_PATH` overrides `-rpath` on some platforms (see below). This might cause some difficulties (or not?).
|
|
Similar to plugins, shared libraries have become a common way of composing large systems. Each shared library can be written in a different language. Compared to static libraries, shared libraries are typically more self-contained. The ability to produce nice self-contained shared libraries from Haskell code would simply the integration of Haskell code into larger existing systems.
|
|
|
|
|
|
|
|
|
|
Assuming we can fix the locations of shared libraries at link time (eg. with -rpath), then:
|
|
A somewhat superficial reason is that it makes your “Hello World” program much smaller because it doesn’t have to include a complete copy of the runtime system and half of the base library. It’s true that in most circumstances disk space is cheap, but if you’ve got some corporate shared storage that’s replicated and meticulously backed-up and if each of your 100 “small” Haskell plugins is actually 10MB big, then the disk space does not look quite so cheap.
|
|
|
|
|
|
1. Installations of GHC outside the system default location hardwire the locations of shared libraries
|
|
|
|
into the binaries they build. (hence such binaries cannot be distributed; this is a drawback)
|
|
|
|
1. Binaries in the distribution must not have rpaths. We should use wrapper scripts that set
|
|
|
|
`LD_LIBRARY_PATH` instead.
|
|
|
|
1. Binaries in the build tree need `LD_LIBRARY_PATH` wrappers.
|
|
|
|
1. A Cabal package may install a shared library outside the standard location, but when linking to
|
|
|
|
it we must do the equivalent of adding -rpath to point to its location.
|
|
|
|
|
|
|
|
|
|
Using shared libraries also makes things a bit easier for Haskell applications that want to do dynamic code loading. For example GHCi itself currently has to load two copies of the base package, the one that is statically linked with and another copy that it loads dynamically. With shared libraries it would just end up with another reference to the same copy of the single shared base library.
|
|
|
|
|
|
ToDo: Windows?
|
|
|
|
|
|
|
|
## 3. libtool
|
|
Shared libs also completely eliminates the need for the “split objs” hack that GHC uses to reduce the size of statically linked programs. This should make our link times a bit quicker.
|
|
|
|
|
|
|
|
|
|
libtool hides the building of shared and static libraries and executables behind a single simple command-line interface. It hides the details of how to build executables against uninstalled shared libraries, and how to install those executables, on multiple platforms.
|
|
Note that we have not mentioned the two major advantages that shared libraries were originally developed for, namely saving memory at runtime (when several programs use the same lib) and making it possible to upgrade libraries without touching the programs that use them. These advantages are more significant in core operating system libraries. C code can be made to follow a stable ABI where as historically this has not been a priority in Haskell implementations (though this may change). Similarly, there are not too many systems yet where having multiple copies of the RTS and base libraries in memory at once is a significant problem. Again, this may change if people choose to target memory-constrained systems.
|
|
|
|
|
|
|
|
## TODO
|
|
|
|
|
|
When building an object file for a library, libtool builds both the PIC and non-PIC versions.
|
|
|
|
|
|
|
|
|
|
More stuff to explain:
|
|
|
|
|
|
When building a library, libtool builds both the shared and static version, and remembers where the shared version will be installed later (you have to supply this path when building the library).
|
|
- Position independent code, what it is, why we need it on some systems
|
|
|
|
- relationship between ghc flags -dynamic, -shared and -fPIC
|
|
|
|
- difference between C and ghc in compiling for shared libs
|
|
When building an executable against shared libraries, libtool creates an executable ready for installation (in `.libs`): this either has no paths embedded (if the shared libs are to be installed in system locations), or with appropriate `-rpath` settings pointing to the locations that the shared libs are to be installed. `libtool` also creates a script for running the program in-place. The script relinks the executable against uninstalled shared libraries (using `-rpath` on Linux) on demand, caches the resulting executable in `.libs`.
|
|
- peculiarities of ELF and PE |
|
|
|
|
|
# Platform support for locating shared libraries
|
|
|
|
|
|
|
|
|
|
|
|
The following analysis is mostly from Reilly Hayes on the cvs-ghc mailing list.
|
|
|
|
|
|
|
|
## On Linux
|
|
|
|
|
|
|
|
|
|
|
|
An ELF executable can have an embedded list of paths to search for dynamic libraries (the DT_RPATH entry). This can be set by using -rpath with ld. DT_RPATH is deprecated. This list applies to all shared libraries used by the executable (it is not per shared library). There is no default value placed in the DT_RPATH entry. You must use -rpath to set it.
|
|
|
|
|
|
|
|
|
|
|
|
There is a new entry, DT_RUNPATH. DT_RUNPATH works similarly to DT_RPATH. However, when it is set, DT_RPATH is ignored. DT_RUNPATH is also set using -rpath, but you must use the --enable-new-dtags switch as well.
|
|
|
|
|
|
|
|
|
|
|
|
When looking for a shared library, the dynamic linker(ld.so) checks the paths listed in DT_RPATH (unless DT_RUNPATH Is set) , the paths listed in the environment variable LD_LIBRARY_PATH, the paths listed in DT_RUNPATH, the libraries listed in /etc/ld.so.cache, and finally /usr/lib and /lib. It checks in that order and takes the first library found. At least on my linux box, LD_LIBRARY_PATH does NOT override the paths in DT_RPATH even though the documentation implies that it does. LD_LIBRARY_PATH does override DT_RUNPATH.
|
|
|
|
|
|
|
|
|
|
|
|
You CAN override the search path embedded using DT_RPATH by using the LD_PRELOAD environment variable. This variable contains a \*whitespace-separated\* list of libraries (not directories to search) to load prior to the search process. The listed libraries are loaded whether or not they are needed to resolve a dependency in the executable.
|
|
|
|
|
|
|
|
|
|
|
|
Finally, an ELF shared library can also have a DT_RPATH entry. This only impacts the search for shared libraries that are dependencies of the shared library and not the executable. As with the DT_RPATH entry in an ELF executable, this is not overridden by LD_LIBRARY_PATH but can be overridden using LD_PRELOAD as above.
|
|
|
|
|
|
|
|
## On Mac OS X
|
|
|
|
|
|
|
|
|
|
|
|
A Mach-O executable can embed the full path name for each shared library (as well as rules for acceptable substitutes). This is called the "install name" for the library and it is included by default when building an executable. The install name for the library is NOT based on where the static linker (ld) found the library when the executable was built. The static linker (ld) extracts the install name from the shared library when building the executable. The install name of the shared library is set when building the shared library. When you build a shared library you should know where the library is going to be installed so that the install name is set correctly.
|
|
|
|
|
|
|
|
|
|
|
|
When looking for shared libraries, the dynamic linker (dyld) first scans the directories in DYLD_LIBRARY_PATH, then checks the location in the install name (which is per library), and finally checks the standard locations.
|
|
|
|
|
|
|
|
|
|
|
|
DYLD_LIBRARY_PATH successfully overrides the the path embedded in the executable.
|
|
|
|
|
|
|
|
|
|
|
|
Caveat 1: LD_LIBRARY_PATH has no runtime impact, but it does impact where the static linker looks for share libraries. It looks first in the directories specified using -L, the the directories in LD_LIBRARY_PATH, and finally in /lib, /usr/lib, & /usr/local/lib. This is particularly confusing because many configure scripts seem to ignore LD_LIBRARY_PATH and you can get inconsistent results from configure and gcc/ld on whether a library is present.
|
|
|
|
|
|
|
|
|
|
|
|
Caveat 2: Mac OS X has a set of compiler/linker switches for dealing with Frameworks (packages of shared libraries and include files). These are installed outside the typical \*nix directory structure. These switches act like -I (to gcc) and -L (to ld). If you end up totally confused about where to find something, read up on this. The OpenGL and OpenAL headers and libraries are in Frameworks, for example.
|
|
|
|
|
|
|
|
## On Windows
|
|
|
|
|
|
|
|
|
|
|
|
ToDo: link to the MSDN page about how DLLs are found, and the details about manifests. Manifests provide a way to do rpath-like things, I think.
|
|
|
|
|
|
|
|
## Conclusions
|
|
|
|
|
|
|
|
|
|
|
|
For Mac: The -rpath switch is not available on Mac OS X because it is superfluous. The default behavior of embedding a location for each individual shared library is at least as good. Cabal (and the GHC build process) should use their knowledge of the ultimate install location to set the install name when shared libraries are built. In-place compilation can override this with DYLD_LIBRARY_PATH
|
|
|
|
|
|
|
|
|
|
|
|
For Linux: On linux, we should be sure to use the --enable-new-dtags switch if we use -rpath. Otherwise we risk having paths that can't be overridden by LD_LIBRARY_PATH. |
|
|