...
 
Commits (38)
  • Sebastian Graf's avatar
    Fix parsing of maximum residency in runstdtest · f3521319
    Sebastian Graf authored
    Summary:
    `runstdtest` switched from `-S` to `-s` output a while ago.
    That broke parsing of maximum and average residency numbers.
    
    This commit makes sure that at least maximum residency is parsed
    correctly from `-s` output, while leaving the logic for `-S` output.
    
    Reviewers: simonmar, bgamari, simonpj, osa1, AndreasK, O26 nofib
    
    Reviewed By: bgamari
    
    GHC Trac Issues: #16003
    
    Differential Revision: https://phabricator.haskell.org/D5418
    f3521319
  • Fangyi Zhou's avatar
    Fix some broken links (#15733) · 44fc21a3
    Fangyi Zhou authored
    Summary:
    For links in subpackages as well.
    https://phabricator.haskell.org/D5257
    
    Test Plan: Manually verify links
    
    Reviewers: mpickering, bgamari, O26 nofib, osa1
    
    Reviewed By: osa1
    
    GHC Trac Issues: #15733
    
    Differential Revision: https://phabricator.haskell.org/D5260
    44fc21a3
  • Sebastian Graf's avatar
    Compare results of compress by hashing · 042cf0be
    Sebastian Graf authored
    Summary:
    We don't want the result in the repo as it's a sizeable binary file that
    doesn't compress well.
    
    Storing the output file in the repository becomes infeasible for large
    inputs. There are two possible remedies:
    
    1. Generate the result files during `make boot` (Phab:D5426). We
       discovered some drawbacks (like missing dependency files to build
       `compress` during boot) to this approach which make it infeasible.
    2. Shrink the output files, for example by hashing the string that we would
       normally output and compare that instead.
    
    This patch implements the second alternative. This somewhat distorts the
    runtime profile, so we might want to consider doing hashing within the
    benchmark runner in the future.
    
    Test Plan: make boot
    
    Reviewers: AndreasK, nomeata, O26 nofib, osa1
    
    Reviewed By: AndreasK, osa1
    
    Subscribers: osa1
    
    Differential Revision: https://phabricator.haskell.org/D5469
    042cf0be
  • Sebastian Graf's avatar
    CLEAN_FILES for `reverse-complement` · 8b3c9596
    Sebastian Graf authored
    Summary:
    `reverse-complement` didn't have its `CLEAN_FILES` set.
    
    That led to incomplete rebuilds of binaries and changes in
    i.e. `FAST_OPTS` taking no effect, despite a prior `make clean`.
    
    Reviewers: osa1, AndreasK, bgamari, O26 nofib
    
    Differential Revision: https://phabricator.haskell.org/D5467
    8b3c9596
  • Greif's avatar
    Typofix · 44bff05d
    Greif authored
    44bff05d
  • Ben Gamari's avatar
    Add continuous integration support · e1869fd3
    Ben Gamari authored
    e1869fd3
  • Sebastian Graf's avatar
    Compare output of compress2 by hashing · c3acdcc6
    Sebastian Graf authored
    This should fix the build in the same way as !2.
    c3acdcc6
  • Sebastian Graf's avatar
    Replace mentions of {-# STRICT -#} with bangs in compress2 · 95c1dccb
    Sebastian Graf authored
    The pragma hasn't been recognised for a long time.
    Replacing it with bangs will change program semantics to how it was
    intended. The perf CI build is currently broken anyway, so this
    shouldn't skew numbers too much.
    95c1dccb
  • Andreas Klebinger's avatar
    Move required language extensions into pragmas for shootout. · c985746f
    Andreas Klebinger authored
    Summary: That way they are easier to compile with plain calls to ghc.
    
    Test Plan: make clean && make boot && make
    
    Reviewers: O26 nofib, bgamari
    
    Reviewed By: bgamari
    
    Subscribers: bgamari
    
    Differential Revision: https://phabricator.haskell.org/D5437
    c985746f
  • Sebastian Graf's avatar
    Disable timer-based context switches · e2d614e4
    Sebastian Graf authored
    Summary:
    In the past, we repeatedly had problems with non-deterministic allocations due to stack squeezing during context switches (#4450, #8611).
    This patch adds `+RTS -V0 -RTS` as extra `RUNTEST_OPTS` for every single-threaded benchmark.
    
    Is this the right place to add the flags? Should we also do this for all the other benchmarks?
    
    Reviewers: simonmar, osa1, nomeata, O26 nofib
    
    GHC Trac Issues: #8611
    
    Differential Revision: https://phabricator.haskell.org/D5470
    e2d614e4
  • Sebastian Graf's avatar
    Stabilise benchmarks wrt. GC · 8632268a
    Sebastian Graf authored
    Summary:
    This is due to #15999, a follow-up on #5793 and #15357 and changes all
    benchmarks, some of them (i.e. `wheel-sieve1`, `awards`) rather drastically.
    
    The general plan is outlined in #15999: Identify GC-sensitive benchmarks by
    looking at how productivity rates change over different nursery sizes and
    iterate `main` of these benchmarks often enough for the non-monotony and
    discontinuities to go away.
    
    I was paying attention that the benchmarked logic is actually run $n times more
    often, rather than just benchmarking IO operations printing the result of CAFs.
    
    When I found benchmarks with insignificant runtime (#15357), I made sure that
    parameters/input files were adjusted so that runtime of the different modes
    fall within the ranges proposed in
    https://ghc.haskell.org/trac/ghc/ticket/15357#comment:4
    
    - fast: 0.1-0.2s
    - norm: 1-2s
    - slow: 5-10s
    
    This is what I did:
    
    - Stabilise bernoulli
    - Stabilise digits-of-e1
    - Stabilise digits-of-e2
    - Stabilise gen_regexp
    - Adjust running time of integrate
    - Adjust running time of kahan
    - Stabilise paraffins
    - Stabilise primes
    - Adjust running time of rfib
    - Adjust running time of tak
    - Stabilise wheel-sieve1
    - Stabilise wheel-sieve2
    - Adjust running time of x2n1
    - Adjust running time of ansi
    - Adjust running time of atom
    - Make awards benchmark something other than IO
    - Adjust running time of banner
    - Stabilise boyer
    - Adjust running time of boyer2
    - Adjust running time of queens
    - Adjust running time of calendar
    - Adjust runtime of cichelli
    - Stabilise circsim
    - Stabilise clausify
    - Stabilise constraints with moderate success
    - Adjust running time of cryptarithm1
    - Adjust running time of cryptarythm2
    - Adjust running time of cse
    - Adjust running time of eliza
    - Adjust running time of exact-reals
    - Adjust running time of expert
    - Stabilise fft2
    - Stabilise fibheaps
    - Stabilise fish
    - Adjust running time for gcd
    - Stabilise comp_lab_zift
    - Stabilise event
    - Stabilise fft
    - Stabilise genfft
    - Stabilise ida
    - Adjust running time for listcompr
    - Adjust running time for listcopy
    - Adjust running time of nucleic2
    - Attempt to stabilise parstof
    - Stabilise sched
    - Stabilise solid
    - Adjust running time of transform
    - Adjust running time of typecheck
    - Stabilise wang
    - Stabilise wave4main
    - Adjust running time of integer
    - Adjust running time of knights
    - Stabilise lambda
    - Stabilise lcss
    - Stabilise life
    - Stabilise mandel
    - Stabilise mandel2
    - Adjust running time of mate
    - Stabilise minimax
    - Adjust running time of multiplier
    - Adjust running time of para
    - Stabilise power
    - Adjust running time of primetest
    - Stabilise puzzle with mild success
    - Adjust running time for rewrite
    - Stabilise simple with mild success
    - Stabilise sorting
    - Stabilise sphere
    - Stabilise treejoin
    - Stabilise anna
    - Stabilise bspt
    - Stabilise cacheprof
    - Stablise compress
    - Stablise compress2
    - Stabilise fem
    - Adjust running time of fluid
    - Stabilise fulsom
    - Stabilise gamteb
    - Stabilise gg
    - Stabilise grep
    - Adjust running time of hidden
    - Stabilise hpg
    - Stabilise infer
    - Stabilise lift
    - Stabilise linear
    - Attempt to stabilise maillist
    - Stabilise mkhprog
    - Stabilise parser
    - Stabilise pic
    - Stabilise prolog
    - Attempt to stabilise reptile
    - Adjust running time of rsa
    - Adjust running time of scs
    - Stabilise symalg
    - Stabilise veritas
    - Stabilise binary-trees
    - Adjust running time of fasta
    - Adjust running time of k-nucleotide
    - Adjust running time of pidigits
    - Adjust running time of reverse-complement
    - Adjust running time of spectral-norm
    - Adjust running time of fannkuch-redux
    - Adjust running time for n-body
    
    Problematic benchmarks:
    
    - `last-piece`: Unclear how to stabilise. Runs for 300ms and I can't make up smaller inputs because I don't understand what it does.
    - `pretty`: It's just much too small to be relevant at all. Maybe we want to get rid of this one?
    - `scc`: Same as `pretty`. The input graph for which SCC analysis is done is much too small and I can't find good directed example graphs on the internet.
    - `secretary`: Apparently this needs `-package random` and consequently hasn't been run for a long time.
    - `simple`: Same as `last-piece`. Decent runtime (70ms), but it's unstable and I see no way to iterate it ~100 times in fast mode.
    - `eff`: Every benchmark is problematic here. Not from the point of view of allocations, but because the actual logic is vacuous. IMO, these should be performance tests, not actual benchmarks. Alternatively, write an actual application that makes use of algebraic effects.
    - `maillist`: Too trivial. It's just String/list manipulation, not representative of any Haskell code we would write today (no use of base library functions which could be fused, uses String instead of Text). It's only 75 loc according to `cloc`, that's not a `real` application.
    
    Reviewers: simonpj, simonmar, bgamari, AndreasK, osa1, alpmestan, O26 nofib
    
    GHC Trac Issues: #15999
    
    Differential Revision: https://phabricator.haskell.org/D5438
    8632268a
  • Abhiroop Sarkar's avatar
    Update README.md · e62a49d3
    Abhiroop Sarkar authored
    e62a49d3
  • Sebastian Graf's avatar
    Make `git clean -nxd` silent after `make distclean` · cc8fc601
    Sebastian Graf authored
    Also check that invariant in CI: After a `make distclean`
    `git clean -nxd` should find no files.
    
    The hope is that this catches cases were a `make clean` would forget to
    list a `.gitignore`d file that should either be registered as a
    `(DIST_)CLEAN_FILES` or should be unignored and committed into the
    repository.
    cc8fc601
  • Sebastian Graf's avatar
    Re-enable cacheprof · 6cfa0980
    Sebastian Graf authored
    6cfa0980
  • Sebastian Graf's avatar
  • Sebastian Graf's avatar
    Stabilise gen_regexp again · 271dc00f
    Sebastian Graf authored
    While investigating a 5% runtime regression, I realised that
    `gen_regexp` wasn't sufficiently stable wrt. GC parameterisations.
    
    This is now rectified by iterating 500 instead of 100 times.
    271dc00f
  • Sebastian Graf's avatar
    b760674c
  • Sebastian Graf's avatar
    Add sgraf812 and bgamari as CODEOWNERS · 7a702cc1
    Sebastian Graf authored
    [skip ci]
    7a702cc1
  • Oleg Grenrus's avatar
    Build nofib-analyse with cabal v2-build · 807930f1
    Oleg Grenrus authored
    807930f1
  • Oleg Grenrus's avatar
    Check versions in CI · 8947fb20
    Oleg Grenrus authored
    8947fb20
  • Oleg Grenrus's avatar
    Update .gitlab-ci.yml · 7815bbba
    Oleg Grenrus authored
    7815bbba
  • Oleg Grenrus's avatar
    mostyclean dist and .ghc.environment · 291463e5
    Oleg Grenrus authored
    291463e5
  • Oleg Grenrus's avatar
    Move DOCKER_REV to commit in master · 0675e131
    Oleg Grenrus authored
    0675e131
  • Sebastian Graf's avatar
    9c2966dc
  • Andreas Klebinger's avatar
    Add core of the dom-lt package to nofib. · a850e654
    Andreas Klebinger authored
    Of interest is in particular that the code uses -XStrict.
    a850e654
  • Andreas Klebinger's avatar
    Add stdout files for dom-lt. · 06f6834f
    Andreas Klebinger authored
    06f6834f
  • Andreas Klebinger's avatar
    Place dom-lt under spectral. · 347e0f00
    Andreas Klebinger authored
    347e0f00
  • Andreas Klebinger's avatar
  • Ben Gamari's avatar
    gitlab-ci: Set tags properly · 4d39f6ba
    Ben Gamari authored
    4d39f6ba
  • Oleg Grenrus's avatar
    Add a note about make clean and boot · 03dc0738
    Oleg Grenrus authored
    03dc0738
  • Oleg Grenrus's avatar
    Add easy.sh · 71889668
    Oleg Grenrus authored
    This is what I wrote, using easy.sh is easy as
    
        ./easy.sh /code/ghc1/_build/stage1/bin/ghc /code/ghc2/_build/stage2/bin/ghc
    
    or even
    
        ./easy.sh /code/ghc1 /code/ghc2
    71889668
  • Oleg Grenrus's avatar
    Add easy.sh README section · 8d75c7ee
    Oleg Grenrus authored
    8d75c7ee
  • Andreas Klebinger's avatar
    Record dom-lt in spectral Makefile. · 52e761b9
    Andreas Klebinger authored
    Otherwise it isn't run by default.
    52e761b9
  • Ben Gamari's avatar
    Update runstdtest.prl for total memory output changes · 81065129
    Ben Gamari authored
    As reported in ghc#17196, GHC now prints "MiB" instead of
    "MB" in its RTS reporting output.
    81065129
  • Sylvain Henry's avatar
    Fix for GHC module renaming · a6cbac8f
    Sylvain Henry authored
    a6cbac8f
  • Gabor Greif's avatar
    Don't use gcc verbatim · d41a2d00
    Gabor Greif authored
    it might have a different name
    d41a2d00
  • Vladislav Zavialov's avatar
    Whitespace forward compatibility for proposal #229 · 3b7f96fe
    Vladislav Zavialov authored
    GHC Proposal #229 changes the lexical rules of Haskell, which may
    require slight whitespace adjustments in certain cases.
    
    This patch changes formatting in nofib in a way that enables it to
    compile under the proposed rules.
    3b7f96fe
  • Vladislav Zavialov's avatar
    Whitespace forward compatibility follow-up · c9fe4e92
    Vladislav Zavialov authored
    Required due to an amendment that disallows as-patterns with leading
    whitespace:
    
      f x@ pat = rhs  -- now disallowed
    c9fe4e92
......@@ -8,6 +8,8 @@ cachegrind.out.*
cachegrind.out.summary
perf.data
perf.data.*
dist-newstyle/
.ghc.environment.*
# Specific generated files
nofib-analyse/nofib-analyse
......@@ -33,11 +35,7 @@ real/anna/anna
real/bspt/bspt
real/cacheprof/cacheprof
real/compress/compress
real/compress/compress.stdin
real/compress/compress.stdout
real/compress2/compress2
real/compress2/compress2.stdin
real/compress2/compress2.stdout
real/eff/CS/CS
real/eff/CSD/CSD
real/eff/FS/FS
......@@ -56,7 +54,7 @@ real/hpg/hpg
real/infer/infer
real/lift/lift
real/linear/linear
real/maillist/runtime_files/addresses.tex
real/maillist/runtime_files/*.tex
real/maillist/maillist
real/mkhprog/mkhprog
real/parser/parser
......@@ -96,7 +94,11 @@ shootout/spectral-norm/spectral-norm
spectral/ansi/ansi
spectral/atom/atom
spectral/awards/awards
spectral/awards/*.stdout
spectral/awards/*.slowstdout
spectral/banner/banner
spectral/banner/*stdout
spectral/banner/*stdin
spectral/boyer/boyer
spectral/boyer2/boyer2
spectral/calendar/calendar
......@@ -107,6 +109,7 @@ spectral/constraints/constraints
spectral/cryptarithm1/cryptarithm1
spectral/cryptarithm2/cryptarithm2
spectral/cse/cse
spectral/dom-lt/dom-lt
spectral/eliza/eliza
spectral/exact-reals/exact-reals
spectral/expert/expert
......
variables:
DOCKER_REV: 2b69e99de97bd5bf1fbdbf45852231c3dcb602b6
validate:
image: "registry.gitlab.haskell.org/ghc/ci-images/x86_64-linux-deb9:$DOCKER_REV"
tags:
- x86_64-linux
before_script:
- git clean -xdf
- sudo apt install -y time
- ghc --version
- cabal --version
script:
- make clean
- cabal update
- make boot mode=fast
- "make mode=fast NoFibRuns=1 2>&1 | tee log"
- "nofib-analyse/nofib-analyse log"
- |
# The following checks that `make distclean` removes any files reported
# by `git clean -fxd`
make distclean
files=$(git clean -nxd | cut -d" " -f3 | sed "/log/d")
if ! [ -z $files ]
then
echo "The following files weren't cleaned:\n$files"
exit 1
fi
# Syntax: https://docs.gitlab.com/ee/user/project/code_owners.html
* @sgraf812 @bgamari
\ No newline at end of file
......@@ -3,33 +3,69 @@
This is the root directory of the "NoFib Haskell benchmark suite". It
should be part of a GHC source tree, that is the 'nofib' directory
should be at the same level in the tree as 'compiler' and 'libraries'.
This makes sure that NoFib picks up the stage 2 compiler from the
surrounding GHC source tree.
## Package Depedencies
You can also clone this repository in isolation, in which case it will
pick `$(which ghc)` or whatever the `HC` environment variable is set to.
Please make sure you have the following packages installed for your
system GHC:
* html
* regex-compat (will install: mtl, regex-base, regex-posix)
Additional information can also be found on
[NoFib's wiki page](https://ghc.haskell.org/trac/ghc/wiki/Building/RunningNoFib).
There's also a `easy.sh` helper script, which as name implies, is
automated and easy way to run `nofib`.
See the section at the end of README for its usage.
## Using
<details>
<summary>Git symlink support for Windows machines</summary>
NoFib uses a few symlinks here and there to share code between benchmarks.
Git for Windows has symlinks support for some time now, but
[it may not be enabled by default](https://stackoverflow.com/a/42137273/388010).
You will notice strange `make boot` failures if it's not enabled for you.
Make sure you follow the instructions in the link to enable symlink support,
possibly as simple as through `git config core.symlinks true` or cloning with
`git clone -c core.symlinks=true <URL>`.
</details>
Install [`cabal-install-2.4`](https://www.haskell.org/cabal/download.html) or later.
Then, to run the tests, execute:
make clean
make boot
make 2>&1 | tee nofib-log
```
$ make clean # or git clean -fxd, it's faster
$ # Generates input files for the benchmarks and builds compilation
$ # dependencies for make (ghc -M)
$ make boot
$ # Builds the benchmarks and runs them $NoFibRuns (default: 5) times
$ make
```
This will put the results in the file `nofib-log`. You can pass extra
options to a nofib run using the `EXTRA_HC_OPTS` variable like this:
make clean
make boot
make EXTRA_HC_OPTS="-fllvm" >&1 | tee nofib-llvm-log
```
$ make clean
$ make boot
$ make EXTRA_HC_OPTS="-fllvm"
```
To compare the results of multiple runs, use the program in
`../utils/nofib-analyse`, for example:
**Note:** to get all the results, you have to `clean` and `boot` between
separate `nofib` runs.
nofib-analyse nofib-log-6.4.2 nofib-log-6.6
To compare the results of multiple runs, save the output in a logfile
and use the program in `./nofib-analyse/nofib-analyse`, for example:
```
...
$ make 2>&1 | tee nofib-log-6.4.2
...
$ make 2>&1 | tee nofib-log-6.6
$ nofib-analyse nofib-log-6.4.2 nofib-log-6.6 | less
```
to generate a comparison of the runs in captured in `nofib-log-6.4.2`
and `nofib-log-6.6`. When making comparisons, be careful to ensure
......@@ -39,6 +75,55 @@ GHC version, GCC version, C libraries, static vs. dynamic GMP library,
build options, run options, and probably lots more. To be on the safe
side, make both runs on the same unloaded machine.
## Modes
Each benchmark is runnable in three different time `mode`s:
- `fast`: 0.1-0.2s
- `norm`: 1-2s
- `slow`: 5-10s
You can control which mode to run by setting an additional `mode` variable for
`make`. The default is `mode=norm`. Example for `mode=fast`:
```
$ make clean
$ make boot mode=fast
$ make mode=fast
```
Note that the `mode`s set in `make boot` and `make` need to agree. Otherwise you
will get output errors, because `make boot` will generate input files for a
different `mode`. A more DRY way to control the `mode` would be
```
$ make clean
$ export mode=fast
$ make boot
$ make
```
As CPU architectures advance, the above running times may drift and
occasionally, all benchmarks will need adjustments.
Be aware that `nofib-analyse` will ignore the result if it falls below 0.2s.
This is the default of its `-i` option, which is of course incompatible with
`mode=fast`. In that case, you should just set `-i` as appropriate, even
deactivate it with `-i 0`.
## Boot vs. benchmarked GHC
The `nofib-analyse` utility is compiled with `BOOT_HC` compiler,
which may be different then the GHC under the benchmark.
You can control which GHC you benchmark with `HC` variable
```
$ make clean
$ make boot HC=ghc-head
$ make HC=ghc-head 2>&1 | tee nofib-log-ghc-head
```
## Configuration
There are some options you might want to tweak; search for nofib in
......@@ -50,6 +135,16 @@ To get instruction counts, memory reads/writes, and "cache misses",
you'll need to get hold of Cachegrind, which is part of
[Valgrind](http://valgrind.org).
You can then pass `-cachegrind` as `EXTRA_RUNTEST_OPTS`. Counting
instructions slows down execution by a factor of ~30. But it's
a deterministic metric, so you can combine it with `NoFibRuns=1`:
```
$ (make EXTRA_RUNTEST_OPTS="-cachegrind" NoFibRuns=1) 2>&1 | tee nofib-log
```
Optionally combine this with `mode=fast`, see [Modes](#modes).
## Extra Packages
Some benchmarks aren't run by default and require extra packages are
......@@ -59,11 +154,80 @@ installed for the GHC compiler being tested. These packages include:
## Adding benchmarks
If you add a benchmark try to set the problem sizes for
fast/normal/slow reasonably.
fast/normal/slow reasonably. [Modes](#modes) lists the recommended brackets for
each mode.
### Benchmark Categories
So you have a benchmark to submit but don't know in which subfolder to put it? Here's some
advice on the intended semantics of each category.
#### Single threaded benchmarks
These are run when you just type `make`. Their semantics is explained in
[the Nofib paper](https://link.springer.com/chapter/10.1007%2F978-1-4471-3215-8_17)
(You can find a .ps online, thanks to @bgamari. Alternatively grep for
'Spectral' in docs/paper/paper.verb).
- `imaginary`: Mostly toy benchmarks, solving puzzles like n-queens.
- `spectral`: Algorithmic kernels, like FFT. If you want to add a benchmark of a
library, this most certainly the place to put it.
- `real`: Actual applications, with a command-line interface and all. Because of
the large dependency footprint of today's applications, these have become
rather aged.
- `shootout`: Benchmarks from
[the benchmarks game](https://benchmarksgame-team.pages.debian.net/benchmarksgame/),
formerly known as "language shootout".
Most of the benchmarks are quite old and aren't really written in way one would
write high-performance Haskell code today (e.g., use of `String`, lists,
redefining own list combinators that don't take part in list fusion, rare use of
strictness annotations or unboxed data), so new benchmarks for the `real` and
`spectral` in brackets in particular are always welcome!
#### Other categories
Other than the default single-threaded categories above, there are the
following (SG: I'm guessing here, have never run them):
- `gc`: Run by `make -C gc` (though you'll probably have to edit the Makefile to
your specific config). Select benchmarks from `spectral` and `real`, plus a
few more (Careful, these have not been touched by #15999/!5, see the next
subsection). Testdrives different GC configs, apparently.
- `smp`: Microbenchmarks for the `-threaded` runtime, measuring scheduler
performance on concurrent and STM-heavy code.
### Stability wrt. GC paramerisations
Additionally, pay attention that your benchmarks are stable wrt. different
GC paramerisations, so that small changes in allocation don't lead to big,
unexplicable jumps in performance. See #15999 for details. Also make sure
that you run the benchmark with the default GC settings, as enlarging Gen 0 or
Gen 1 heaps just amplifies the problem.
As a rule of thumb on how to ensure this: Make sure that your benchmark doesn't
just build up one big data and consume it in a final step, but rather that the
working set grows and shrinks (e.g. is approximately constant) over the whole
run of the benchmark. You can ensure this by iterating your main logic $n times
(how often depends on your program, but in the ball park of 100-1000).
You can test stability by plotting productivity curves for your `fast` settings
with the `prod.py` script attached to #15999.
If in doubt, ask Sebastian Graf for help.
## easy.sh
```
./easy.sh - easy nofib
Runtimes for normal should be above 0.3s if that can be reasonably
achieved. Less than that and there is a chance
nofib-analyse will ignore the result if it falls below 0.2s.
Usage: ./easy.sh [ -m mode ] /path/to/baseline/ghc /path/to/new/ghc"
GHC paths can point to the root of the GHC repository,
if it's build with Hadrian.
Available options:
-m MODE nofib mode: fast norm slow
This script caches the results using the sha256 of ghc executable.
Remove these files, if you want to rerun the benchmark.
```
......@@ -428,12 +428,6 @@ Same issue with GHC.IO.Encoding.UTF8 as treejoin
Real suite
---------------------------------------
cacheprof
~~~~~~~~~
Successive runs with the same data can yield different allocation
totals, for some reason.
Reported at https://ghc.haskell.org/trac/ghc/ticket/8611
gg
~~
Same issue with GHC.IO.Encoding.UTF8 as treejoin
......
{-# LANGUAGE CPP #-}
module NofibUtils where
import Data.Char (ord)
import Data.List (foldl')
import System.Environment (getArgs)
-- | A very simple hash function so that we don't have to store and compare
-- huge output files.
hash :: String -> Int
hash = foldl' (\acc c -> ord c + acc*31) 0
-- | Using @salt xs@ on an loop-invariant @xs@ inside a loop prevents the
-- compiler from floating out the input parameter.
#ifdef __GLASGOW_HASKELL__
salt :: a -> IO a
salt = pure
{-# NOINLINE salt #-}
#else
salt :: [a] -> IO [a]
-- this won't work with real/lift, but I can't think of another way
salt xs = do
s <- length <$> getArgs
-- Invariant: There are less than 'maxBound' parameters passed to the
-- executable, otherwise this isn't really 'pure'
-- anymore.
pure (take (max (maxBound - 1) s) xs)
#endif
\ No newline at end of file
#!/bin/sh
echo '\033]0;NOFIB: starting...\007'
# Settings
#######################################################################
mode=norm
# "Library" part
#######################################################################
show_usage () {
cat <<EOF
./easy.sh - easy nofib
Usage: ./easy.sh [ -m mode ] /path/to/baseline/ghc /path/to/new/ghc"
GHC paths can point to the root of the GHC repository,
if it's build with Hadrian.
Available options:
-m MODE nofib mode: fast norm slow
This script caches the results using the sha256 of ghc executable.
Remove these files, if you want to rerun the benchmark.
EOF
}
hashoffile () {
shasum -a 256 $1 | awk '{ print $1 }'
}
# getopt
#######################################################################
while getopts 'm:' flag; do
case $flag in
m)
case $OPTARG in
slow)
mode=$OPTARG
;;
norm)
mode=$OPTARG
;;
fast)
mode=$OPTARG
;;
*)
echo "Unknown mode: $OPTARG"
show_usage
exit 1
;;
esac
;;
?) show_usage
;;
esac
done
shift $((OPTIND - 1))
if [ $# -ne 2 ]; then
echo "Expected two arguments: ghc executables or roots of source repositories"
show_usage
exit 1
fi
OLD_HC=$1
NEW_HC=$2
# Set up
#######################################################################
# Arguments can point to GHC repository roots
if [ -d $OLD_HC -a -f "$OLD_HC/_build/stage1/bin/ghc" ]; then
OLD_HC="$OLD_HC/_build/stage1/bin/ghc"
fi
if [ -d $NEW_HC -a -f "$NEW_HC/_build/stage1/bin/ghc" ]; then
NEW_HC="$NEW_HC/_build/stage1/bin/ghc"
fi
# Check we have executables
if [ ! -f $NEW_HC -a -x $OLD_HC ]; then
echo "$OLD_HC is not an executable"
exit 1
fi
if [ ! -f $NEW_HC -a -x $NEW_HC ]; then
echo "$NEW_HC is not an executable"
exit 1
fi
# Info before we get going
#######################################################################
echo "Running nofib (mode=$mode) with $OLD_HC and $NEW_HC"
echo "Running nofib (mode=$mode) with $OLD_HC and $NEW_HC" | sed 's/./-/g'
sleep 2
# Run nofib
#######################################################################
# Run with old ghc
echo '\033]0;NOFIB: old\007'
OLD_HASH=$(hashoffile $OLD_HC)
OLD_OUTPUT=result-$OLD_HASH-$mode.txt
if [ -f $OLD_OUTPUT ]; then
echo "$OLD_OUTPUT exists; not re-running."
else
echo '\033]0;NOFIB: old, cleaning...\007'
make clean
echo '\033]0;NOFIB: old, booting...\007'
make boot mode=$mode HC=$OLD_HC
echo '\033]0;NOFIB: old, benchmarking...\007'
make mode=$mode HC=$OLD_HC 2>&1 | tee $OLD_OUTPUT
fi
# Run with new ghc
echo '\033]0;NOFIB: new\007'
NEW_HASH=$(hashoffile $NEW_HC)
NEW_OUTPUT=result-$NEW_HASH-$mode.txt
if [ -f $NEW_OUTPUT ]; then
echo "$NEW_OUTPUT exists; not re-running."
else
echo '\033]0;NOFIB: new, cleaning...\007'
make clean
echo '\033]0;NOFIB: new, booting...\007'
make boot mode=$mode HC=$NEW_HC
echo '\033]0;NOFIB: new, benchmarking...\007'
make mode=$mode HC=$NEW_HC 2>&1 | tee $NEW_OUTPUT
fi
# Done
#######################################################################
echo '\033]0;NOFIB: done\007'
# Analyse
./nofib-analyse/nofib-analyse $OLD_OUTPUT $NEW_OUTPUT > report.txt
# Show report
less report.txt
......@@ -75,7 +75,7 @@ the roots of two binomial trees and makes the larger a child of the
smaller (thus bumping its degree by one). It is essential that this
only be called on binomial trees of equal degree.
>link (a @ (Node x as)) (b @ (Node y bs)) =
>link (a@(Node x as)) (b@(Node y bs)) =
> if x <= y then Node x (b:as) else Node y (a:bs)
It will also be useful to extract the minimum element from a tree.
......
......@@ -4,7 +4,7 @@ module Parser ( parseModule, parseStmt, parseIdentifier, parseType,
#include "HsVersions.h"
import HsSyn
import GHC.Hs
import RdrHsSyn
import HscTypes ( IsBootInterface, DeprecTxt )
import Lexer
......
......@@ -16,7 +16,7 @@ module Parser ( parseModule, parseStmt, parseIdentifier, parseType,
#include "HsVersions.h"
import HsSyn
import GHC.Hs
import RdrHsSyn
import HscTypes ( IsBootInterface, DeprecTxt )
import Lexer
......
......@@ -7,6 +7,8 @@
import Data.Ratio
import System.Environment
import Control.Monad
import NofibUtils
-- powers = [[r^n | r<-[2..]] | n<-1..]
-- type signature required for compilers lacking the monomorphism restriction
......@@ -23,7 +25,7 @@ pascal:: [[Integer]]
pascal = [1,2,1] : map (\line -> zipWith (+) (line++[0]) (0:line)) pascal
bernoulli 0 = 1
bernoulli 1 = -(1%2)
bernoulli 1 = -(1%2)
bernoulli n | odd n = 0
bernoulli n =
(-1)%2
......@@ -33,8 +35,7 @@ bernoulli n =
| (k,combs)<- zip [2..n] pascal]
where powers = (neg_powers!!(n-1))
main = do
main = replicateM_ 500 $ do
[arg] <- getArgs
let n = (read arg)::Int
putStr $ "Bernoulli of " ++ (show n) ++ " is "
print (bernoulli n)
print (hash (show (bernoulli n)))
......@@ -5,9 +5,9 @@ include $(TOP)/mk/boilerplate.mk
# we don't want to include paraffins.c
SRCS=Main.hs
FAST_OPTS = 500
NORM_OPTS = 1200
SLOW_OPTS = 1200
FAST_OPTS = 60
NORM_OPTS = 180
SLOW_OPTS = 320
include $(TOP)/mk/target.mk
../../common/NofibUtils.hs
\ No newline at end of file
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
......@@ -3,6 +3,8 @@ Original program due to Dale Thurston, Aug 2001
> module Main where
> import System.Environment (getArgs)
> import Control.Monad (replicateM_)
> import NofibUtils (hash)
> type ContFrac = [Integer]
......@@ -33,14 +35,13 @@ digit regardless of what the input is; i.e., to see if the interval
Finally, we convert a continued fraction to digits by repeatedly multiplying by 10.
> toDigits :: ContFrac -> [Integer]
> toDigits (x:xs) = x:toDigits (ratTrans (10,0,0,1) xs)
> takeDigits :: Int -> ContFrac -> [Integer]
> takeDigits 0 _ = []
> takeDigits n (x:xs) = x:takeDigits (n-1) (ratTrans (10,0,0,1) xs)
> e :: [Integer]
> e = toDigits eContFrac
> e :: Int -> [Integer]
> e n = takeDigits n eContFrac
> main = do
> main = replicateM_ 100 $ do
> [digits] <- getArgs
> print (take (read digits) e)
> print (hash (show (e (read digits))))
......@@ -3,6 +3,6 @@ include $(TOP)/mk/boilerplate.mk
include $(TOP)/mk/target.mk
FAST_OPTS = 450
NORM_OPTS = 2000
SLOW_OPTS = 2200
FAST_OPTS = 50
NORM_OPTS = 150
SLOW_OPTS = 320
../../common/NofibUtils.hs
\ No newline at end of file
[2,7,1,8,2,8,1,8,2,8,4,5,9,0,4,5,2,3,5,3,6,0,2,8,7,4,7,1,3,5,2,6,6,2,4,9,7,7,5,7,2,4,7,0,9,3,6,9,9,9,5,9,5,7,4,9,6,6,9,6,7,6,2,7,7,2,4,0,7,6,6,3,0,3,5,3,5,4,7,5,9,4,5,7,1,3,8,2,1,7,8,5,2,5,1,6,6,4,2,7,4,2,7,4,6,6,3,9,1,9,3,2,0,0,3,0,5,9,9,2,1,8,1,7,4,1,3,5,9,6,6,2,9,0,4,3,5,7,2,9,0,0,3,3,4,2,9,5,2,6,0,5,9,5,6,3,0,7,3,8,1,3,2,3,2,8,6,2,7,9,4,3,4,9,0,7,6,3,2,3,3,8,2,9,8,8,0,7,5,3,1,9,5,2,5,1,0,1,9,0,1,1,5,7,3,8,3,4,1,8,7,9,3,0,7,0,2,1,5,4,0,8,9,1,4,9,9,3,4,8,8,4,1,6,7,5,0,9,2,4,4,7,6,1,4,6,0,6,6,8,0,8,2,2,6,4,8,0,0,1,6,8,4,7,7,4,1,1,8,5,3,7,4,2,3,4,5,4,4,2,4,3,7,1,0,7,5,3,9,0,7,7,7,4,4,9,9,2,0,6,9,5,5,1,7,0,2,7,6,1,8,3,8,6,0,6,2,6,1,3,3,1,3,8,4,5,8,3,0,0,0,7,5,2,0,4,4,9,3,3,8,2,6,5,6,0,2,9,7,6,0,6,7,3,7,1,1,3,2,0,0,7,0,9,3,2,8,7,0,9,1,2,7,4,4,3,7,4,7,0,4,7,2,3,0,6,9,6,9,7,7,2,0,9,3,1,0,1,4,1,6,9,2,8,3,6,8,1,9,0,2,5,5,1,5,1,0,8,6,5,7,4,6,3,7,7,2,1,1,1,2,5,2,3,8,9,7,8,4,4,2,5,0,5,6,9,5,3,6,9]
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
[2,7,1,8,2,8,1,8,2,8,4,5,9,0,4,5,2,3,5,3,6,0,2,8,7,4,7,1,3,5,2,6,6,2,4,9,7,7,5,7,2,4,7,0,9,3,6,9,9,9,5,9,5,7,4,9,6,6,9,6,7,6,2,7,7,2,4,0,7,6,6,3,0,3,5,3,5,4,7,5,9,4,5,7,1,3,8,2,1,7,8,5,2,5,1,6,6,4,2,7,4,2,7,4,6,6,3,9,1,9,3,2,0,0,3,0,5,9,9,2,1,8,1,7,4,1,3,5,9,6,6,2,9,0,4,3,5,7,2,9,0,0,3,3,4,2,9,5,2,6,0,5,9,5,6,3,0,7,3,8,1,3,2,3,2,8,6,2,7,9,4,3,4,9,0,7,6,3,2,3,3,8,2,9,8,8,0,7,5,3,1,9,5,2,5,1,0,1,9,0,1,1,5,7,3,8,3,4,1,8,7,9,3,0,7,0,2,1,5,4,0,8,9,1,4,9,9,3,4,8,8,4,1,6,7,5,0,9,2,4,4,7,6,1,4,6,0,6,6,8,0,8,2,2,6,4,8,0,0,1,6,8,4,7,7,4,1,1,8,5,3,7,4,2,3,4,5,4,4,2,4,3,7,1,0,7,5,3,9,0,7,7,7,4,4,9,9,2,0,6,9,5,5,1,7,0,2,7,6,1,8,3,8,6,0,6,2,6,1,3,3,1,3,8,4,5,8,3,0,0,0,7,5,2,0,4,4,9,3,3,8,2,6,5,6,0,2,9,7,6,0,6,7,3,7,1,1,3,2,0,0,7,0,9,3,2,8,7,0,9,1,2,7,4,4,3,7,4,7,0,4,7,2,3,0,6,9,6,9,7,7,2,0,9,3,1,0,1,4,1,6,9,2,8,3,6,8,1,9,0,2,5,5,1,5,1,0,8,6,5,7,4,6,3,7,7,2,1,1,1,2,5,2,3,8,9,7,8,4,4,2,5,0,5,6,9,5,3,6,9,6,7,7,0,7,8,5,4,4,9,9,6,9,9,6,7,9,4,6,8,6,4,4,5,4,9,0,5,9,8,7,9,3,1,6,3,6,8,8,9,2,3,0,0,9,8,7,9,3,1,2,7,7,3,6,1,7,8,2,1,5,4,2,4,9,9,9,2,2,9,5,7,6,3,5,1,4,8,2,2,0,8,2,6,9,8,9,5,1,9,3,6,6,8,0,3,3,1,8,2,5,2,8,8,6,9,3,9,8,4,9,6,4,6,5,1,0,5,8,2,0,9,3,9,2,3,9,8,2,9,4,8,8,7,9,3,3,2,0,3,6,2,5,0,9,4,4,3,1,1,7,3,0,1,2,3,8,1,9,7,0,6,8,4,1,6,1,4,0,3,9,7,0,1,9,8,3,7,6,7,9,3,2,0,6,8,3,2,8,2,3,7,6,4,6,4,8,0,4,2,9,5,3,1,1,8,0,2,3,2,8,7,8,2,5,0,9,8,1,9,4,5,5,8,1,5,3,0,1,7,5,6,7,1,7,3,6,1,3,3,2,0,6,9,8,1,1,2,5,0,9,9,6,1,8,1,8,8,1,5,9,3,0,4,1,6,9,0,3,5,1,5,9,8,8,8,8,5,1,9,3,4,5,8,0,7,2,7,3,8,6,6,7,3,8,5,8,9,4,2,2,8,7,9,2,2,8,4,9,9,8,9,2,0,8,6,8,0,5,8,2,5,7,4,9,2,7,9,6,1,0,4,8,4,1,9,8,4,4,4,3,6,3,4,6,3,2,4,4,9,6,8,4,8,7,5,6,0,2,3,3,6,2,4,8,2,7,0,4,1,9,7,8,6,2,3,2,0,9,0,0,2,1,6,0,9,9,0,2,3,5,3,0,4,3,6,9,9,4,1,8,4,9,1,4,6,3,1,4,0,9,3,4,3,1,7,3,8,1,4,3,6,4,0,5,4,6,2,5,3,1,5,2,0,9,6,1,8,3,6,9,0,8,8,8,7,0,7,0,1,6,7,6,8,3,9,6,4,2,4,3,7,8,1,4,0,5,9,2,7,1,4,5,6,3,5,4,9,0,6,1,3,0,3,1,0,7,2,0,8,5,1,0,3,8,3,7,5,0,5,1,0,1,1,5,7,4,7,7,0,4,1,7,1,8,9,8,6,1,0,6,8,7,3,9,6,9,6,5,5,2,1,2,6,7,1,5,4,6,8,8,9,5,7,0,3,5,0,3,5,4,0,2,1,2,3,4,0,7,8,4,9,8,1,9,3,3,4,3,2,1,0,6,8,1,7,0,1,2,1,0,0,5,6,2,7,8,8,0,2,3,5,1,9,3,0,3,3,2,2,4,7,4,5,0,1,5,8,5,3,9,0,4,7,3,0,4,1,9,9,5,7,7,7,7,0,9,3,5,0,3,6,6,0,4,1,6,9,9,7,3,2,9,7,2,5,0,8,8,6,8,7,6,9,6,6,4,0,3,5,5,5,7,0,7,1,6,2,2,6,8,4,4,7,1,6,2,5,6,0,7,9,8,8,2,6,5,1,7,8,7,1,3,4,1,9,5,1,2,4,6,6,5,2,0,1,0,3,0,5,9,2,1,2,3,6,6,7,7,1,9,4,3,2,5,2,7,8,6,7,5,3,9,8,5,5,8,9,4,4,8,9,6,9,7,0,9,6,4,0,9,7,5,4,5,9,1,8,5,6,9,5,6,3,8,0,2,3,6,3,7,0,1,6,2,1,1,2,0,4,7,7,4,2,7,2,2,8,3,6,4,8,9,6,1,3,4,2,2,5,1,6,4,4,5,0,7,8,1,8,2,4,4,2,3,5,2,9,4,8,6,3,6,3,7,2,1,4,1,7,4,0,2,3,8,8,9,3,4,4,1,2,4,7,9,6,3,5,7,4,3,7,0,2,6,3,7,5,5,2,9,4,4,4,8,3,3,7,9,9,8,0,1,6,1,2,5,4,9,2,2,7,8,5,0,9,2,5,7,7,8,2,5,6,2,0,9,2,6,2,2,6,4,8,3,2,6,2,7,7,9,3,3,3,8,6,5,6,6,4,8,1,6,2,7,7,2,5,1,6,4,0,1,9,1,0,5,9,0,0,4,9,1,6,4,4,9,9,8,2,8,9,3,1,5,0,5,6,6,0,4,7,2,5,8,0,2,7,7,8,6,3,1,8,6,4,1,5,5,1,9,5,6,5,3,2,4,4,2,5,8,6,9,8,2,9,4,6,9,5,9,3,0,8,0,1,9,1,5,2,9,8,7,2,1,1,7,2,5,5,6,3,4,7,5,4,6,3,9,6,4,4,7,9,1,0,1,4,5,9,0,4,0,9,0,5,8,6,2,9,8,4,9,6,7,9,1,2,8,7,4,0,6,8,7,0,5,0,4,8,9,5,8,5,8,6,7,1,7,4,7,9,8,5,4,6,6,7,7,5,7,5,7,3,2,0,5,6,8,1,2,8,8,4,5,9,2,0,5,4,1,3,3,4,0,5,3,9,2,2,0,0,0,1,1,3,7,8,6,3,0,0,9,4,5,5,6,0,6,8,8,1,6,6,7,4,0,0,1,6,9,8,4,2,0,5,5,8,0,4,0,3,3,6,3,7,9,5,3,7,6,4,5,2,0,3,0,4,0,2,4,3,2,2,5,6,6,1,3,5,2,7,8,3,6,9,5,1,1,7,7,8,8,3,8,6,3,8,7,4,4,3,9,6,6,2,5,3,2,2,4,9,8,5,0,6,5,4,9,9,5,8,8,6,2,3,4,2,8,1,8,9,9,7,0,7,7,3,3,2,7,6,1,7,1,7,8,3,9,2,8,0,3,4,9,4,6,5,0,1,4,3,4,5,5,8,8,9,7,0,7,1,9,4,2,5,8,6,3,9,8,7,7,2,7,5,4,7,1,0,9,6,2,9,5,3,7,4,1,5,2,1,1,1,5,1,3,6,8,3,5,0,6,2,7,5,2,6,0,2,3,2,6,4,8,4,7,2,8,7,0,3,9,2,0,7,6,4,3,1,0,0,5,9,5,8,4,1,1,6,6,1,2,0,5,4,5,2,9,7,0,3,0,2,3,6,4,7,2,5,4,9,2,9,6,6,6,9,3,8,1,1,5,1,3,7,3,2,2,7,5,3,6,4,5,0,9,8,8,8,9,0,3,1,3,6,0,2,0,5,7,2,4,8,1,7,6,5,8,5,1,1,8,0,6,3,0,3,6,4,4,2,8,1,2,3,1,4,9,6,5,5,0,7,0,4,7,5,1,0,2,5,4,4,6,5,0,1,1,7,2,7,2,1,1,5,5,5,1,9,4,8,6,6,8,5,0,8,0,0,3,6,8,5,3,2,2,8,1,8,3,1,5,2,1,9,6,0,0,3,7,3,5,6,2,5,2,7,9,4,4,9,5,1,5,8,2,8,4,1,8,8,2,9,4,7,8,7,6,1,0,8,5,2,6,3,9,8,1,3,9,5,5,9,9,0,0,6,7,3,7,6,4,8,2,9,2,2,4,4,3,7,5,2,8,7,1,8,4,6,2,4,5,7,8,0,3,6,1,9,2,9,8,1,9,7,1,3,9,9,1,4,7,5,6,4,4,8,8,2,6,2,6,0,3,9,0,3,3,8,1,4,4,1,8,2,3,2,6,2,5,1,5,0,9,7,4,8,2,7,9,8,7,7,7,9,9,6,4,3,7,3,0,8,9,9,7,0,3,8,8,8,6,7,7,8,2,2,7,1,3,8,3,6,0,5,7,7,2,9,7,8,8,2,4,1,2,5,6,1,1,9,0,7,1,7,6,6,3,9,4,6,5,0,7,0,6,3,3,0,4,5,2,7,9,5,4,6,6,1,8,5,5,0,9,6,6,6,6,1,8,5,6,6,4,7,0,9,7,1,1,3,4,4,4,7,4,0,1,6]
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
[2,7,1,8,2,8,1,8,2,8,4,5,9,0,4,5,2,3,5,3,6,0,2,8,7,4,7,1,3,5,2,6,6,2,4,9,7,7,5,7,2,4,7,0,9,3,6,9,9,9,5,9,5,7,4,9,6,6,9,6,7,6,2,7,7,2,4,0,7,6,6,3,0,3,5,3,5,4,7,5,9,4,5,7,1,3,8,2,1,7,8,5,2,5,1,6,6,4,2,7,4,2,7,4,6,6,3,9,1,9,3,2,0,0,3,0,5,9,9,2,1,8,1,7,4,1,3,5,9,6,6,2,9,0,4,3,5,7,2,9,0,0,3,3,4,2,9,5,2,6,0,5,9,5,6,3,0,7,3,8,1,3,2,3,2,8,6,2,7,9,4,3,4,9,0,7,6,3,2,3,3,8,2,9,8,8,0,7,5,3,1,9,5,2,5,1,0,1,9,0,1,1,5,7,3,8,3,4,1,8,7,9,3,0,7,0,2,1,5,4,0,8,9,1,4,9,9,3,4,8,8,4,1,6,7,5,0,9,2,4,4,7,6,1,4,6,0,6,6,8,0,8,2,2,6,4,8,0,0,1,6,8,4,7,7,4,1,1,8,5,3,7,4,2,3,4,5,4,4,2,4,3,7,1,0,7,5,3,9,0,7,7,7,4,4,9,9,2,0,6,9,5,5,1,7,0,2,7,6,1,8,3,8,6,0,6,2,6,1,3,3,1,3,8,4,5,8,3,0,0,0,7,5,2,0,4,4,9,3,3,8,2,6,5,6,0,2,9,7,6,0,6,7,3,7,1,1,3,2,0,0,7,0,9,3,2,8,7,0,9,1,2,7,4,4,3,7,4,7,0,4,7,2,3,0,6,9,6,9,7,7,2,0,9,3,1,0,1,4,1,6,9,2,8,3,6,8,1,9,0,2,5,5,1,5,1,0,8,6,5,7,4,6,3,7,7,2,1,1,1,2,5,2,3,8,9,7,8,4,4,2,5,0,5,6,9,5,3,6,9,6,7,7,0,7,8,5,4,4,9,9,6,9,9,6,7,9,4,6,8,6,4,4,5,4,9,0,5,9,8,7,9,3,1,6,3,6,8,8,9,2,3,0,0,9,8,7,9,3,1,2,7,7,3,6,1,7,8,2,1,5,4,2,4,9,9,9,2,2,9,5,7,6,3,5,1,4,8,2,2,0,8,2,6,9,8,9,5,1,9,3,6,6,8,0,3,3,1,8,2,5,2,8,8,6,9,3,9,8,4,9,6,4,6,5,1,0,5,8,2,0,9,3,9,2,3,9,8,2,9,4,8,8,7,9,3,3,2,0,3,6,2,5,0,9,4,4,3,1,1,7,3,0,1,2,3,8,1,9,7,0,6,8,4,1,6,1,4,0,3,9,7,0,1,9,8,3,7,6,7,9,3,2,0,6,8,3,2,8,2,3,7,6,4,6,4,8,0,4,2,9,5,3,1,1,8,0,2,3,2,8,7,8,2,5,0,9,8,1,9,4,5,5,8,1,5,3,0,1,7,5,6,7,1,7,3,6,1,3,3,2,0,6,9,8,1,1,2,5,0,9,9,6,1,8,1,8,8,1,5,9,3,0,4,1,6,9,0,3,5,1,5,9,8,8,8,8,5,1,9,3,4,5,8,0,7,2,7,3,8,6,6,7,3,8,5,8,9,4,2,2,8,7,9,2,2,8,4,9,9,8,9,2,0,8,6,8,0,5,8,2,5,7,4,9,2,7,9,6,1,0,4,8,4,1,9,8,4,4,4,3,6,3,4,6,3,2,4,4,9,6,8,4,8,7,5,6,0,2,3,3,6,2,4,8,2,7,0,4,1,9,7,8,6,2,3,2,0,9,0,0,2,1,6,0,9,9,0,2,3,5,3,0,4,3,6,9,9,4,1,8,4,9,1,4,6,3,1,4,0,9,3,4,3,1,7,3,8,1,4,3,6,4,0,5,4,6,2,5,3,1,5,2,0,9,6,1,8,3,6,9,0,8,8,8,7,0,7,0,1,6,7,6,8,3,9,6,4,2,4,3,7,8,1,4,0,5,9,2,7,1,4,5,6,3,5,4,9,0,6,1,3,0,3,1,0,7,2,0,8,5,1,0,3,8,3,7,5,0,5,1,0,1,1,5,7,4,7,7,0,4,1,7,1,8,9,8,6,1,0,6,8,7,3,9,6,9,6,5,5,2,1,2,6,7,1,5,4,6,8,8,9,5,7,0,3,5,0,3,5,4,0,2,1,2,3,4,0,7,8,4,9,8,1,9,3,3,4,3,2,1,0,6,8,1,7,0,1,2,1,0,0,5,6,2,7,8,8,0,2,3,5,1,9,3,0,3,3,2,2,4,7,4,5,0,1,5,8,5,3,9,0,4,7,3,0,4,1,9,9,5,7,7,7,7,0,9,3,5,0,3,6,6,0,4,1,6,9,9,7,3,2,9,7,2,5,0,8,8,6,8,7,6,9,6,6,4,0,3,5,5,5,7,0,7,1,6,2,2,6,8,4,4,7,1,6,2,5,6,0,7,9,8,8,2,6,5,1,7,8,7,1,3,4,1,9,5,1,2,4,6,6,5,2,0,1,0,3,0,5,9,2,1,2,3,6,6,7,7,1,9,4,3,2,5,2,7,8,6,7,5,3,9,8,5,5,8,9,4,4,8,9,6,9,7,0,9,6,4,0,9,7,5,4,5,9,1,8,5,6,9,5,6,3,8,0,2,3,6,3,7,0,1,6,2,1,1,2,0,4,7,7,4,2,7,2,2,8,3,6,4,8,9,6,1,3,4,2,2,5,1,6,4,4,5,0,7,8,1,8,2,4,4,2,3,5,2,9,4,8,6,3,6,3,7,2,1,4,1,7,4,0,2,3,8,8,9,3,4,4,1,2,4,7,9,6,3,5,7,4,3,7,0,2,6,3,7,5,5,2,9,4,4,4,8,3,3,7,9,9,8,0,1,6,1,2,5,4,9,2,2,7,8,5,0,9,2,5,7,7,8,2,5,6,2,0,9,2,6,2,2,6,4,8,3,2,6,2,7,7,9,3,3,3,8,6,5,6,6,4,8,1,6,2,7,7,2,5,1,6,4,0,1,9,1,0,5,9,0,0,4,9,1,6,4,4,9,9,8,2,8,9,3,1,5,0,5,6,6,0,4,7,2,5,8,0,2,7,7,8,6,3,1,8,6,4,1,5,5,1,9,5,6,5,3,2,4,4,2,5,8,6,9,8,2,9,4,6,9,5,9,3,0,8,0,1,9,1,5,2,9,8,7,2,1,1,7,2,5,5,6,3,4,7,5,4,6,3,9,6,4,4,7,9,1,0,1,4,5,9,0,4,0,9,0,5,8,6,2,9,8,4,9,6,7,9,1,2,8,7,4,0,6,8,7,0,5,0,4,8,9,5,8,5,8,6,7,1,7,4,7,9,8,5,4,6,6,7,7,5,7,5,7,3,2,0,5,6,8,1,2,8,8,4,5,9,2,0,5,4,1,3,3,4,0,5,3,9,2,2,0,0,0,1,1,3,7,8,6,3,0,0,9,4,5,5,6,0,6,8,8,1,6,6,7,4,0,0,1,6,9,8,4,2,0,5,5,8,0,4,0,3,3,6,3,7,9,5,3,7,6,4,5,2,0,3,0,4,0,2,4,3,2,2,5,6,6,1,3,5,2,7,8,3,6,9,5,1,1,7,7,8,8,3,8,6,3,8,7,4,4,3,9,6,6,2,5,3,2,2,4,9,8,5,0,6,5,4,9,9,5,8,8,6,2,3,4,2,8,1,8,9,9,7,0,7,7,3,3,2,7,6,1,7,1,7,8,3,9,2,8,0,3,4,9,4,6,5,0,1,4,3,4,5,5,8,8,9,7,0,7,1,9,4,2,5,8,6,3,9,8,7,7,2,7,5,4,7,1,0,9,6,2,9,5,3,7,4,1,5,2,1,1,1,5,1,3,6,8,3,5,0,6,2,7,5,2,6,0,2,3,2,6,4,8,4,7,2,8,7,0,3,9,2,0,7,6,4,3,1,0,0,5,9,5,8,4,1,1,6,6,1,2,0,5,4,5,2,9,7,0,3,0,2,3,6,4,7,2,5,4,9,2,9,6,6,6,9,3,8,1,1,5,1,3,7,3,2,2,7,5,3,6,4,5,0,9,8,8,8,9,0,3,1,3,6,0,2,0,5,7,2,4,8,1,7,6,5,8,5,1,1,8,0,6,3,0,3,6,4,4,2,8,1,2,3,1,4,9,6,5,5,0,7,0,4,7,5,1,0,2,5,4,4,6,5,0,1,1,7,2,7,2,1,1,5,5,5,1,9,4,8,6,6,8,5,0,8,0,0,3,6,8,5,3,2,2,8,1,8,3,1,5,2,1,9,6,0,0,3,7,3,5,6,2,5,2,7,9,4,4,9,5,1,5,8,2,8,4,1,8,8,2,9,4,7,8,7,6,1,0,8,5,2,6,3,9,8,1,3]
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961
-3573863879128417961