Commit 8632268a authored by Sebastian Graf's avatar Sebastian Graf

Stabilise benchmarks wrt. GC

Summary:
This is due to #15999, a follow-up on #5793 and #15357 and changes all
benchmarks, some of them (i.e. `wheel-sieve1`, `awards`) rather drastically.

The general plan is outlined in #15999: Identify GC-sensitive benchmarks by
looking at how productivity rates change over different nursery sizes and
iterate `main` of these benchmarks often enough for the non-monotony and
discontinuities to go away.

I was paying attention that the benchmarked logic is actually run $n times more
often, rather than just benchmarking IO operations printing the result of CAFs.

When I found benchmarks with insignificant runtime (#15357), I made sure that
parameters/input files were adjusted so that runtime of the different modes
fall within the ranges proposed in
https://ghc.haskell.org/trac/ghc/ticket/15357#comment:4

- fast: 0.1-0.2s
- norm: 1-2s
- slow: 5-10s

This is what I did:

- Stabilise bernoulli
- Stabilise digits-of-e1
- Stabilise digits-of-e2
- Stabilise gen_regexp
- Adjust running time of integrate
- Adjust running time of kahan
- Stabilise paraffins
- Stabilise primes
- Adjust running time of rfib
- Adjust running time of tak
- Stabilise wheel-sieve1
- Stabilise wheel-sieve2
- Adjust running time of x2n1
- Adjust running time of ansi
- Adjust running time of atom
- Make awards benchmark something other than IO
- Adjust running time of banner
- Stabilise boyer
- Adjust running time of boyer2
- Adjust running time of queens
- Adjust running time of calendar
- Adjust runtime of cichelli
- Stabilise circsim
- Stabilise clausify
- Stabilise constraints with moderate success
- Adjust running time of cryptarithm1
- Adjust running time of cryptarythm2
- Adjust running time of cse
- Adjust running time of eliza
- Adjust running time of exact-reals
- Adjust running time of expert
- Stabilise fft2
- Stabilise fibheaps
- Stabilise fish
- Adjust running time for gcd
- Stabilise comp_lab_zift
- Stabilise event
- Stabilise fft
- Stabilise genfft
- Stabilise ida
- Adjust running time for listcompr
- Adjust running time for listcopy
- Adjust running time of nucleic2
- Attempt to stabilise parstof
- Stabilise sched
- Stabilise solid
- Adjust running time of transform
- Adjust running time of typecheck
- Stabilise wang
- Stabilise wave4main
- Adjust running time of integer
- Adjust running time of knights
- Stabilise lambda
- Stabilise lcss
- Stabilise life
- Stabilise mandel
- Stabilise mandel2
- Adjust running time of mate
- Stabilise minimax
- Adjust running time of multiplier
- Adjust running time of para
- Stabilise power
- Adjust running time of primetest
- Stabilise puzzle with mild success
- Adjust running time for rewrite
- Stabilise simple with mild success
- Stabilise sorting
- Stabilise sphere
- Stabilise treejoin
- Stabilise anna
- Stabilise bspt
- Stabilise cacheprof
- Stablise compress
- Stablise compress2
- Stabilise fem
- Adjust running time of fluid
- Stabilise fulsom
- Stabilise gamteb
- Stabilise gg
- Stabilise grep
- Adjust running time of hidden
- Stabilise hpg
- Stabilise infer
- Stabilise lift
- Stabilise linear
- Attempt to stabilise maillist
- Stabilise mkhprog
- Stabilise parser
- Stabilise pic
- Stabilise prolog
- Attempt to stabilise reptile
- Adjust running time of rsa
- Adjust running time of scs
- Stabilise symalg
- Stabilise veritas
- Stabilise binary-trees
- Adjust running time of fasta
- Adjust running time of k-nucleotide
- Adjust running time of pidigits
- Adjust running time of reverse-complement
- Adjust running time of spectral-norm
- Adjust running time of fannkuch-redux
- Adjust running time for n-body

Problematic benchmarks:

- `last-piece`: Unclear how to stabilise. Runs for 300ms and I can't make up smaller inputs because I don't understand what it does.
- `pretty`: It's just much too small to be relevant at all. Maybe we want to get rid of this one?
- `scc`: Same as `pretty`. The input graph for which SCC analysis is done is much too small and I can't find good directed example graphs on the internet.
- `secretary`: Apparently this needs `-package random` and consequently hasn't been run for a long time.
- `simple`: Same as `last-piece`. Decent runtime (70ms), but it's unstable and I see no way to iterate it ~100 times in fast mode.
- `eff`: Every benchmark is problematic here. Not from the point of view of allocations, but because the actual logic is vacuous. IMO, these should be performance tests, not actual benchmarks. Alternatively, write an actual application that makes use of algebraic effects.
- `maillist`: Too trivial. It's just String/list manipulation, not representative of any Haskell code we would write today (no use of base library functions which could be fused, uses String instead of Text). It's only 75 loc according to `cloc`, that's not a `real` application.

Reviewers: simonpj, simonmar, bgamari, AndreasK, osa1, alpmestan, O26 nofib

GHC Trac Issues: #15999

Differential Revision: https://phabricator.haskell.org/D5438
parent e2d614e4
Pipeline #967 passed with stage
in 4 minutes and 42 seconds
......@@ -33,9 +33,7 @@ real/anna/anna
real/bspt/bspt
real/cacheprof/cacheprof
real/compress/compress
real/compress/compress.stdin
real/compress2/compress2
real/compress2/compress2.stdin
real/eff/CS/CS
real/eff/CSD/CSD
real/eff/FS/FS
......@@ -54,7 +52,7 @@ real/hpg/hpg
real/infer/infer
real/lift/lift
real/linear/linear
real/maillist/runtime_files/addresses.tex
real/maillist/runtime_files/*.tex
real/maillist/maillist
real/mkhprog/mkhprog
real/parser/parser
......@@ -94,7 +92,11 @@ shootout/spectral-norm/spectral-norm
spectral/ansi/ansi
spectral/atom/atom
spectral/awards/awards
spectral/awards/*.stdout
spectral/awards/*.slowstdout
spectral/banner/banner
spectral/banner/*stdout
spectral/banner/*stdin
spectral/boyer/boyer
spectral/boyer2/boyer2
spectral/calendar/calendar
......
......@@ -3,6 +3,14 @@
This is the root directory of the "NoFib Haskell benchmark suite". It
should be part of a GHC source tree, that is the 'nofib' directory
should be at the same level in the tree as 'compiler' and 'libraries'.
This makes sure that NoFib picks up the stage 2 compiler from the
surrounding GHC source tree.
You can also clone this repository in isolation, in which case it will
pick `$(which ghc)` or whatever the `HC` environment variable is set to.
Additional information can also be found on
[NoFib's wiki page](https://ghc.haskell.org/trac/ghc/wiki/Building/RunningNoFib).
## Package Depedencies
......@@ -15,21 +23,34 @@ system GHC:
Then, to run the tests, execute:
make clean
make boot
make 2>&1 | tee nofib-log
```
$ make clean # or git clean -fxd, it's faster
$ # Generates input files for the benchmarks and builds compilation
$ # dependencies for make (ghc -M)
$ make boot
$ # Builds the benchmarks and runs them $NoFibRuns (default: 5) times
$ make
```
This will put the results in the file `nofib-log`. You can pass extra
options to a nofib run using the `EXTRA_HC_OPTS` variable like this:
make clean
make boot
make EXTRA_HC_OPTS="-fllvm" >&1 | tee nofib-llvm-log
```
$ make clean
$ make boot
$ make EXTRA_HC_OPTS="-fllvm"
```
To compare the results of multiple runs, use the program in
`../utils/nofib-analyse`, for example:
To compare the results of multiple runs, save the output in a logfile
and use the program in `../utils/nofib-analyse`, for example:
nofib-analyse nofib-log-6.4.2 nofib-log-6.6
```
...
$ make 2>&1 | tee nofib-log-6.4.2
...
$ make 2>&1 | tee nofib-log-6.6
$ nofib-analyse nofib-log-6.4.2 nofib-log-6.6 | less
```
to generate a comparison of the runs in captured in `nofib-log-6.4.2`
and `nofib-log-6.6`. When making comparisons, be careful to ensure
......@@ -39,6 +60,42 @@ GHC version, GCC version, C libraries, static vs. dynamic GMP library,
build options, run options, and probably lots more. To be on the safe
side, make both runs on the same unloaded machine.
## Modes
Each benchmark is runnable in three different time `mode`s:
- `fast`: 0.1-0.2s
- `norm`: 1-2s
- `slow`: 5-10s
You can control which mode to run by setting an additional `mode` variable for
`make`. The default is `mode=norm`. Example for `mode=fast`:
```
$ make clean
$ make boot mode=fast
$ make mode=fast
```
Note that the `mode`s set in `make boot` and `make` need to agree. Otherwise you
will get output errors, because `make boot` will generate input files for a
different `mode`. A more DRY way to control the `mode` would be
```
$ make clean
$ export mode=fast
$ make boot
$ make
```
As CPU architectures advance, the above running times may drift and
occasionally, all benchmarks will need adjustments.
Be aware that `nofib-analyse` will ignore the result if it falls below 0.2s.
This is the default of its `-i` option, which is of course incompatible with
`mode=fast`. In that case, you should just set `-i` as appropriate, even
deactivate it with `-i 0`.
## Configuration
There are some options you might want to tweak; search for nofib in
......@@ -50,6 +107,16 @@ To get instruction counts, memory reads/writes, and "cache misses",
you'll need to get hold of Cachegrind, which is part of
[Valgrind](http://valgrind.org).
You can then pass `-cachegrind` as `EXTRA_RUNTEST_OPTS`. Counting
instructions slows down execution by a factor of ~30. But it's
a deterministic metric, so you can combine it with `NoFibRuns=1`:
```
$ (make EXTRA_RUNTEST_OPTS="-cachegrind" NoFibRuns=1) 2>&1 | tee nofib-log
```
Optionally combine this with `mode=fast`, see [Modes](#modes).
## Extra Packages
Some benchmarks aren't run by default and require extra packages are
......@@ -59,11 +126,23 @@ installed for the GHC compiler being tested. These packages include:
## Adding benchmarks
If you add a benchmark try to set the problem sizes for
fast/normal/slow reasonably.
Runtimes for normal should be above 0.3s if that can be reasonably
achieved. Less than that and there is a chance
nofib-analyse will ignore the result if it falls below 0.2s.
fast/normal/slow reasonably. [Modes](#modes) lists the recommended brackets for
each mode.
### Stability wrt. GC paramerisations
Additionally, pay attention that your benchmarks are stable wrt. different
GC paramerisations, so that small changes in allocation don't lead to big,
unexplicable jumps in performance. See Trac #15999 for details. Also make sure
that you run the benchmark with the default GC settings, as enlarging Gen 0 or
Gen 1 heaps just amplifies the problem.
As a rule of thumb on how to ensure this: Make sure that your benchmark doesn't
just build up one big data and consume it in a final step, but rather that the
working set grows and shrinks (e.g. is approximately constant) over the whole
run of the benchmark. You can ensure this by iterating your main logic $n times
(how often depends on your program, but in the ball park of 100-1000).
You can test stability by plotting productivity curves for your `fast` settings
with the `prod.py` script attached to Trac #15999.
If in doubt, ask Sebastian Graf for help.
......@@ -428,12 +428,6 @@ Same issue with GHC.IO.Encoding.UTF8 as treejoin
Real suite
---------------------------------------
cacheprof
~~~~~~~~~
Successive runs with the same data can yield different allocation
totals, for some reason.
Reported at https://ghc.haskell.org/trac/ghc/ticket/8611
gg
~~
Same issue with GHC.IO.Encoding.UTF8 as treejoin
......
{-# LANGUAGE CPP #-}
module NofibUtils where
import Data.Char (ord)
import Data.List (foldl')
import System.Environment (getArgs)
-- | A very simple hash function so that we don't have to store and compare
-- huge output files.
hash :: String -> Int
hash = foldl' (\acc c -> ord c + acc*31) 0
-- | Using @salt xs@ on an loop-invariant @xs@ inside a loop prevents the
-- compiler from floating out the input parameter.
#ifdef __GLASGOW_HASKELL__
salt :: a -> IO a
salt = pure
{-# NOINLINE salt #-}
#else
salt :: [a] -> IO [a]
-- this won't work with real/lift, but I can't think of another way
salt xs = do
s <- length <$> getArgs
-- Invariant: There are less than 'maxBound' parameters passed to the
-- executable, otherwise this isn't really 'pure'
-- anymore.
pure (take (max (maxBound - 1) s) xs)
#endif
\ No newline at end of file
......@@ -7,6 +7,8 @@
import Data.Ratio
import System.Environment
import Control.Monad
import NofibUtils
-- powers = [[r^n | r<-[2..]] | n<-1..]
-- type signature required for compilers lacking the monomorphism restriction
......@@ -23,7 +25,7 @@ pascal:: [[Integer]]
pascal = [1,2,1] : map (\line -> zipWith (+) (line++[0]) (0:line)) pascal
bernoulli 0 = 1
bernoulli 1 = -(1%2)
bernoulli 1 = -(1%2)
bernoulli n | odd n = 0
bernoulli n =
(-1)%2
......@@ -33,8 +35,7 @@ bernoulli n =
| (k,combs)<- zip [2..n] pascal]
where powers = (neg_powers!!(n-1))
main = do
main = replicateM_ 500 $ do
[arg] <- getArgs
let n = (read arg)::Int
putStr $ "Bernoulli of " ++ (show n) ++ " is "
print (bernoulli n)
print (hash (show (bernoulli n)))
......@@ -5,9 +5,9 @@ include $(TOP)/mk/boilerplate.mk
# we don't want to include paraffins.c
SRCS=Main.hs
FAST_OPTS = 500
NORM_OPTS = 1200
SLOW_OPTS = 1200
FAST_OPTS = 60
NORM_OPTS = 180
SLOW_OPTS = 320
include $(TOP)/mk/target.mk
../../common/NofibUtils.hs
\ No newline at end of file
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
......@@ -3,6 +3,8 @@ Original program due to Dale Thurston, Aug 2001
> module Main where
> import System.Environment (getArgs)
> import Control.Monad (replicateM_)
> import NofibUtils (hash)
> type ContFrac = [Integer]
......@@ -33,14 +35,13 @@ digit regardless of what the input is; i.e., to see if the interval
Finally, we convert a continued fraction to digits by repeatedly multiplying by 10.
> toDigits :: ContFrac -> [Integer]
> toDigits (x:xs) = x:toDigits (ratTrans (10,0,0,1) xs)
> takeDigits :: Int -> ContFrac -> [Integer]
> takeDigits 0 _ = []
> takeDigits n (x:xs) = x:takeDigits (n-1) (ratTrans (10,0,0,1) xs)
> e :: [Integer]
> e = toDigits eContFrac
> e :: Int -> [Integer]
> e n = takeDigits n eContFrac
> main = do
> main = replicateM_ 100 $ do
> [digits] <- getArgs
> print (take (read digits) e)
> print (hash (show (e (read digits))))
......@@ -3,6 +3,6 @@ include $(TOP)/mk/boilerplate.mk
include $(TOP)/mk/target.mk
FAST_OPTS = 450
NORM_OPTS = 2000
SLOW_OPTS = 2200
FAST_OPTS = 50
NORM_OPTS = 150
SLOW_OPTS = 320
../../common/NofibUtils.hs
\ No newline at end of file
[2,7,1,8,2,8,1,8,2,8,4,5,9,0,4,5,2,3,5,3,6,0,2,8,7,4,7,1,3,5,2,6,6,2,4,9,7,7,5,7,2,4,7,0,9,3,6,9,9,9,5,9,5,7,4,9,6,6,9,6,7,6,2,7,7,2,4,0,7,6,6,3,0,3,5,3,5,4,7,5,9,4,5,7,1,3,8,2,1,7,8,5,2,5,1,6,6,4,2,7,4,2,7,4,6,6,3,9,1,9,3,2,0,0,3,0,5,9,9,2,1,8,1,7,4,1,3,5,9,6,6,2,9,0,4,3,5,7,2,9,0,0,3,3,4,2,9,5,2,6,0,5,9,5,6,3,0,7,3,8,1,3,2,3,2,8,6,2,7,9,4,3,4,9,0,7,6,3,2,3,3,8,2,9,8,8,0,7,5,3,1,9,5,2,5,1,0,1,9,0,1,1,5,7,3,8,3,4,1,8,7,9,3,0,7,0,2,1,5,4,0,8,9,1,4,9,9,3,4,8,8,4,1,6,7,5,0,9,2,4,4,7,6,1,4,6,0,6,6,8,0,8,2,2,6,4,8,0,0,1,6,8,4,7,7,4,1,1,8,5,3,7,4,2,3,4,5,4,4,2,4,3,7,1,0,7,5,3,9,0,7,7,7,4,4,9,9,2,0,6,9,5,5,1,7,0,2,7,6,1,8,3,8,6,0,6,2,6,1,3,3,1,3,8,4,5,8,3,0,0,0,7,5,2,0,4,4,9,3,3,8,2,6,5,6,0,2,9,7,6,0,6,7,3,7,1,1,3,2,0,0,7,0,9,3,2,8,7,0,9,1,2,7,4,4,3,7,4,7,0,4,7,2,3,0,6,9,6,9,7,7,2,0,9,3,1,0,1,4,1,6,9,2,8,3,6,8,1,9,0,2,5,5,1,5,1,0,8,6,5,7,4,6,3,7,7,2,1,1,1,2,5,2,3,8,9,7,8,4,4,2,5,0,5,6,9,5,3,6,9]
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
4882301751198926001
[2,7,1,8,2,8,1,8,2,8,4,5,9,0,4,5,2,3,5,3,6,0,2,8,7,4,7,1,3,5,2,6,6,2,4,9,7,7,5,7,2,4,7,0,9,3,6,9,9,9,5,9,5,7,4,9,6,6,9,6,7,6,2,7,7,2,4,0,7,6,6,3,0,3,5,3,5,4,7,5,9,4,5,7,1,3,8,2,1,7,8,5,2,5,1,6,6,4,2,7,4,2,7,4,6,6,3,9,1,9,3,2,0,0,3,0,5,9,9,2,1,8,1,7,4,1,3,5,9,6,6,2,9,0,4,3,5,7,2,9,0,0,3,3,4,2,9,5,2,6,0,5,9,5,6,3,0,7,3,8,1,3,2,3,2,8,6,2,7,9,4,3,4,9,0,7,6,3,2,3,3,8,2,9,8,8,0,7,5,3,1,9,5,2,5,1,0,1,9,0,1,1,5,7,3,8,3,4,1,8,7,9,3,0,7,0,2,1,5,4,0,8,9,1,4,9,9,3,4,8,8,4,1,6,7,5,0,9,2,4,4,7,6,1,4,6,0,6,6,8,0,8,2,2,6,4,8,0,0,1,6,8,4,7,7,4,1,1,8,5,3,7,4,2,3,4,5,4,4,2,4,3,7,1,0,7,5,3,9,0,7,7,7,4,4,9,9,2,0,6,9,5,5,1,7,0,2,7,6,1,8,3,8,6,0,6,2,6,1,3,3,1,3,8,4,5,8,3,0,0,0,7,5,2,0,4,4,9,3,3,8,2,6,5,6,0,2,9,7,6,0,6,7,3,7,1,1,3,2,0,0,7,0,9,3,2,8,7,0,9,1,2,7,4,4,3,7,4,7,0,4,7,2,3,0,6,9,6,9,7,7,2,0,9,3,1,0,1,4,1,6,9,2,8,3,6,8,1,9,0,2,5,5,1,5,1,0,8,6,5,7,4,6,3,7,7,2,1,1,1,2,5,2,3,8,9,7,8,4,4,2,5,0,5,6,9,5,3,6,9,6,7,7,0,7,8,5,4,4,9,9,6,9,9,6,7,9,4,6,8,6,4,4,5,4,9,0,5,9,8,7,9,3,1,6,3,6,8,8,9,2,3,0,0,9,8,7,9,3,1,2,7,7,3,6,1,7,8,2,1,5,4,2,4,9,9,9,2,2,9,5,7,6,3,5,1,4,8,2,2,0,8,2,6,9,8,9,5,1,9,3,6,6,8,0,3,3,1,8,2,5,2,8,8,6,9,3,9,8,4,9,6,4,6,5,1,0,5,8,2,0,9,3,9,2,3,9,8,2,9,4,8,8,7,9,3,3,2,0,3,6,2,5,0,9,4,4,3,1,1,7,3,0,1,2,3,8,1,9,7,0,6,8,4,1,6,1,4,0,3,9,7,0,1,9,8,3,7,6,7,9,3,2,0,6,8,3,2,8,2,3,7,6,4,6,4,8,0,4,2,9,5,3,1,1,8,0,2,3,2,8,7,8,2,5,0,9,8,1,9,4,5,5,8,1,5,3,0,1,7,5,6,7,1,7,3,6,1,3,3,2,0,6,9,8,1,1,2,5,0,9,9,6,1,8,1,8,8,1,5,9,3,0,4,1,6,9,0,3,5,1,5,9,8,8,8,8,5,1,9,3,4,5,8,0,7,2,7,3,8,6,6,7,3,8,5,8,9,4,2,2,8,7,9,2,2,8,4,9,9,8,9,2,0,8,6,8,0,5,8,2,5,7,4,9,2,7,9,6,1,0,4,8,4,1,9,8,4,4,4,3,6,3,4,6,3,2,4,4,9,6,8,4,8,7,5,6,0,2,3,3,6,2,4,8,2,7,0,4,1,9,7,8,6,2,3,2,0,9,0,0,2,1,6,0,9,9,0,2,3,5,3,0,4,3,6,9,9,4,1,8,4,9,1,4,6,3,1,4,0,9,3,4,3,1,7,3,8,1,4,3,6,4,0,5,4,6,2,5,3,1,5,2,0,9,6,1,8,3,6,9,0,8,8,8,7,0,7,0,1,6,7,6,8,3,9,6,4,2,4,3,7,8,1,4,0,5,9,2,7,1,4,5,6,3,5,4,9,0,6,1,3,0,3,1,0,7,2,0,8,5,1,0,3,8,3,7,5,0,5,1,0,1,1,5,7,4,7,7,0,4,1,7,1,8,9,8,6,1,0,6,8,7,3,9,6,9,6,5,5,2,1,2,6,7,1,5,4,6,8,8,9,5,7,0,3,5,0,3,5,4,0,2,1,2,3,4,0,7,8,4,9,8,1,9,3,3,4,3,2,1,0,6,8,1,7,0,1,2,1,0,0,5,6,2,7,8,8,0,2,3,5,1,9,3,0,3,3,2,2,4,7,4,5,0,1,5,8,5,3,9,0,4,7,3,0,4,1,9,9,5,7,7,7,7,0,9,3,5,0,3,6,6,0,4,1,6,9,9,7,3,2,9,7,2,5,0,8,8,6,8,7,6,9,6,6,4,0,3,5,5,5,7,0,7,1,6,2,2,6,8,4,4,7,1,6,2,5,6,0,7,9,8,8,2,6,5,1,7,8,7,1,3,4,1,9,5,1,2,4,6,6,5,2,0,1,0,3,0,5,9,2,1,2,3,6,6,7,7,1,9,4,3,2,5,2,7,8,6,7,5,3,9,8,5,5,8,9,4,4,8,9,6,9,7,0,9,6,4,0,9,7,5,4,5,9,1,8,5,6,9,5,6,3,8,0,2,3,6,3,7,0,1,6,2,1,1,2,0,4,7,7,4,2,7,2,2,8,3,6,4,8,9,6,1,3,4,2,2,5,1,6,4,4,5,0,7,8,1,8,2,4,4,2,3,5,2,9,4,8,6,3,6,3,7,2,1,4,1,7,4,0,2,3,8,8,9,3,4,4,1,2,4,7,9,6,3,5,7,4,3,7,0,2,6,3,7,5,5,2,9,4,4,4,8,3,3,7,9,9,8,0,1,6,1,2,5,4,9,2,2,7,8,5,0,9,2,5,7,7,8,2,5,6,2,0,9,2,6,2,2,6,4,8,3,2,6,2,7,7,9,3,3,3,8,6,5,6,6,4,8,1,6,2,7,7,2,5,1,6,4,0,1,9,1,0,5,9,0,0,4,9,1,6,4,4,9,9,8,2,8,9,3,1,5,0,5,6,6,0,4,7,2,5,8,0,2,7,7,8,6,3,1,8,6,4,1,5,5,1,9,5,6,5,3,2,4,4,2,5,8,6,9,8,2,9,4,6,9,5,9,3,0,8,0,1,9,1,5,2,9,8,7,2,1,1,7,2,5,5,6,3,4,7,5,4,6,3,9,6,4,4,7,9,1,0,1,4,5,9,0,4,0,9,0,5,8,6,2,9,8,4,9,6,7,9,1,2,8,7,4,0,6,8,7,0,5,0,4,8,9,5,8,5,8,6,7,1,7,4,7,9,8,5,4,6,6,7,7,5,7,5,7,3,2,0,5,6,8,1,2,8,8,4,5,9,2,0,5,4,1,3,3,4,0,5,3,9,2,2,0,0,0,1,1,3,7,8,6,3,0,0,9,4,5,5,6,0,6,8,8,1,6,6,7,4,0,0,1,6,9,8,4,2,0,5,5,8,0,4,0,3,3,6,3,7,9,5,3,7,6,4,5,2,0,3,0,4,0,2,4,3,2,2,5,6,6,1,3,5,2,7,8,3,6,9,5,1,1,7,7,8,8,3,8,6,3,8,7,4,4,3,9,6,6,2,5,3,2,2,4,9,8,5,0,6,5,4,9,9,5,8,8,6,2,3,4,2,8,1,8,9,9,7,0,7,7,3,3,2,7,6,1,7,1,7,8,3,9,2,8,0,3,4,9,4,6,5,0,1,4,3,4,5,5,8,8,9,7,0,7,1,9,4,2,5,8,6,3,9,8,7,7,2,7,5,4,7,1,0,9,6,2,9,5,3,7,4,1,5,2,1,1,1,5,1,3,6,8,3,5,0,6,2,7,5,2,6,0,2,3,2,6,4,8,4,7,2,8,7,0,3,9,2,0,7,6,4,3,1,0,0,5,9,5,8,4,1,1,6,6,1,2,0,5,4,5,2,9,7,0,3,0,2,3,6,4,7,2,5,4,9,2,9,6,6,6,9,3,8,1,1,5,1,3,7,3,2,2,7,5,3,6,4,5,0,9,8,8,8,9,0,3,1,3,6,0,2,0,5,7,2,4,8,1,7,6,5,8,5,1,1,8,0,6,3,0,3,6,4,4,2,8,1,2,3,1,4,9,6,5,5,0,7,0,4,7,5,1,0,2,5,4,4,6,5,0,1,1,7,2,7,2,1,1,5,5,5,1,9,4,8,6,6,8,5,0,8,0,0,3,6,8,5,3,2,2,8,1,8,3,1,5,2,1,9,6,0,0,3,7,3,5,6,2,5,2,7,9,4,4,9,5,1,5,8,2,8,4,1,8,8,2,9,4,7,8,7,6,1,0,8,5,2,6,3,9,8,1,3,9,5,5,9,9,0,0,6,7,3,7,6,4,8,2,9,2,2,4,4,3,7,5,2,8,7,1,8,4,6,2,4,5,7,8,0,3,6,1,9,2,9,8,1,9,7,1,3,9,9,1,4,7,5,6,4,4,8,8,2,6,2,6,0,3,9,0,3,3,8,1,4,4,1,8,2,3,2,6,2,5,1,5,0,9,7,4,8,2,7,9,8,7,7,7,9,9,6,4,3,7,3,0,8,9,9,7,0,3,8,8,8,6,7,7,8,2,2,7,1,3,8,3,6,0,5,7,7,2,9,7,8,8,2,4,1,2,5,6,1,1,9,0,7,1,7,6,6,3,9,4,6,5,0,7,0,6,3,3,0,4,5,2,7,9,5,4,6,6,1,8,5,5,0,9,6,6,6,6,1,8,5,6,6,4,7,0,9,7,1,1,3,4,4,4,7,4,0,1,6]
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337
-3940360796357121337