GitLab

Runtime for nofib benchmarks was all over the place.
This patch adjusts runtime for most benchmarks such
that it falls into the 0.2-2s range.

This means that:
* A default run will take longer
* Time spent will be better distributed among benchmarks.
* More benchmarks have runtimes long enough to be used
  for runtime analysis.

Some more changes were done which go hand in hand
with changing runtimes.
* Some benchmarks now create their input files during boot.
* Moved input files for anna in their own directory.
* Remove printing of output for some of the floating
  point heavy benchmarks.
* Added a comment about desired runtimes to README.
* Made grep actually benchmark something.
* Throw cachgrind out of the default benchmarks.
  The nondeterministic behaviour has been an issue for a
  while and it doesn't seem like an essential benchmark.

Test Plan: run nofib in modes slow/normal/fast

Reviewers: O26 nofib, alpmestan

Reviewed By: alpmestan

Subscribers: sgraf, alpmestan

Differential Revision: https://phabricator.haskell.org/D4989

Summary: This removes a lot of noise from the nofib output.

Test Plan: running nofib

Reviewers: O26 nofib, alpmestan

Reviewed By: alpmestan

Subscribers: alpmestan

Differential Revision: https://phabricator.haskell.org/D4952

Summary:
The next step of the MonadFail Proposal broke nofib's lambda benchmark.
This commit fixes that in an unintrusive way.

Reviewers: O26 nofib, RyanGlScott, bgamari

Reviewed By: RyanGlScott

Subscribers: monoidal

Differential Revision: https://phabricator.haskell.org/D5058

We set -O2 in NoFibHcOpts which is then applied to all benchmarks run.

Adding -O2 in individual benchmarks is therefore redundant.
It also leads to issues when testing performance flags via
EXTRA_HC_OPTS.
Individual Makefiles attach -O2 last. This means all
flags set by -O2 could not be disabled with -fno-<flag> as they
were reenabled by the later -O2 switch.

Test Plan: Using

Reviewers: bgamari, jmct, O26 nofib

Differential Revision: https://phabricator.haskell.org/D4829

Reviewers: michalt, bgamari, O26 nofib, goldfire

Reviewed By: michalt, O26 nofib

Differential Revision: https://phabricator.haskell.org/D4045

Reviewers: O26 nofib, michalt

Reviewed By: O26 nofib, michalt

Subscribers: michalt

Differential Revision: https://phabricator.haskell.org/D4391

These use putStrLn so clearly we are not producing binary data.
This caused failures on windows.

Reviewers: O26 nofib, bgamari

Reviewed By: bgamari

Subscribers: bgamari

GHC Trac Issues: #14656

Differential Revision: https://phabricator.haskell.org/D4705

Summary: This makes it easier to run nofib using nix.

Test Plan: Using it.

Reviewers: O26 nofib, bgamari

Reviewed By: bgamari

Differential Revision: https://phabricator.haskell.org/D4756

Summary:
My msys instance at least reports MINGW instead of MSYS for uname.
This commit adjusts the regex to match either one.

Without this some errors occur because of missing .exe extensions in
file names.

This fixes #14654

Test Plan: make

Reviewers: O26 nofib, Phyx, bgamari

Reviewed By: Phyx, bgamari

Subscribers: bgamari, Phyx

GHC Trac Issues: #14654

Differential Revision: https://phabricator.haskell.org/D4297

Reviewers: O26 nofib

Differential Revision: https://phabricator.haskell.org/D4320

if I read http://gcc.gnu.org/onlinedocs/gcc/Inline.html correctly.

Also, clean generated files.

so that it builds with GHC-8.3 post 8ae263c.

See https://phabricator.haskell.org/D3030#98590.

The current master uses the diff alias for comparison, which pollutes
stdout. I fixed the tr -d '\r' calls that previously were in place, but
broken due to escaping. I also did the same for golden master generation
in shootout Makefiles.

Test Plan:

```
make clean && \
make NoFibRuns=1 2>&1 > nofib.log && \
nofib-analyse/nofib-analyse nofib.log
```

Differential Revision: https://phabricator.haskell.org/D3450

Summary:
It's not required by the POSIX specification and OS X doesn't support
it; instead use uname -s.

Test Plan: V

Reviewers: O26 nofib, michalt, mpickering

Reviewed By: O26 nofib, michalt, mpickering

Subscribers: mpickering

GHC Trac Issues: #13711

Differential Revision: https://phabricator.haskell.org/D3594

Summary:
They are originally from
https://github.com/AndrasKovacs/misc-stuff/blob/master/haskell/Eff/EffBench.hs

They show interesting interactions with call arity, spec constr and SAT.

Reviewers: O26 nofib, michalt, simonpj, bgamari

Reviewed By: bgamari

Subscribers: RyanGlScott

GHC Trac Issues: #13892

Differential Revision: https://phabricator.haskell.org/D3683

Reviewers: O26 nofib, michalt

Reviewed By: O26 nofib, michalt

Subscribers: michalt

Differential Revision: https://phabricator.haskell.org/D3730

Summary:
By following the naming conventions of the build system we can
simplify the `Makefile`s a bit.  This patch also avoids having to
explicitly pass the flags to `runstdtest` for `fasta`, `k-nucleotide`
and `reverse-complement`.  The only consquence of this is that it's a
requirement to run `make boot` before `make` for these benchmarks. But
that is already the case since `.depend` files are generated by with
`make boot`.

Finally, this also update the `.gitignore` with the new names of
the output/input files.
Signed-off-by: Michal Terepeta <michal.terepeta@gmail.com>

Test Plan: build & run

Reviewers: bgamari

Reviewed By: bgamari

Differential Revision: https://phabricator.haskell.org/D3412

Summary:
This creates `runtime_files` subdirectory for benchmarks that need
some files at runtime. This make it clear what files are actually
needed to run the benchmarks and makes it much easier to support
building/running them out of separate build directory.

This affects the following benchmarks:
- real/anna
- real/cacheprof
- real/fluid
- real/hidden
- real/maillist
- real/prolog
- real/scs
- spectral/expert
- spectral/mate
- spectral/para
- spectral/treejoin
Signed-off-by: Michal Terepeta <michal.terepeta@gmail.com>

Test Plan: build & run

Reviewers: bgamari

Reviewed By: bgamari

Subscribers: snowleopard

Differential Revision: https://phabricator.haskell.org/D3411

I've mistakenly added it as a workaround to build problems, but the
whole problem was only due a missing `.depend` file (which is created
by `make boot` and interestigly is not cleaned by `make clean`). In
any way, the `HS_SRCS` is simply unnecessary and can be removed.
Signed-off-by: Michal Terepeta <michal.terepeta@gmail.com>

Test Plan: build & run

Reviewers: bgamari

Reviewed By: bgamari

Differential Revision: https://phabricator.haskell.org/D3389

By simply renaming the expected output files to follow the usual
conventions, we can simplify the `Makefile` a bit.
Signed-off-by: Michal Terepeta <michal.terepeta@gmail.com>

Test Plan: build & run

Reviewers: bgamari

Reviewed By: bgamari

Differential Revision: https://phabricator.haskell.org/D3366

The benchmark only needed a small update to the expected stdandard
output file (missing newline).
Signed-off-by: Michal Terepeta <michal.terepeta@gmail.com>

Test Plan: build & run

Reviewers: dfeuer, bgamari

Reviewed By: dfeuer

Differential Revision: https://phabricator.haskell.org/D3363

Summary:
This removes the unnecessary `OTHER_SUBDIRS` variable, since the
subdirectories mentioned there are already in `SUBDIRS` (the
benchmarks have been enabled in previous commits).
Signed-off-by: Michal Terepeta <michal.terepeta@gmail.com>

Test Plan: build & run

Reviewers: bgamari, dfeuer

Reviewed By: dfeuer

Differential Revision: https://phabricator.haskell.org/D3342

Summary:
Because it's at the beginning of the line with `SRC_HC_OPTS`, it
causes `make` ignore those options. This commit removes that character
and makes everything work.
Signed-off-by: Michal Terepeta <michal.terepeta@gmail.com>

Test Plan: manually verify that the options are passed to GHC

Reviewers: bgamari, dfeuer

Reviewed By: dfeuer

Subscribers: dfeuer

Differential Revision: https://phabricator.haskell.org/D3344

Summary:
Unfortunately `secretary` requires `random` package to compile and
this broke perf.haskell.org. Let's disable the benchmark for now until
we have a good story for dealing with benchmarks that require
packages.
Signed-off-by: Michal Terepeta <michal.terepeta@gmail.com>

Test Plan: build & run

Reviewers: bgamari, nomeata

Reviewed By: nomeata

Differential Revision: https://phabricator.haskell.org/D3347

Summary:
It doesn't compile, nobody is running it and upstream seems to have
abandoned it too. IMHO, at this point we should simply remove it and
focus on improving the benchmarks that do work & adding new ones.

See also the discussion in #11501
Signed-off-by: Michal Terepeta <michal.terepeta@gmail.com>

Test Plan: compile & run nofib

Reviewers: bgamari

Subscribers:

The benchmark hasn't been enabled and doesn't compile. Fixing it is
easy, but it runs very quickly with the default parameter `5` (takes
~0.04s). When I bumped the parameter to `6`, it didn't finish within
~90s CPU time (and 5GiB of RAM).

So it seems unlikely to be useful - let's just remove it.
Signed-off-by: Michal Terepeta <michal.terepeta@gmail.com>

Test Plan: run nofib

Reviewers: dfeuer, bgamari

Reviewed By: bgamari

Subscribers: dfeuer

Differential Revision: https://phabricator.haskell.org/D3085

It's been disabled for a long time and there's no input to actually
run it.
Signed-off-by: Michal Terepeta <michal.terepeta@gmail.com>

Test Plan: build & run

Reviewers: bgamari

Differential Revision: https://phabricator.haskell.org/D3330

The benchmark doesn't compile and was not enabled. I tried fixing it,
but it seems to take excessive amount of time & memory (didn't finish
in 60s, which required over 10GiB of RAM). Sounds like another
candidate for removal.
Signed-off-by: Michal Terepeta <michal.terepeta@gmail.com>

Test Plan: build & run

Reviewers: bgamari

Differential Revision: https://phabricator.haskell.org/D3329

This should fix the benchmark by:
- Not importing `IOExts`.
- Using `randomRs` with a predictable seed instead of `randomRIOs` to
  make the runs reproducible (the latter is using global RNG).
- Bumping one of the parameter to make it run for a bit longer (2s
  instead of 0.4s).
Signed-off-by: Michal Terepeta <michal.terepeta@gmail.com>

Test Plan: build & run

Reviewers: bgamari

Differential Revision: https://phabricator.haskell.org/D3328