Draft: Fix #8045 - Move I/O manager benchmarks into the GHC tree

This MR replaces !4955 (closed).

The following points are relevant for this MR:

• Relevant changes in the base library have been made for Event-related modules

• All original benchmark programs have been included in the branch for comparison as the some of these required extensive updates/changes for them to be able to work with the newer GHC versions and/or libraries.

• The following libraries need to be locally installed:

  • network-v3.1.2.1
  • attoparsec-0.14.1
  • ghc-prim-0.7.0
  • hashable-1.3.2.0
  • integer-logarithms-1.0.3.1
  • primitive-0.7.1.0
  • scientific-0.3.7.0

  and minor changes need to be made in the above libraries for compatibility with GHC 9+. The steps outlined at https://gitlab.haskell.org/ghc/ghc/-/wikis/debugging/installing-packages-inplace under the plan B heading are useful. After this, the makefile can with make and the benchmarks.cabal file can be run as: cabal build --with-compiler=$topdir/inplace/bin/ghc-stage2 --package-db=$topdir/inplace/lib/package.conf.d

• All Bash shell and Python scripts have been updated for Python 3.8.5 and Bash 5+

• Example outputs for the bench-timers.sh and bench-thread-delay.sh scripts have been added for reference.

mentioned in issue #8045

added 1 commit

• 8eb25aef - Change network submodule url to GitLab copy (will need to be updated later)

Compare with previous version

changed title from FIx #8045 - Move I/O manager benchmarks into the GHC tree to Fix #8045 - Move I/O manager benchmarks into the GHC tree

added 87 commits

• 8eb25aef...633bbc1f - 35 commits from branch master
• 686e4dc1 - Modify Event module exports
• 4cc733d7 - Modify Event.Poll module exports
• 44d971a9 - Modify .cabal for base library to expose Event-related modules
• 46384efe - Move GHC.Event.PSQ module out of os conditional block to avoid compilation errors
• 495d9330 - Add LICENSE for benchmarking utilities
• 1551838d - Add Timers benchmarking utility
• ed99351f - Timers: Update imports; add Semigroup instance for Config
• 775c8de4 - Add Args benchmarking utility
• 0a1ccc0d - Args: Clean up imports, code clean up
• e8d7badf - Add C file for DeadConn benchmarking utility
• e0a27be8 - Add DeadConn benchmarking utility
• e3d12dc7 - DeadConn: Add Semigroup instance for Monoid Config; update Socket function calls
• 1c136363 - Add EventFile benchmarking utility
• bc1379b8 - EventFile: Update imports; make minor formatting improvements
• 6fd60cc1 - Add EventHttp benchmarking utility
• f1eeeb94 - EventHttp: Update imports, Socket functions, variables to avoid unuse, shadow errors
• fc4b5d75 - Add EventUtil benchmarking utility
• 6f7bfb85 - EventUtil: Improve exception-catching function type signatures
• 452c73f5 - Add EventSocket benchmarking utility
• c22356fe - EventSocket: Update imports, functions, types for Socket-related functions
• 4bcfe90f - Add HttpClient benchmarking utility
• 0d2d9922 - HttpClient: Update function names, imports, time types, Config Semigroup instance
• da2b1171 - Add IntMap benchmarking utility
• 19ee3c58 - IntMap: Update import names
• 355101df - IntMap: Update function for correct IntMap insertWith return type
• d1065c54 - Add PongServer benchmarking utility
• 3138b72e - PongServer: Update imports, Socket functions, Semigroup instance for Config
• bfbe6b55 - Add PSQ benchmarking utility
• e8cb0750 - PSQ: Update module imports for GHC.Event
• 54664ffe - Add NoPush interface benchmarking utility
• 43cb13bd - NoPush: Update imports, Socket type and withFdSocket function
• d173b440 - Add RFC benchmarking utility
• e828dc93 - RFC: Update imports, qualify function calls with module names
• bcfa1ef8 - Add Signal benchmarking utility
• a77c756c - Export emState record field to manage (read for Signal) EventManager state
• bcc32b96 - Signal: Update imports, functions to manage signals (System.Posix.Signals)
• a30bff83 - Add Simple benchmarking utility
• 44d07182 - Simple: Update imports, Semigroup instance for Config
• 843b8ad0 - Add StaticHttp benchmarking utility
• ecac08a2 - StaticHttp: Update imports, functions for Socket, string append
• 89f61238 - Add ThreadDelay benchmarking utility
• 4bade9bf - ThreadDelay: Update imports, add Semigroup instance for Config
• 75cea86e - Add updated Python script for setting up benchmarking environment via using sysctl
• 41959241 - Add original bench-thread-delay shell script
• 355a46f0 - Add original bench-timers shell script
• 755b312a - Add .cabal for building benchmark executables
• 6bc32008 - Add updated Makefile for generating benchmark executables
• 13491e0c - Update Makefile targets for cleaning up data, program executables
• 019c10f8 - Update gnuplot options to generate events vs. threads benchmarking plot
• 5b17fdcc - Update gnuplot options to generate thread-delay benchmarking plot for GHC I/O manager
• b35ca958 - An example output for the bench-timers.sh script
• d3cafa72 - An example output for the bench-thread-delay.sh script

Compare with previous version

changed the description

added 24 commits

• 526c9505 - PongServer: Update imports, Socket functions, Semigroup instance for Config
• 1f95af2e - Add PSQ benchmarking utility
• e4d1d859 - PSQ: Update module imports for GHC.Event
• 645cff7d - Add NoPush interface benchmarking utility
• ab77d308 - NoPush: Update imports, Socket type and withFdSocket function
• 5dea97b6 - Add RFC benchmarking utility
• 7d6143d6 - RFC: Update imports, qualify function calls with module names
• f19486b3 - Add Signal benchmarking utility
• 5b7e996a - Export emState record field to manage (read for Signal) EventManager state
• 825077bb - Signal: Update imports, functions to manage signals (System.Posix.Signals)
• 601e8650 - Add Simple benchmarking utility
• e08ccf17 - Simple: Update imports, Semigroup instance for Config
• 87a778c9 - Add StaticHttp benchmarking utility
• 3f3aa978 - StaticHttp: Update imports, functions for Socket, string append
• 93fdfde2 - Add ThreadDelay benchmarking utility
• 2914c1da - ThreadDelay: Update imports, add Semigroup instance for Config
• 4f42d86a - Add updated Python script for setting up benchmarking environment via using sysctl
• 22625615 - Add original bench-thread-delay shell script
• 94108d62 - Add original bench-timers shell script
• ae78f6b5 - Add .cabal for building benchmark executables
• a92b3068 - Add updated Makefile for generating benchmark executables
• 01fb4f16 - Update Makefile targets for cleaning up data, program executables
• c6170c46 - Update gnuplot options to generate events vs. threads benchmarking plot
• 36b775a8 - Update gnuplot options to generate thread-delay benchmarking plot for GHC I/O manager

Compare with previous version

added 1 commit

• 332ee342 - testsuite: Use appropriate threadDelay in T16916

Compare with previous version

Annoyingly, I'm not sure whether we can just export these modules without involving the @core-libraries committee...

I approve of exporting them. @chessai you agree?

+1

I'm not so eager to expose more internal GHC state to users. It's also not quite clear to me why a benchmark that's supposed to be testing external interfaces needs internal interfaces that most users today can't access.

If these need to be exported then I argue that a GHC API entry need to be defined for them so these internal interfaces don't become ABI.

If a GHC API entry needs to be created, I'll need some help for the task as I have only basic familiarity with the API.

changed this line in version 8 of the diff

I'm afraid I can't review any of the specifics here. Maybe @Phyx can do a better job of reviewing here or appoint someone who can?

We recently moved a number a benchmarks that depend on external packages out of the tree. So I'm not sure if that's the right step?

@mpickering looked into that. Maybe he can comment.

I agree with you here. I don't like increasing the dependencies on GHC/Base. In particular I'm weary about the network dependency.

Most Haskell packages have a long, long dependency chain and these not only have an impact on compile time but but they also have an impact on debugging. This adds an aweful lot of dependencies..

Is there any reason these can't leave out of tree like nofib?

@Phyx As described in the original issue #8045:

When we developed the scalable I/O manager (i.e. the first version that used epoll) we wrote a bunch of benchmarks that were never moved to the GHC tree or integrated into the make system:

https://github.com/tibbe/event/tree/master/benchmarks

It would be nice to move this into the GHC tree to catch future regressions and use them when trying to make improvements.

the benchmarks have been put in the GHC tree. A decision on whether they should be left in tree or otherwise needs to be made. Since @mpickering had last commented on the original issue, his input would help?

That still does not explain why they need access to internal structures. Are they micro-benchmarks for specific parts?

Yes.

added 1 commit

• f0276597 - Move event benchmarks to utils/benchmarks/events/

Compare with previous version

added 144 commits

• f0276597...41d6cfc4 - 93 commits from branch master
• 68cd40a1 - Modify Event module exports
• 0a695ab0 - Modify Event.Poll module exports
• 630da3d0 - Add LICENSE for benchmarking utilities
• bf816128 - Add Timers benchmarking utility
• 23be8460 - Timers: Update imports; add Semigroup instance for Config
• 91511464 - Change Event.TimerManager looping function name to avoid clash with Event.Manager function
• 527fc71b - Add Args benchmarking utility
• 02fd8eb4 - Args: Clean up imports, code clean up
• b3343a80 - Add C file for DeadConn benchmarking utility
• 075df5c7 - Add DeadConn benchmarking utility
• 9c36288b - DeadConn: Add Semigroup instance for Monoid Config; update Socket function calls
• 88900a38 - Add EventFile benchmarking utility
• 94ea2991 - EventFile: Update imports; make minor formatting improvements
• 6107ae67 - Add EventHttp benchmarking utility
• 665dbe9e - EventHttp: Update imports, Socket functions, variables to avoid unuse, shadow errors
• 2c22e6d8 - Add EventUtil benchmarking utility
• 99763fac - EventUtil: Improve exception-catching function type signatures
• 20b081de - Add EventSocket benchmarking utility
• 03ac18af - EventSocket: Update imports, functions, types for Socket-related functions
• 105c0f62 - Add HttpClient benchmarking utility
• c9e887a9 - HttpClient: Update function names, imports, time types, Config Semigroup instance
• 3d9b165c - Add IntMap benchmarking utility
• df024842 - IntMap: Update import names
• 2fd62ea9 - IntMap: Update function for correct IntMap insertWith return type
• ec591dac - Add PongServer benchmarking utility
• dc118844 - PongServer: Update imports, Socket functions, Semigroup instance for Config
• dff19b19 - Add PSQ benchmarking utility
• 4d4f91ea - PSQ: Update module imports for GHC.Event
• ed53a5bf - Add NoPush interface benchmarking utility
• a78f8eb1 - NoPush: Update imports, Socket type and withFdSocket function
• 6a0571df - Add RFC benchmarking utility
• fff80181 - RFC: Update imports, qualify function calls with module names
• 94aa0820 - Add Signal benchmarking utility
• efcc06b9 - Export emState record field to manage (read for Signal) EventManager state
• e01e16a7 - Signal: Update imports, functions to manage signals (System.Posix.Signals)
• 825bd5b9 - Add Simple benchmarking utility
• a0b2706c - Simple: Update imports, Semigroup instance for Config
• 91f486c5 - Add StaticHttp benchmarking utility
• 55d5b836 - StaticHttp: Update imports, functions for Socket, string append
• 9fcd63a3 - Add ThreadDelay benchmarking utility
• 20e11f3e - ThreadDelay: Update imports, add Semigroup instance for Config
• 2c39d37a - Add updated Python script for setting up benchmarking environment via using sysctl
• c00f096d - Add original bench-thread-delay shell script
• 9ec5caaf - Add original bench-timers shell script
• 445bc4a4 - Add .cabal for building benchmark executables
• 7ca72263 - Add updated Makefile for generating benchmark executables
• c841af50 - Update Makefile targets for cleaning up data, program executables
• aa5ba1f8 - Update gnuplot options to generate events vs. threads benchmarking plot
• b9d9b40a - Update gnuplot options to generate thread-delay benchmarking plot for GHC I/O manager
• de9fedc3 - testsuite: Use appropriate threadDelay in T16916
• c708b969 - Move event benchmarks to utils/benchmarks/events/

Compare with previous version

@Phyx @sgraf812 I have reverted exposing modules in libraries/base/GHC/Event/ and so now, all types, functions, constructors needed to run the benchmark programs are only exported via libraries/base/GHC/Event.hs . I have also updated the imports in the benchmark programs accordingly.

Thanks, I like this better than the exposing of the modules.

I'll leave the rest up to the good folks here to decide.

Sorry for the delay

@mpickering Your input is needed to finalize work on this issue related to the following:

• figuring out if the event benchmarks should be out of the GHC tree similar to nofib
• checking if the most recent change in exporting all necessary functions etc via libraries/base/GHC/Event.hs is the way to go
• or if the GHC API needs to be used for access to entities in the Event module.

closed

assigned to @alinab

Unless someone is actively working on #18380, it would be good to go over GHC.Event.TimerManager and find out refactor code as needed.

reopened

assigned to @mpickering

unassigned @alinab

Yes I think these benchmarks should be out the tree like the nofib ones. We don't want to add all these dependencies.

1. Make a new repo in the ghc/ namespace.
2. Cabalise the benchmarks, so they can be run using cabal run.
3. I will add support to head.hackage to build the benchmarks suite in the nightly CI run so they don't bitrot.

Let me know if you need any help with that.

I think exporting the extra parts is fine but others seem to have a stronger opinion.

assigned to @alinab and unassigned @mpickering

Marking this MR as draft for now as there are things left to do. Please change it back when you want this back in the review queue!

marked this merge request as draft