Skip to content

Performance regressions in the benchmark suite of effectful between 9.6.2 and 9.8.0.20230727

Steps to reproduce:

  1. git clone https://github.com/haskell-effectful/effectful.git
  2. cd effectful
  3. git checkout 7f8299d77831fa056c13ab6f284afbaf28067854
  4. Add allow-newer: * to cabal.project.
  5. cabal run -w ghc-9.6.2 bench -- --csv baseline.csv
  6. cabal run -w ghc-9.8.0.20230727 bench -- --baseline baseline.csv

Somewhat similar to #22758 (closed), only this time it's only benchmarks from the concurrency group:

All
  concurrency
    shallow
      unmask 1x
        async (IO):                         OK (0.14s)
          258  ns ±  25 ns, 1.4 KB allocated,   1 B  copied, 7.0 MB peak memory,  9% less than baseline
        async (Eff):                        OK (0.14s)
          488  ns ±  48 ns, 2.5 KB allocated,   2 B  copied, 7.0 MB peak memory,       same as baseline
        Fork (localUnliftIO/withLiftMapIO): OK (0.19s)
          717  ns ±  43 ns, 3.5 KB allocated,   3 B  copied, 7.0 MB peak memory,       same as baseline
        Fork (localUnlift/withLiftMap):     OK (0.12s)
          909  ns ±  87 ns, 4.5 KB allocated,   4 B  copied, 7.0 MB peak memory,       same as baseline
        Fork (localLiftUnliftIO):           OK (0.12s)
          838  ns ±  82 ns, 4.0 KB allocated,  51 B  copied, 7.0 MB peak memory,       same as baseline
        Fork (localLiftUnlift):             OK (0.15s)
          1.13 μs ±  83 ns, 6.1 KB allocated, 103 B  copied, 7.0 MB peak memory,       same as baseline
      unmask 10x
        async (IO):                         OK (0.18s)
          332  ns ±  21 ns, 1.6 KB allocated,   1 B  copied, 7.0 MB peak memory,  9% less than baseline
        async (Eff):                        OK (0.23s)
          868  ns ±  45 ns, 4.0 KB allocated,   4 B  copied, 7.0 MB peak memory, 39% more than baseline
        Fork (localUnliftIO/withLiftMapIO): OK (0.15s)
          1.12 μs ±  87 ns, 5.1 KB allocated,   5 B  copied, 7.0 MB peak memory, 24% more than baseline
        Fork (localUnlift/withLiftMap):     OK (0.19s)
          1.40 μs ±  88 ns, 6.4 KB allocated,   8 B  copied, 7.0 MB peak memory, 18% more than baseline
        Fork (localLiftUnliftIO):           OK (0.14s)
          2.02 μs ± 169 ns,  40 KB allocated,  78 B  copied,  10 MB peak memory, 29% more than baseline
        Fork (localLiftUnlift):             OK (0.19s)
          2.77 μs ± 207 ns,  44 KB allocated, 130 B  copied,  10 MB peak memory, 18% more than baseline
      unmask 100x
        async (IO):                         OK (0.20s)
          1.47 μs ±  97 ns,  36 KB allocated,  40 B  copied,  10 MB peak memory, 10% less than baseline
        async (Eff):                        OK (0.15s)
          4.47 μs ± 401 ns,  52 KB allocated,  55 B  copied,  10 MB peak memory, 69% more than baseline
        Fork (localUnliftIO/withLiftMapIO): OK (0.18s)
          5.36 μs ± 331 ns,  53 KB allocated,  56 B  copied,  10 MB peak memory, 62% more than baseline
        Fork (localUnlift/withLiftMap):     OK (0.21s)
          6.26 μs ± 400 ns,  57 KB allocated,  56 B  copied,  10 MB peak memory, 51% more than baseline
        Fork (localLiftUnliftIO):           OK (0.17s)
          10.1 μs ± 844 ns,  74 KB allocated,  97 B  copied,  10 MB peak memory, 20% more than baseline
        Fork (localLiftUnlift):             OK (0.26s)
          15.4 μs ± 768 ns, 101 KB allocated, 204 B  copied,  10 MB peak memory, 11% more than baseline
    deep
      unmask 1x
        async (Eff):                        OK (0.14s)
          2.03 μs ± 182 ns,  15 KB allocated,  17 B  copied,  10 MB peak memory,       same as baseline
        Fork (localUnliftIO/withLiftMapIO): OK (0.15s)
          2.21 μs ± 184 ns,  16 KB allocated,  25 B  copied,  10 MB peak memory,       same as baseline
        Fork (localUnlift/withLiftMap):     OK (0.17s)
          2.56 μs ± 178 ns,  18 KB allocated,  40 B  copied,  10 MB peak memory,       same as baseline
        Fork (localLiftUnliftIO):           OK (0.16s)
          2.34 μs ± 185 ns,  16 KB allocated,  66 B  copied,  10 MB peak memory,       same as baseline
        Fork (localLiftUnlift):             OK (0.20s)
          2.97 μs ± 178 ns,  22 KB allocated, 130 B  copied,  10 MB peak memory,       same as baseline
      unmask 10x
        async (Eff):                        OK (0.16s)
          2.38 μs ± 167 ns,  16 KB allocated,  26 B  copied,  10 MB peak memory,       same as baseline
        Fork (localUnliftIO/withLiftMapIO): OK (0.18s)
          2.67 μs ± 168 ns,  17 KB allocated,  37 B  copied,  10 MB peak memory,       same as baseline
        Fork (localUnlift/withLiftMap):     OK (0.21s)
          3.06 μs ± 186 ns,  20 KB allocated,  43 B  copied,  10 MB peak memory,  8% more than baseline
        Fork (localLiftUnliftIO):           OK (0.23s)
          3.47 μs ± 174 ns,  52 KB allocated, 122 B  copied,  10 MB peak memory, 11% more than baseline
        Fork (localLiftUnlift):             OK (0.15s)
          4.46 μs ± 394 ns,  60 KB allocated, 186 B  copied,  10 MB peak memory,       same as baseline
      unmask 100x
        async (Eff):                        OK (0.20s)
          5.76 μs ± 402 ns,  64 KB allocated,  63 B  copied,  10 MB peak memory, 38% more than baseline
        Fork (localUnliftIO/withLiftMapIO): OK (0.23s)
          6.85 μs ± 386 ns,  66 KB allocated,  80 B  copied,  10 MB peak memory, 39% more than baseline
        Fork (localUnlift/withLiftMap):     OK (0.13s)
          7.67 μs ± 689 ns,  71 KB allocated,  92 B  copied,  10 MB peak memory, 29% more than baseline
        Fork (localLiftUnliftIO):           OK (0.20s)
          11.6 μs ± 811 ns,  87 KB allocated, 176 B  copied,  10 MB peak memory, 16% more than baseline
        Fork (localLiftUnlift):             OK (0.15s)
          17.3 μs ± 1.5 μs, 117 KB allocated, 318 B  copied,  10 MB peak memory, 13% more than baseline
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information