Agda seems pretty slow to compile

Whilst working on #20294 (closed) I noticed that Agda was very slow to compile. I built a profiled HEAD and here's the report of the -p profile.

The profile isn't very illuminating really but I just post it here anyway: ghc.prof

        total time  =      706.42 secs   (706418 ticks @ 1000 us, 1 processor)
        total alloc = 1,038,335,597,920 bytes  (excludes profiling overheads)

COST CENTRE              MODULE                 SRC                                                  %time %alloc

Simplify                 GHC.Core.Opt.Pipeline  compiler/GHC/Core/Opt/Pipeline.hs:487:34-54           23.5   26.9
Core2Core                GHC.Driver.Main        compiler/GHC/Driver/Main.hs:1501:7-55                 23.1   26.8
simplIF                  GHC.Core.Opt.Simplify  compiler/GHC/Core/Opt/Simplify.hs:1148:61-79           8.4    8.3
deSugar                  GHC.Driver.Main        compiler/GHC/Driver/Main.hs:623:7-44                   6.5    2.4
simplRecOrTopPair-normal GHC.Core.Opt.Simplify  compiler/GHC/Core/Opt/Simplify.hs:(306,5)-(307,62)     2.7    1.9
OccAnal                  GHC.Core.Opt.Pipeline  compiler/GHC/Core/Opt/Pipeline.hs:(713,22)-(714,42)    2.6    2.3
tcRnIface                GHC.Driver.Main        compiler/GHC/Driver/Main.hs:1061:19-94                 2.5    4.4
simplCast-simplCoercion  GHC.Core.Opt.Simplify  compiler/GHC/Core/Opt/Simplify.hs:1538:57-77           2.2    1.5
MkFinalIface             GHC.Driver.Main        compiler/GHC/Driver/Main.hs:(1528,12)-(1529,68)        1.6    0.1
FloatOutwards            GHC.Core.Opt.Pipeline  compiler/GHC/Core/Opt/Pipeline.hs:499:34-82            1.6    1.4
CoreTidy                 GHC.Driver.Main        compiler/GHC/Driver/Main.hs:884:15-58                  1.6    1.7
CorePrep                 GHC.Driver.Main        compiler/GHC/Driver/Main.hs:(1566,24)-(1567,57)        1.5    1.8
occAnalBind.assoc        GHC.Core.Opt.OccurAnal compiler/GHC/Core/Opt/OccurAnal.hs:812:13-64           1.4    1.2
pprNativeCode            GHC.CmmToAsm           compiler/GHC/CmmToAsm.hs:427:37-64                     1.0    1.3
substTyWith              GHC.Core.TyCo.Subst    compiler/GHC/Core/TyCo/Subst.hs:(559,23)-(560,50)      1.0    1.0
DmdAnal                  GHC.Core.Opt.Pipeline  compiler/GHC/Core/Opt/Pipeline.hs:511:34-103           0.8    1.0