Fix cabal-install of ghc package

!10547 (closed) has regressed support for building ghc library using cabal-install as part of the build now needs to be generated using a python script (which is not called from Setup.hs).

Either we need to revert this patch or fix the Setup.hs.

I am not sure if we should add a build dependency on python like this though. An alternative is to rewrite the generation script in Haskell so we can depend on it using build-tool-depends like the normal primops generation logic.

changed milestone to %9.8.1

added Phigh label

Sorry, I didn't realize the build logic for primops was duplicated between hadrian and compiler/Setup.hs. I thought the build dependency on python was OK because the RTS already has one in rts/gen_event_types.py.

I have other work based on top of the patch in question; I want to see its status resolved quickly. What are the options here? And how do I test that a proposed fix works?

At the moment we don't support building rts package with cabal so the python dependency isn't a problem there.. but something we might have to revisit in the future if we wish to support that.

I think the best way to resolve it is to rewrite the python script into a haskell executable so it can be built/run using the build-tool-depends logic in cabal. (like genprimopcode).

Does @bgamari have an opinion here?

added Tbug packaging labels

The patch also broke HLS on GHC for me, as I'm getting errors of the form

python3: createProcess: does not exist

That was a separate issue that was surfaced by the patch; I fixed it in !10863 (closed) which has since landed.

Should we revert !10547 (closed) for now? Or at least in the ghc-9.8 branch.

mentioned in merge request !10097 (closed)

I can't work on this properly because I still don't have an answer to this question:

And how do I test that a proposed fix works?

And this is still blocking me.

You can see the definition of the job in the .gitlab-ci.yml file. Which essentially uses the build-cabal hadrian target.

Forgive me if I'm being a bit dense here, but:

• I had a brief look at .gitlab-ci.yml and am not sure which job you're describing. Can you be more specific?

• I tried running hadrian/build build-cabal at 9c8fdda3 and get seemingly-unrelated errors after a lot of building, which look like this:

  src/Utils/Containers/Internal/TypeError.hs:11:1: error: [GHC-47808]
      Failed to load dynamic interface file for GHC.TypeLits:
        Exception when reading interface file  /path/to/tree/_build/stage1/lib/../lib/x86_64-linux-ghc-9.9.20230719/base-4.18.0.0-inplace/GHC/TypeLits.dyn_hi
          /path/to/tree/_build/stage1/lib/../lib/x86_64-linux-ghc-9.9.20230719/base-4.18.0.0-inplace/GHC/TypeLits.dyn_hi: withBinaryFile: does not exist (No such file or directory)
     |
  11 | import GHC.TypeLits
     | ^^^^^^^^^^^^^^^^^^^

  This looks a bug in hadrian. (I've selected devel2 as my default flavour.) If the *.dyn_hi files will be needed later, either they should get built or hadrian should complain immediately without building intermediate junk for 90 minutes first.

• I tried running hadrian/build-cabal. This just succeeds without indicating any problem.

The job in question is test-cabal-reinstall-x86_64-linux-deb10 which defines BUILD_FLAVOUR=validate. It seems that the build-cabal target doesn't work with the devel2 flavour for some reason (which is a bug but on a not very well traveled code path).

./hadrian/build build-cabal --flavour=validate

You can also apply the test-reinstall label to a MR for it to test this job.

mentioned in commit 6c1f4176

mentioned in merge request !10945 (closed)

mentioned in commit 0ed8a76d

mentioned in issue #23708

With !10945 (closed), I still can't get the build-cabal target working. Using cabal-install 3.8 to avoid #23281 (closed) and #23117 (closed) and using the validate flavour to avoid #23708, the build still fails with:

Building library for ghc-9.9..

GHC/Data/Word64Set.hs:7:2: error:
     fatal error: containers.h: No such file or directory
        7 |
          |  ^
  |
7 |
  |  ^
compilation terminated.

GHC/Data/Word64Set/Internal.hs:16:2: error:
     fatal error: containers.h: No such file or directory
       16 |
          |  ^
   |
16 |
   |  ^
compilation terminated.

GHC/Data/Word64Map.hs:12:2: error:
     fatal error: containers.h: No such file or directory
       12 |
          |  ^
   |
12 |
   |  ^
compilation terminated.

GHC/Data/Word64Map/Internal.hs:18:2: error:
     fatal error: containers.h: No such file or directory
       18 |
          |  ^
   |
18 |
   |  ^
compilation terminated.

GHC/Data/Word64Map/Lazy.hs:7:2: error:
     fatal error: containers.h: No such file or directory
        7 |
          |  ^
  |
7 |
  |  ^
compilation terminated.

GHC/Data/Word64Map/Strict.hs:8:2: error:
     fatal error: containers.h: No such file or directory
        8 |
          |  ^
  |
8 |
  |  ^
compilation terminated.

GHC/Data/Word64Map/Strict/Internal.hs:8:2: error:
     fatal error: containers.h: No such file or directory
        8 |
          |  ^
  |
8 |
  |  ^
compilation terminated.

GHC/Utils/Containers/Internal/BitUtil.hs:10:2: error:
     fatal error: containers.h: No such file or directory
       10 |
          |  ^
   |
10 |
   |  ^
compilation terminated.

GHC/Utils/Containers/Internal/StrictPair.hs:7:2: error:
     fatal error: containers.h: No such file or directory
        7 |
          |  ^
  |
7 |
  |  ^
compilation terminated.

GHC/Data/Word64Set.hs:7:2: error:
     fatal error: containers.h: No such file or directory
        7 |
          |  ^
  |
7 |
  |  ^
compilation terminated.

GHC/Data/Word64Map/Strict.hs:8:2: error:
     fatal error: containers.h: No such file or directory
        8 |
          |  ^
  |
8 |
  |  ^
compilation terminated.

GHC/Data/Word64Map.hs:12:2: error:
     fatal error: containers.h: No such file or directory
       12 |
          |  ^
   |
12 |
   |  ^
compilation terminated.

GHC/Data/Word64Map.hs:1:1: error:
    `gcc' failed in phase `Haskell C pre-processor'. (Exit code: 1)
  |
1 | {-# LANGUAGE CPP #-}
  | ^

GHC/Data/Word64Map/Internal.hs:1:1: error:
    `gcc' failed in phase `Haskell C pre-processor'. (Exit code: 1)
  |
1 | {-# LANGUAGE CPP #-}
  | ^

GHC/Data/Word64Map/Lazy.hs:1:1: error:
    `gcc' failed in phase `Haskell C pre-processor'. (Exit code: 1)
  |
1 | {-# LANGUAGE CPP #-}
  | ^

GHC/Data/Word64Map/Strict.hs:1:1: error:
    `gcc' failed in phase `Haskell C pre-processor'. (Exit code: 1)
  |
1 | {-# LANGUAGE CPP #-}
  | ^

GHC/Data/Word64Map/Strict/Internal.hs:1:1: error:
    `gcc' failed in phase `Haskell C pre-processor'. (Exit code: 1)
  |
1 | {-# LANGUAGE CPP #-}
  | ^

GHC/Data/Word64Set.hs:1:1: error:
    `gcc' failed in phase `Haskell C pre-processor'. (Exit code: 1)
  |
1 | {-# LANGUAGE CPP #-}
  | ^

GHC/Data/Word64Set/Internal.hs:1:1: error:
    `gcc' failed in phase `Haskell C pre-processor'. (Exit code: 1)
  |
1 | {-# LANGUAGE CPP #-}
  | ^

GHC/Utils/Containers/Internal/BitUtil.hs:1:1: error:
    `gcc' failed in phase `Haskell C pre-processor'. (Exit code: 1)
  |
1 | {-# LANGUAGE CPP #-}
  | ^

GHC/Utils/Containers/Internal/StrictPair.hs:1:1: error:
    `gcc' failed in phase `Haskell C pre-processor'. (Exit code: 1)
  |
1 | {-# LANGUAGE CPP #-}
  | ^
Command failed

That is, the Word64{Set,Map} stuff added in !10568 (closed) (cc @Noughtmare) uses a containers.h include from... somewhere... and it isn't being found. My guess is that it isn't being picked up by the sdist generation for the ghc package; I haven't looked any further.

But !10568 (closed) is not present in the ghc-9.8 branch and I think !10945 (closed) is enough to unblock !10097 (closed). Once !10945 (closed) lands we can lower the priority of this ticket.

mentioned in commit c709cb45

mentioned in commit b6b7ec71

closed with commit 32cae784

mentioned in commit 32cae784

mentioned in commit ab002df7

mentioned in commit 8d95542a