ghc-pkg on 8.8.1 with cabal 3.0 does not list user installed packages

added needs triage label

I don't really think this is a bug in ghc-pkg. It has no concept of these environment files, if you passed the options which were present in the environment file then it would display the package as expected.

Perhaps a solution is for cabal to add a cabal ghc-pkg target which passes the right flags to ghc-pkg to see these "installed" packages.

closed

ok, but it is a change in behaviour from the view point of an end user.

Thanks for responding; from your response, the env file, and the doc I was able to figure out that I now need to type: "ghc-pkg list -f /Users/gcolpitts/.cabal/store/ghc-8.8.1/package.db" Not sure if this change should be in the release notes.

This feels to me like a step backwards from just typing "ghc-pkg list" as the user doesn't have to know anything about package dbs nor their location. Not sure how to go about improving the current state but thanks for your help.

I'll update the cabal issue. Thanks again.

@mpickering Well it's not exactly a bug no, but the question still remains should we maybe teach ghc-pkg about these env files since GHC itself certainly knows about them, so why not the ghc package manager?

reopened

added Tfeature request ghc-pkg labels

removed needs triage label

I had a quick chat with @hvr about this a couple of days ago and he also seemed to think teaching ghc-pkg about env files would make sense.

We do still have to figure out how to go about that without breaking anything though.

Thanks Daniel! I agree it would be good to get this back to where it was. From an end user viewpoint it is a regression.

I don't agree this is a regression. ghc-pkg continues to work as before, it is cabal which has changed its default behavior.

Yes, cabal changed its default behaviour and this is the source of the issue. Given that, I hope you agree that this is a high priority feature request. I am very unhappy with cabal 3.0 changes in behaviour; they make using the ghc ecosystem harder to use and are a step backward in my opinion. This is what I meant by regression. I should have clarified that it is a regression in cabal which IMHO necessitates a change in ghc-pkg. I'm just trying to make the ghc ecosystem better and appreciate everyone's help in doing so and getting the issues to the right people.

This started out as a cabal issue: https://github.com/haskell/cabal/issues/6262. I will be filing more cabal bugs. They may end up as Cabal 3.0 feature requests.

It is not simply cabal which changed the behavior. The real change is that a raw ghc or ghci will make use of environment files, but ghc-pkg will not, so regardless of how those files are written (via Cabal or otherwise), this is an inconsistency in the ghc toolchain.

Any chance that this will be fixed for 8.10.1? If not, can we give it a milestone?

Can I add a milestone of 10.8.2? If so, how do I do that? If not, can somebody else add it? Is this worth noting in the 8.10.1 release notes?

Thanks George

mentioned in issue #17764 (closed)

I've been trying to get Cabal to work myself. It seems that all that's required for ghc-pkg and the default package environment to play nicely is for there to be a wildcard in the default environment that selects the latest version of all packages that are visible in the package db. Then ghc-pkg can function as before, and code requiring a specific environment can name its requirements.

It would also be nice if Cabal and GHC could both work on the user db. I've told GHC where to find its db for now by saying: ln -s ~/.cabal/store/ghc-8.6.5/package.db ~/.ghc/x86_64-darwin-8.6.5/package.conf.d

We ran into this today. ghc was reporting it is seeing multiple modules with the same name, but using ghc-pkg we could not see a single user package installed. It took a while to figure out where the packages were coming from. If GHC knows about the packages then why not ghc-pkg? Or there should be some other way to list what packages ghc knows about, a ghc option? or yet another other ghc tool to operate on environment files?

changed milestone to %9.0.1

changed milestone to %8.10.2

Ok, let me add a suggestion from irc chat here: The base problem is just as described above. It breaks user-expectations for ghc-pkg to not show packages that a raw invocation of ghc or ghci sees. The simplest solution is just for ghc-pkg to detect when there's an env file around, and then, on ghc-pkg list (at least) simply warn that there's such a file, and that file will determine what ghc sees.

If we can't get this fixed for ghc 8.10.2/8.12.1 can we document how to find the packages that have been installed?

Should I submit a doc bug for "If we can't get this fixed for ghc 8.10.2/8.12.1 can we document how to find the packages that have been installed?". Perhaps this is documented in the release notes for 9.0 but I can't find them. From my entry above it seems that is currently:

ghc-pkg list -f $HOME/.cabal/store/ghc-9.0.0.20200925/package.db
ghc-pkg list

cabal v1-install is no longer an option as some packages, e.g. pandoc, install with cabal install but not with cabal v1-install

I'm disappointed that it doesn't seem that this will be fixed. I realize that cabal is separate from ghc but from an end users perspective it's all one ecosystem. cabal 3 seems to me very backward incompatible with cabal 2 and ghc.

I agree that ghc and ghc-pkg really shouldn't have different defaults in this matter.

But for the time being documenting how to work around this would be a good start.

Thanks Andreas, filed #18933 to document issue and workaround in release notes

changed milestone to %9.2.1

marked this issue as related to #18915 (closed)

marked this issue as related to #17764 (closed)

mentioned in issue #18933

I ran into this while trying to compile a rather unorthodox project: https://github.com/B-Lang-org/bsc

It uses all sorts of Makefiles and suggests to use cabal v1-install. I don't really want to rewrite their entire build system, nor do I want to use legacy v1-install. They invoke ghc-pkg directly to check whether the packages are installed: https://github.com/B-Lang-org/bsc/blob/ad02e9317ae9d808f6011567bae5c14cbd6753ec/src/Makefile#L30

So I went ahead, installed cabal-env and did GHC_ENVIRONMENT=bsc make install. It didn't work.

So I'd argue this definitely breaks toolchain expectations and is a bug.

changed milestone to %9.4.1

Just ran into this as well, as I wanted to use ghc-pkg to find out the data-dir of something that I installed. Also, I find ghc-pkg list very useful to understand what ghc sees.

assigned to @mpickering

Can we get this fixed in 9.2.2? If not can we get the documentation bug #18933 done for the 9.2.2 release notes?

@trac-george.colpitts Do you have an opinion about how this is supposed to behave? An environment file contains not only links to where the package database is but also a list of packages it exposes from one of the package databases. ghc-pkg currently doesn't understand anything to do with -package flags.

Is it still useful if we just get the visible package databases from the environment file and display everything in the package database (ignoring the package visibility flags?)

Hi

IMHO it is supposed to behave the way it used to, i.e. show all the packages I have. Now (in 9.2.2) I have to type two commands to see this, previously "ghc-pkg list" would show all packages. Here are the two commands I now type to see all the packages I have what I want is to see all this by just typing the first command.

bash-3.2$ ghc-pkg list
/usr/local/lib/ghc-9.2.2/lib/package.conf.d
    Cabal-3.6.3.0
    array-0.5.4.0
    base-4.16.1.0
    binary-0.8.9.0
    bytestring-0.11.3.0
    containers-0.6.5.1
    deepseq-1.4.6.1
    directory-1.3.6.2
    exceptions-0.10.4
    filepath-1.4.2.2
    (ghc-9.2.2)
    ghc-bignum-1.2
    ghc-boot-9.2.2
    ghc-boot-th-9.2.2
    ghc-compact-0.1.0.0
    ghc-heap-9.2.2
    ghc-prim-0.8.0
    ghci-9.2.2
    haskeline-0.8.2
    hpc-0.6.1.0
    integer-gmp-1.1
    libiserv-9.2.2
    mtl-2.2.2
    parsec-3.1.15.0
    pretty-1.1.3.6
    process-1.6.13.2
    rts-1.0.2
    stm-2.5.0.2
    template-haskell-2.18.0.0
    terminfo-0.4.1.5
    text-1.2.5.0
    time-1.11.1.1
    transformers-0.5.6.2
    unix-2.7.2.2
    xhtml-3000.2.2.1

bash-3.2$ ghc-pkg list -f $HOME/.cabal/store/ghc-9.2.2/package.db
/Users/gcolpitts/.cabal/store/ghc-9.2.2/package.db
    OneTuple-0.3.1
    QuickCheck-2.14.2
    StateVar-1.2.2
    aeson-2.0.3.0
    assoc-1.0.2
    attoparsec-0.14.4
    (z-attoparsec-z-attoparsec-internal-0.14.4)
    base-compat-0.12.1
    base-compat-batteries-0.12.1
    base-orphans-0.8.6
    bifunctors-5.5.11
    comonad-5.0.8
    contravariant-1.5.5
    data-default-class-0.1.2.0
    data-fix-0.3.2
    distributive-0.6.2.1
    dlist-1.0
    erf-2.0.0.0
    (ghc-lib-parser-9.2.2.20220307)
    ghc-lib-parser-ex-9.2.0.3
    hashable-1.4.0.2
    hmatrix-0.20.2
    indexed-traversable-0.1.2
    indexed-traversable-instances-0.1.1
    integer-logarithms-1.0.3.1
    math-functions-0.3.4.2
    mwc-random-0.15.0.2
    primitive-0.7.3.0
    probability-0.2.7
    random-1.2.1
    scientific-0.3.7.0
    semialign-1.2.0.1
    semigroupoids-5.3.7
    semigroups-0.20
    split-0.2.3.4
    splitmix-0.1.0.4
    storable-complex-0.2.3.0
    strict-0.4.0.1
    syb-0.7.2.1
    tagged-0.8.6.1
    text-short-0.1.5
    th-abstraction-0.4.3.0
    these-1.1.1.1
    time-compat-1.9.6.1
    transformers-compat-0.7.1
    uniplate-1.6.13
    unordered-containers-0.2.17.0
    utility-ht-0.0.16
    uuid-types-1.0.5
    vector-0.12.3.1
    witherable-0.4.2

bash-3.2$

@trac-george.colpitts ok, but the output of the second command ignores the visibility settings given by the environment files so you might not be actually able to use all those packages.

MR is.. !7754 but I am unsure about the safety of the changes and what to do about package flags in the environment file.

Sorry I don't understand all the issues. Is it possible to fix the reported issue, i.e if I install a package I should be able to see the package when I do "ghc-pkg list"? Thanks

I think that the expected behavior would be that the ghc-pkg list should, when making use of an environment file, first, loudly proclaim it is doing such a thing, and second, only list as exposed those packages specified in the environment file.

I.e. its default behavior should be to present the packages to the user the same way they will experience them if they open ghci.

@gershomb, @mpickering in the past I haven't needed to be aware of env files and as far as I know I am not using them. In this case will the fix give me what I want: when I install a package I will see the package when I do "ghc-pkg list"?

@trac-george.colpitts This confusion is exactly why this ticket has not been fixed until now. ghc-pkg is a low level way to interact with package databases, it has commands which allow you to show and modify package databases. As Gershom points out, he would think the expected thing is to see the view that the environment files provides of the the package db, but then what do all the commands which modify a package db do? Modify the database still? Modify the environment file somehow? It's really unclear this feature should exist.

Is the feature you actually want is to see which packages the GHC you have can see by default? It turns out if you have an environment file you can see this by passing -v2 to GHC.. then I see

Loaded package environment from /home/matt/.ghc/x86_64-linux-8.10.2/environments/default
Using binary package database: /nix/store/lry5v6jkdgx6sqgn8yhqk9sdb6g2h2ck-ghc-8.10.2/lib/ghc-8.10.2/package.conf.d/package.cache
Using binary package database: /home/matt/.cabal/store/ghc-8.10.2/package.db/package.cache
package flags [-package-id ghc-8.10.2{unit ghc-8.10.2 True ([])},
               -package-id bytestring-0.10.10.0{unit bytestring-0.10.10.0 True ([])},
               -package-id unix-2.7.2.2{unit unix-2.7.2.2 True ([])},
               -package-id base-4.14.1.0{unit base-4.14.1.0 True ([])},
               -package-id time-1.11.1.1-348d322acf6f5ca6a165031c2ef9a79f732ebadf4c857693fdb6600ca83cdf9c{unit time-1.11.1.1-348d322acf6f5ca6a165031c2ef9a79f732ebadf4c857693fdb6600ca83cdf9c True ([])},
               -package-id hpc-0.6.1.0{unit hpc-0.6.1.0 True ([])},
               -package-id filepath-1.4.2.1{unit filepath-1.4.2.1 True ([])},
               -package-id process-1.6.9.0{unit process-1.6.9.0 True ([])},
               -package-id array-0.5.4.0{unit array-0.5.4.0 True ([])},
               -package-id integer-gmp-1.0.3.0{unit integer-gmp-1.0.3.0 True ([])},
               -package-id containers-0.6.2.1{unit containers-0.6.2.1 True ([])},
               -package-id ghc-boot-8.10.2{unit ghc-boot-8.10.2 True ([])},
               -package-id binary-0.8.8.0{unit binary-0.8.8.0 True ([])},
               -package-id ghc-prim-0.6.1{unit ghc-prim-0.6.1 True ([])},
               -package-id ghci-8.10.2{unit ghci-8.10.2 True ([])},
               -package-id rts{unit rts True ([])},
               -package-id terminfo-0.4.1.4{unit terminfo-0.4.1.4 True ([])},
               -package-id transformers-0.5.6.2{unit transformers-0.5.6.2 True ([])},
               -package-id deepseq-1.4.4.0{unit deepseq-1.4.4.0 True ([])},
               -package-id ghc-boot-th-8.10.2{unit ghc-boot-th-8.10.2 True ([])},
               -package-id pretty-1.1.3.6{unit pretty-1.1.3.6 True ([])},
               -package-id template-haskell-2.16.0.0{unit template-haskell-2.16.0.0 True ([])},
               -package-id directory-1.3.6.0{unit directory-1.3.6.0 True ([])},
               -package-id text-1.2.3.2{unit text-1.2.3.2 True ([])},
               -package-id tree-diff-0.2.1.1-41cc5ce32ee37f9fe7c76a14d0edef5231bbb92a91fc525b3224bea0ada997d9{unit tree-diff-0.2.1.1-41cc5ce32ee37f9fe7c76a14d0edef5231bbb92a91fc525b3224bea0ada997d9 True ([])}]

What I want is for ghc-pkg list to show all packages I have installed.

I have an environment file which ghci seems to be aware of but ghc-pkg is not aware of it. Do I need to set an env variable? i.e. ghc-pkg -v2 list output starts with

ghc-pkg -v2 list
GHC package manager version 9.2.2
Timestamp 2022-03-10 22:44:08.034359036 UTC for /usr/local/lib/ghc-9.2.2/lib/package.conf.d/package.cache
using cache: /usr/local/lib/ghc-9.2.2/lib/package.conf.d/package.cache
db stack: ["/usr/local/lib/ghc-9.2.2/lib/package.conf.d"]
flag db stack: ["/usr/local/lib/ghc-9.2.2/lib/package.conf.d"]
/usr/local/lib/ghc-9.2.2/lib/package.conf.d
    Cabal-3.6.3.0 (Cabal-3.6.3.0)
...

No mention of an env file similar to the one in your output:

Loaded package environment from /home/matt/.ghc/x86_64-linux-8.10.2/environments/default

However ghci is aware of my default env file

 ghci
ghci
Loaded package environment from /Users/gcolpitts/.ghc/x86_64-darwin-9.2.2/environments/default
GHCi, version 9.2.2: https://www.haskell.org/ghc/  :? for help
...

This takes me back to what @gershomb wrote earlier on this ticket: #17341 (comment 265957)

Maybe @hvr and @nomeata can comment again as they are a lot more knowledgeable than me and have commented above.

I definitely think that any command besides list should not take env files into account. Otherwise tools that interact with ghc-pkg (including cabal itself) might start to have very unexpected behavior.

Its just the case that when there's an env file, list should make the information it provides available to a user, nicely, and in a way that explains what's going on.

There is also the undocumented --show-packges GHC command which seems a bit useless as it doesn't understand visibility given by environment files.

The commit which introduced --show-packages (a4cb9a61) states:

commit a4cb9a6173f0af76a32b812c022bbdd76b2abfac
Author: Duncan Coutts <duncan@well-typed.com>
Date:   Sun Aug 24 03:38:39 2014 +0100

    Add a ghc -show-packages mode to display ghc's view of the package env
    
    You can use ghc -show-packages, in addition to any -package -package-conf
    -hide-package, etc flags and see just what ghc's package info looks like.
    The format is much like ghc-pkg show.
    
    Like the existing verbose tracing, but a specific mode.
    Re-introduce pretty printed package info (Cabal handled this previously).

So it seems like a bug that it doesn't respect visibility.

This is not going to happen for 9.4.1.. I will remilstone for 9.6.1 but the design needs to be worked out first because as the previous comments show it's really not clear how ghc-pkg should interact with environment files.

changed milestone to %9.6.1

@mpickering I don't understand, on reviewing the previous comments, what the outstanding question is.

If you could say what the question is, then this can be cleared up. If it isn't clearly articulated, then it will never be answered, and this issue will be postponed indefinitely

@trac-gershomb The issue is that

@trac-gershomb indicates that he wants list to take into account environment files and specifically their visibility filter.
ghc-pkg solely understands about package databases (nothing to do with this visibility filter) and extending it to understand the filter seems undesirable.

Therefore I suggested another option which was to add or modify a command which ghc has in order to get ghc to provide an output like ghc-pkg list but can just directly use the logic ghc has for discovering package databases and understanding environment files. That seems like it will solve the root issue of being able to discover which packages GHC knows about?

Ah that's the source of the confusion. The root issue is not simply "being able to discover which packages GHC knows about" but rather that people have been taught for a very long time that the way to do that is ghc-pkg list. So they type ghc-pkg list and get output which is extremely misleading compared to what much documentation still says, and compared to their expectations.

I would be perfectly happy with something as simple as ghc-pkg list figuring out which environment file would be discovered by ghc, and just literally running a "cat" of its contents (appropriately denoted as such) following its current output.

But the issue is really that ghc-pkg list appears to tell users what packages ghc knows about, but that this does not correspond to what will happen when they run ghci and find out what modules are actually in scope. It has been extremely confusing to many users for some years now that this happens, and I'd really like there to be some fix for this. Any command that is not ghc-pkg list no matter how clever or powerful it is will not solve this. But a very crude change that is not at all clever to ghc-pkg list will solve this.

Any command that is not ghc-pkg list no matter how clever or powerful it is will not solve this. But a very crude change that is not at all clever to ghc-pkg list will solve this.

I really don't see why a ghc --list-visible-packages flag wouldn't solve this going forward.

It seems the problem is that there is no good way to tell which packages will be in scope. A new ghc --list-visible-packages flag would solve that.

Of course we could do a "crude change" to ghc-pkg list instead. But the result would just be that for the next few years users will still be confused because it will work on new versions but still be broken on old versions.

Users won't be able to rely on ghc-pkg list until everyone is on a fixed version either way. So we might as well use ghc which already has all the logic and add a new properly specified flag there.

I really don't see why a ghc --list-visible-packages flag wouldn't solve this going forward.

Because the problem is that the existing command does not do what people expect, and there is no signpost in the output of that command that it does not do what people expect.

I agree that there will be versions that exist that continue to not do what people expect. However, by that argument, no ux improvement could be made ever, because there will be cases where older versions do not have it.

I agree that users won't be able to rely on it -- however, there are users who have been taught to rely on it and get confused. For those users, improving the immediate situation seems like a straightforward win.

What I'm suggesting seems terribly easy to implement, and has no particular downside. I don't understand the squeamishness.

I agree that --list-visible-packages is a poor solution for the issue at hand, which is specifically that ghc-pkg is misleading. Adding yet another means for the user to query the package database will only further the confusion.

On the other hand, sadly @mpickering is right that there are rather tricky design decisions and architectural issues that make it difficult to resolve this issue in a satisfactory manner. ghc-pkg currently has access to a very small fraction of GHC's package machinery and fixing this would require moving a significant amount of code from ghc into ghc-boot (which ghc-pkg depends upon). Thusfar ghc-pkg has served solely as an interface to GHC's package database. Package visibility (as controlled by -package) is orthogonal to which packages are registered in the package database; rather it has more to do with the module name resolution logic used during compilation.

My tendency is to look for the simplest possible solution in the design space that addresses the underlying problem. Here I would be happy to have logic even that noticed if there was an env file around, and if there was, printed, along with the current output a notice that the packages displayed by list would not correspond to what a ghci session had in scope, due to this env file, and printed the location of the env file so a user could go and read it for themselves.

Why doesn't cabal exec ghc-pkg list perform this task? The cabal exec --help output reads like it should work, but it doesn't.

ghc-pkg on 8.8.1 with cabal 3.0 does not list user installed packages

Summary

Steps to reproduce

Expected behavior

Environment

Child items ...

Activity

ghc-pkg on 8.8.1 with cabal 3.0 does not list user installed packages

Summary

Steps to reproduce

Expected behavior

Environment

Relates to

Activity