Wrong tar flag on darwin

mentioned in merge request !119 (merged)

Please be more specific. What is the actual bug, does extraction fail? On which version?

I haven't bothered verifying my claim but considering most mac os users update fairly regularly I'd risk breaking it for older versions and waiting for a new issue report.

I think we should verify this (if possible).

I encountered this problem on El Capitan 10.11.6, using /opt/local/libexec/gnubin//tar from macports. Extraction failed. I got around it by putting a "J" into tar's flags.

By the way, I had to make another edit to compile 8.8.1 from source. The "--with-ghc=..." construct seems to be obsolete.

    edo ./configure --prefix="${inst_location}" GHC="${bootstrap_ghc}"

Love the name, @GussyFinkNottle : ) (Just so happens I'm watching the show.)

Do you happen to be the visitor in #haskell that prompted me to submit this contribution?

13:25 < guest_18> Installing ghcup from https://www.haskell.org/ghcup/ with macp
orts /opt/local/libexec/gnubin//tar, there seems to be a glitch.
13:26 < guest_18> (Sorry, newb at IRC). The problem is that the tar command uses
 -xzf flags, which complains that the archive is not in gzip format. I had to ed
it the flags to be just -xf.

Maybe not, because their solution differed from yours.

@maerwald, ok so trying to verify, bsdtar has had XZ support since 2.7.0. So now we need to figure out what version of OSX came with bsdtar 2.7.0 and whether that version supported automatic encoding detection.

It'd probably also be nice to figure out the same for gnutar, though that's less important since it doesn't come with Mac OS.

I think this commit suggests that automatic detection was present from before 2.8.0.

Ok, I think this release log shows that 2.8.3 was released in 2010 and (because it's on the apple website) included in OSX somewhere around that time? So that was 10.6 Snow Leopard.

Ok, so I was wrong. Snow Leopard came with 2.6.2.

OSX 10.7 Lion and onwards have bsdtar version 2.8.3

Also, the type detection is just how bsdtar has always worked. Specifying compression algorithm is only necessary/(allowed even) when creating archives.

Ok, so gnutar supports type detection except when you pipe to it. and has had xz support since 2009-03-05, version 1.22.

So it seems this is specific to El Capitan 10.11.6. I don't really understand all Mac differences. How would an extraction command look like that works for all Mac versions?

No, it affects any version of Mac OS as long as the user's running gnutar. Maybe bsdtar isn't strict about the flags and ignores the extra z?

MR !119 (merged) uses the most compatible formulation possible.

I think you (@toonn) could well be right. I've some executables called bsdtar in /usr/bin (and /opt/local/bin), and with "-tzf" they both seem to cope fine with various xz files I've tried it on.

-rwxr-xr-x 1 root wheel 70752 Jun 29 2018 /usr/bin/bsdtar -rwxr-xr-x+ 1 root admin 69148 Aug 31 07:47 /opt/local/bin/bsdtar

The second might be from macports (but I can't find it there). Heaven knows where the first is from: it's symlinked from /usr/bin/tar. (But that could have been just me hacking my way round some urgent problem.)

I'm inclined to revert !109 (merged) then and go back to the original behavior of requiring 'xz'. @trac-gershomb

MR !119 (merged) uses the most compatible formulation possible.

So did you test it on multiple mac versions?

Please do not revert. If you revert, then any typical mac user (most of whom do not know how to/care to install things through macports or brew) will be absolutely stumped by using ghcup.

We should try to make the current behavior more compatible, sure. However, if it is only incompatible if people are using a tar not bundled with os x (i.e. from a third party manager), then at least this will only bite people who are competent to work around the problem themselves.

So is ( xz -cd "${filename}" | tar xf - ; ) || die "unpacking failed!" compatible with your system?

I tested !119 (merged) on the tar bundled with os x and its fine, so I think the best path forward is to merge this PR. Since the tar version with os x will be compatible, by the above discussion, going back to at least 10.7, this is absolutely fine, as that's well prior to the compat window of the ghcs that ghcup is able to install :-)

Yes, it is. OS X 10.13.6, using only the standard os x command line tools.

@maerwald, really, Mac OS is not the weird case here. The weird thing is all the branching for the seperate decompression and piping. If Mac OS 10.7 had a tar that can handle the detection and decompression for all the formats just fine then the linux hosts that are being targeted with the piping must be ancient by any practical standard.

If it were up to me I'd rewrite everything to tar xf <archive> || case xz) xz | tar xf;; gz) gunzip | tar xf.... (Semantically, I don't mean it has to be this exact syntax, just that the weird systems are the ones with a tar that doesn't support gzip decompression...)

I'm a bit worried that I've raised an unnecessary distraction. My system is an absolute pigsty. For example. here is (unedited) my current echo $PATH (if I succeed in rendering it):

/Users/pgh/.cabal/bin:/Users/pgh/.ghcup/bin:/Users/pgh/.android-sdk-macosx/platform-tools/:/opt/local/libexec/gnubin/:/Users/pgh/bin:/Users/pgh/.cabal/bin:/Users/pgh/Library/Haskell/bin:/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/LilyPond.app/Contents/Resources/bin:/Users/pgh/.android-sdk-macosx/platform-tools/

I don't understand markdown enough to competently show you things that could well be relevant. My judgement is that there is a 52% argument you should change the ghcup decompression stuff to be a bit more idiot-proof.

Why I'd high hopes for ghcup (still do) is that (1) I used to (just about) maintain my own rats-nest of symlinks for various versions of ghc, cabal etc compiled from source and (2) my version of osx (10.11.6 El Capitan) is 3 or 4 generations old, for various good (enough for me) reasons. The binaries around seem to built on mojave.

For me, 8.8.1 (having many attractive improvements), is at the moment out of the question (because of _utimensat, being a mojave thing), though by some miracle I once or twice got to build (and luckily preserve) working versions out of the main bits of software that matter to me, namely ghci, agda and lhs2tex.

Over and, more power to your elbow, out.

@GussyFinkNottle, you sound like a good use case for Nix : ) Afaik they build GHC from source (the binary cache might interfere but you could just not allow fetching from the cache for the particular package).

Another remark in favor of separating the piped versions of the calls is that systems with an old enough version of tar to not have compression detection won't have an xz binary either. So the only extra case the piped calls cover is tar.gz on a system with an ancient tar, which, do we know it'll even run GHC?

I'm starting to get confused by this discussion. Please keep the discussion specific to unpacking on darwin.

I don't have such a system to test and must rely on other people to tell me what works. Linux works fine and doesn't need any adjustments.

So please confirm that !119 (merged) is the correct fix for all relevant OSX versions, whether it's bsdtar or gnutar.

I will confirm again that !119 (merged) remains correct for the bundled tar on all relevant OSX versions. If it is also the correct fix for tars installed by e.g. macports (which is what the author of the patch claimed to begin with) then that suffices to confirm that it is the correct patch in general.

closed via commit 9f083c53

closed via merge request !119 (merged)

@maerwald, just to clarify, I was merely saying that the problem arose in the first place because the way tar is being used for "compatibility with old versions of tar" is quite weird.

As you can read in this thread, gnutar has supported detecting compression since ~1996 or version ~1.15. And bsdtar has probably supported it for longer.

Systems with such an old version of tar wouldn't have an xz binary anyways. So the only invocation that increases compatibility with older versions is gunzip <archive.tar.gz> | gnutar xzf. Imo, that should be the special case. So the code could be reduced to if ancient-tar-and-gz; then gunzip ...; else tar xf <archive.any.extension>;. And note that this special case would only help on systems that are over ~2 decades old. Which probably don't run any of the current supported GHC versions.

Feel free to ask for a better explanation if that's still not clear. Or I could write up a PR of what I think would be the proper way to do this. Code should be the clearest medium to convey explanation about code : )

@toonn ghcup tries to be POSIX compliant, although that's not always possible.

https://pubs.opengroup.org/onlinepubs/007908799/xcu/tar.html

I don't see POSIX tar manpage mentioning automatic compression detection. So, when in doubt, I prefer to stick to what POSIX specifies (or doesn't specify).

Wrong tar flag on darwin

Child items 0

Activity