Allow arguments in response file

Agreed; this should be possible. We have gradually been teaching GHC to call downstream tools with response files and as of 866525a1 we have response file parsing logic in base where it can be used by GHC. However, we have yet to teach GHC itself to accept arguments from response files. Perhaps you would be interested in picking this up?

CCing @ckoparkar who has done work in this area in the past.

added driver newcomer labels

added Windows label

It would indeed be nice to have this. @nmattia there's a almost complete patch which teaches the RTS to parse response files. But it uses parts of GCC as is, which is not good because GCC uses a GPL license. In that patch, everything that needs to be re-written is isolated in a separate file, rts/ResponseFile.c. It's also here, on my fork, but I haven't made any progress in a while. Feel free to commandeer it :)

Related tickets: #15072 and #15732

@ckoparkar I don't see why you would put this in the RTS. I was assuming it would be in Haskell invoked by GHC's argument parser.

@ndmitchell Ah, sorry, that comment is terse and I should've provided more context. The short answer is that +RTS...-RTS flags complicate things a bit.

Technically, the RTS handles some of GHC's argument parsing. Whenever we supply any command-line arguments, the RTS first inspects them[1] -- it separates out the options contained in +RTS...-RTS blocks and updates the runtime configuration with the supplied values. After that, it stores[2] all the non-RTS flags in a variable prog_argv[3]. And getArgs just makes an FFI call to fetch prog_argv[4]. Thats why getArgs does not include any of the RTS flags in the list it returns.

Right now, getArgsWithResponseFiles uses getArgs under the hood. It fetches all the arguments, and tries to expand things that start with '@'. But it does not have a hook into the RTS to say "look for +RTS...-RTS flags in this list, and update your configuration". Also, even if it did, it would mean that getArgsWithResponseFiles would be responsible for an important side-effect! So if we update the GHC binary to use getArgsWithResponseFiles, it'll do the right thing for +RTS...-RTS blocks that are passed over the command line, but any such flags contained in the response files will simply have no effect.

One workaround is to say that this is ok, and we do not support passing RTS flags in response files. Then, this boils down to updating getArgsWithResponseFiles to filter out RTS flags (or maybe we'd like to keep them around?), and using it in the GHC binary. This is a lot easier to do, and it'll be better than the current situation at least. But at the time I thought I could finish that patch in a reasonable time, and it'll avoid any further confusion wrt this RTS flags business. So I chose to do that instead, but never finished it :(

Actually, this discussion has motivated me to pick that up again. I'm going to work on it over the weekend.

Thanks @ckoparkar - that explains it. To be honest, I would have expected it not to deal with RTS flags, and if it did, that would surprise me. If you bake it into the RTS, then every GHC compiled program gains response files, and for some, that might be inappropriate.

@ndmitchell, note that the design suggested by @simonmar proposes to disable response file expansion be default, adding an RTS flag to enable it.

Are we going to keep GHC.ResponseFile if we are going to support this from RTS?

There are current some incompatibilities with gcc on @file, some are important if it's used in general:

empty ("") arguments are not handled
@files are not expanded recursively

some are debatable:

if file does not exist, or cannot be read, then the option will be treated literally, and not removed
non-ascii whitespace are not delimiters
unterminated backslash-escape is silently ignored and unmatched quote is silently closed (it's currently compatible, but may be undesired)

I was about to fix the first two. But if we are not going to keep GHC.ResponseFile, then it is probably not worth fixing them there.

There are current some incompatibilities with gcc on @file, some are important if it's used in general:

Good point! In this patch, I've definitely fixed (2), and added a test for it. I'll do the same for (1) as well.

some are debatable:

I have to admit, I often end up working on Linux and so have never had to reach out for response files too much. As such, I was not aware of a lot of these subtleties until I actually looked at the GCC source a few months ago. Any advice on what the right thing to do in these cases is would be really helpful :)

re:non-existing-file: the new version prints out a warning, and removes that argument. This is perhaps to aggressive, it should only print a warning?

re:non-ascii-whitespace: thanks for pointing this out. It didn't occur to me at all.

I was about to fix the first two. But if we are not going to keep GHC.ResponseFile, then it is probably not worth fixing them there.

@bgamari can field this question.

Technically it only needs to keep getArgsWithResponseFiles around. But because other things are part of the API (expandResponse & unescapeArgs in particular), maybe they have to go through the usual 3 release deprecation process ? It probably depends on how "internal" we consider this module to be, and if people are actually using it.

All of that being said, it's easily going to take a few weeks to get this into master, and %8.10.1. seems like a realistic target. Is it blocking some of you're work, and did you want to put the GHC.ResponseFile fixes into %8.8.1?

mentioned in issue #16510

@ckoparkar, with few exceptions pretty much anything in the GHC namespace is quite internal. ResponseFile in particular I think we can feel free to change freely. Fixing the GHC.ResponseFile issues for %8.10.1 seems reasonable to me. I have opened #16510 to track this.

added Tbug label

We recently hit this in our project. Was there any progress :)?

Wow, you must have quite a large project. Unfortunately I don't think there has been progress on this. That being said, it shouldn't be hard to move forward. Perhaps you are interested in picking this up?

I would call the project oversized I'm gonna split it up to solve the immediate problem. I don't feel i understand what's required enough to actually pick fixing the real issue

Wow, you must have quite a large project.

If you use something like Bazel, that puts files at long paths (containing hashes) and puts each package in its own package database. You don't have to be that big to trigger it.

mentioned in commit 7d645ab5

I'm now being bitten by the equivalent of this on OS X, so this isn't just a Windows issue. I use backpack to produce 1200 or so little libraries full of modules and link them all in, and it er.. rather vastly exceeds the line length that cabal is allowed to feed GHC, causing me to crash 20 or 30 minutes into my build process. I can't shorten module and library names sufficiently to pass through the eye of this needle.

changed milestone to %9.6.1

marked this issue as related to #20393

For what it's worth, I think the option of implementing response file support in the RTS has been closed to us with the merging of multiple home units (fd42ab5f), which uses @ syntax (as an argument to -unit) with slightly different semantics than the RTS would implement.

@bgamari Can you help me understand how the home unit work would block the ability to just pass said flags in a file, despite it having a different syntax? I'm really struggling to follow you here.

I should have been more explicit; the problem is not that we are closed off from implementing support for response files. Rather, we can not use the specific implementation strategy which @simonmar previously suggested where the RTS would be responsible for response file expansion (since GHC does not treat response files passed to -unit via simple expansion).

That being said, I don't think this is a problem; I agree with @ndmitchell that it would be slightly surprising to me if RTS flags could be passed via response file.

Is this any clearer?

mentioned in commit d6bb5205

mentioned in merge request !7908 (closed)

mentioned in commit 32f3837d

mentioned in commit ece9e2e6

mentioned in commit 40b3bb21

mentioned in commit 36e2dfe3

mentioned in commit f02465d3

mentioned in commit c3fe409b

mentioned in commit 6a2e23ef

mentioned in commit 65907bd2

mentioned in commit 65b6af74

mentioned in commit 0d86ac88

mentioned in commit 25e97024

mentioned in commit 5a1e1207

mentioned in commit 25fd4c4c

mentioned in commit 14dacfdb

mentioned in commit 378f2cd5

mentioned in commit 92ee3d36

mentioned in commit 48ff24b2

mentioned in commit de527da4

mentioned in commit f81fa763

mentioned in commit dfa51516

mentioned in commit 43ac0642

mentioned in commit 84977e3e

mentioned in commit 3e4666b9

mentioned in commit 04eac256

mentioned in commit dd287cde

mentioned in commit 4b279a1e

mentioned in commit e893d8c0

Allow arguments in response file

Child items ...

Activity

Allow arguments in response file

Relates to

Activity