Add a flag to optionally keep auto-generated rules around.

This adds a flag to allow keeping auto-generated rules around: see #18532 (closed) and #21917 (closed).

It's unclear if having such a flag is worth having yet! But this makes it easy to try out before we decide on this.