I measured that this makes the comiler allocate a bit more, but it
might also make it faster and reduce residency.  The extra allocation
is probably just because we're not inlining enough somewhere, so I
think this change is a step in the right direction.