rts: ensure we are below maxHeapSize after returning megablocks

When the heap is heavily block fragmented the live byte size might be
low while the memory usage is high. We want to ensure that heap overflow
triggers in these cases.

We do so by checking that we can return enough megablocks to
under maxHeapSize at the end of GC.