rts/RtsFlags: Refactor size parsing

This makes a number of improvements mentioned in #20201:

 * fail if the argument cannot be parsed as a number (`-Mturtles`)
 * fail if an unrecognized unit is given (e.g. `-M1x`)