Added test and changed -Nmax to -maxN, -n was taken

Noticed strange -m behavoir and fixed -m from quietly
ignoring being passed invalid opts, e.g. "-msasd"

Reviewers: simonmar, hvr, austin, thomie, bgamari

Reviewed By: hvr, thomie, bgamari

Subscribers: bgamari, hvr, thomie, simonmar

Differential Revision: https://phabricator.haskell.org/D1677

GHC Trac Issues: #10728