Skip to content
  • Zhou Fangyi's avatar
    Fix integer overflow when encoding doubles (Trac #15271) · 37d14601
    Zhou Fangyi authored
    Summary:
    Ticket #15271 reports a case where 1e1000000000 is incorrectly
    converted to 0.0. After some investigation, I discovered the number is
    converted to rational correctly, but converting the ratio into a double
    introduced an error.
    
    Tracking down to how the conversion is done, I found the rts float
    implementation uses `ldexp`, whose signature is
    `double ldexp (double x, int exp);`
    The callsite passes an `I_` to the second argument, which is
    platform-dependent. On machines where `I_` is 64 bits and `int` is 32 bits, we
    observe integer overflow behaviour.
    
    Here is a mapping from rational to exponent with observations
    1e646457008  -> 2147483645 (result = infinity, positive in int32)
    1e646457009  -> 2147483648 (result = 0.0, overflow to negative in int32)
    1e1000000000 -> 3321928042 (result = infinity, overflow to positive in int32)
    1e1555550000 -> 5167425196 (result = 0.0, overflow to negative in int32)
    
    We fix this issue by comparing STG_INT_MIN/MAX and INT_MIN/MAX and bound the
    value appropriately.
    
    Test Plan: New test cases
    
    Reviewers: bgamari, erikd, simonmar
    
    Reviewed By: bgamari
    
    Subscribers: rwbarton, carter
    
    GHC Trac Issues: #15271
    
    Differential Revision: https://phabricator.haskell.org/D5271
    
    (cherry picked from commit 311a63979cfa2c1e81be54b82205e681f6ec4f14)
    37d14601