Skip to content

Mismatch between documentation and behaviour of encodeFloat

In the process of elaborating the docs to treat #3898 (closed), I wrote

    -- | 'encodeFloat' performs the inverse of 'decodeFloat' in the
    -- sense that for finite @x@ with the exception of @-0.0@,
    -- @'uncurry' 'encodeFloat' ('decodeFloat' x) = x@.
    -- @'encodeFloat' m n@ is the closest representable floating-point
    -- number to @m*b^^n@ (or @±Infinity@ if overflow occurs).

Unfortunately, the last sentence is not quite true. If the end result is a subnormal number, encodeFloat performs two roundings, which may lead to a wrong result (can be 1 ulp off). Changing encodeFloat so that it always finds the closest representable number would cost a lot in performance, it would bring it close to fromRational. So the thing to do is change the docs, I just don't know how yet.

Trac metadata
Trac field Value
Version 7.3
Type Bug
TypeOfFailure OtherFailure
Priority normal
Resolution Unresolved
Component libraries/base
Test case
Differential revisions
BlockedBy
Related
Blocking
CC
Operating system
Architecture
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information