Skip to content
  • tibbe's avatar
    shouldInlinePrimOp: Fix Int overflow · 6f862dfa
    tibbe authored
    There were two overflow issues in shouldInlinePrimOp. The first one is
    due to a negative CmmInt literal being created if the array size was
    given as larger than 2^63-1 (on a 64-bit platform.) This meant that
    large array sizes could compare as being smaller than
    maxInlineAllocSize.
    
    The second issue is that we casted the Integer to an Int in the
    comparison, which again meant that large array sizes could compare as
    being smaller than maxInlineAllocSize.
    
    The attempt to allocate a large array inline then caused a segfault.
    
    Fixes #9416.
    6f862dfa