Calling a complex Haskell function (obtained via FFI wrapper function) from MSVC 64-bit C code (passed in as FunPtr) can leave SSE2 registers in the XMM6-XMM15 range modified
According to the MSDN, in the Microsoft x64 architecture function calls must preserve the SSE2 registers in the range XMM6-XMM15. The Haskell FFI can produce a function pointer via dynamic wrapper that, when called from MSVC x64 C code, does not preserve these registers, causing further floating-point operations in the C code to fail.
I can reproduce this error in this project, which is a DOOM-style raycasting engine written in Haskell, that imports a C DLL with glue for rendering and window management. The Haskell executable generates a FunPtr to a frame update function using the dynamic import mechanism, and passes this to a long-lived C function that runs the update loop. Any time this update function is called from the C loop, subsequent floating point operations produce incorrect results (in this case, the next operations compute a view matrix for the OpenGL window).
The output on every frame showing the view matrix should be:
viewM: 0.003125 0.000000 0.000000 -1.000000 0.000000 0.004167 0.000000 -1.000000 0.000000 0.000000 1.000000 0.000000 0.000000 0.000000 0.000000 1.000000
Running the raycaster with the Release version of the DLL causes the value of this matrix to be corrupted. There is a patch provided (stub.patch in the root folder) that turns the Haskell update function into an empty stub. This causes the program to work. When stepping through the assembly code with this patch applied, I can see in the function prologue where the XMM registers are saved. Without the patch, these registers are not saved. Running the Debug version does not show this error; the register allocation must be different.
I have been attempting to create a much simpler test case to reveal this code-generation issue, however it has been difficult. Even seemingly trivial changes can cause the bug to not show up, it is clearly dependent on the register allocation used internally to produce the assembly code.
Instructions for building the project are in the readme. (You will need the Haskell Stack Tool, and Visual Studio 15).