x86_64 code use several GiB of memory generates: internal error: ASSERTION FAILED: file sm/Storage.c, line 1126
I have some fairly simple code using !ByteStrings to parses fasta files (http://en.wikipedia.org/wiki/Fasta_format) and generate all the M-length suffixes.
When I set it to hold enough M-length suffixes is memory that several GiB of RAM are consumed, it either segment faults or loops infinitely (the pared down code I've attached infinitely loops with the given parameters, while my original code segment faulted). I hooked into the attached version while it was infinitely looping one with gdb and did a backtrace.
#0 0x0000000000460ef4 in free_list_push_forwards () #1 0x0000000000461357 in freeGroup () #2 0x000000000046222f in GarbageCollect () #3 0x000000000045d33f in scheduleDoGC () #4 0x000000000045dd36 in scheduleWaitThread () #5 0x000000000045aa01 in real_main () #6 0x000000000045aad3 in main ()
Disassembling the free_list_push_forwards routine revealed that it was infinitely stuck in the while loop because the tail of the passed block descriptor (list) was circular and the passed block descriptor had a larger block count than any of the block descriptors in its circular tail. I also noted that the block descriptors composing the tail where not non-decreasing by block count up to the circular point (I don't know if this is a problem though).
When I compile with -debug, I get the following (first two lines is program output):
Starting new sorted block at 139609412 (/tmp/PChunk3790.tmp) Starting new sorted block at 255642977 (/tmp/PChunk3791.tmp) Bug-debug: internal error: ASSERTION FAILED: file sm/Storage.c, line 1126 (GHC version 6.8.2 for x86_64_unknown_linux) Please report this as a GHC bug: http://www.haskell.org/ghc/reportabug
I've attached the Bug.hs code (which is the pared down version of my code). A gzipped version of the input file (~3GiB) can be found at ftp://ftp.ncbi.nlm.nih.gov/blast/db/FASTA/nr.gz. To generate the above output, I compiled with "ghc -debug -O2 --make Bug" and with "./Bug-debug nr 8 67108864" (i.e., generate blocks of 67108864 8-character suffixes from the fasta file nr).
From what I've seen so far, the problem seem linked to the complexity of the code and the amount of memory used. I've never had any problems as long as I stay under 4GiB of RAM, and the more I pared down the code, the more memory I had to make it use before it would start to demonstrate the problem (with the parameters above the pared down code uses just over 16GiB). I could possibly pare the code down further.
I do not know at all what I'm talking about, but from the little I've seen so far, I wonder if it is related to the 64bit memory space (e.g., having enough blocks that it can actually loops a 32bit counter tracking them, etc.).
NOTE: I compiled ghc from source from http://www.haskell.org/ghc/dist/stable/dist/ghc-6.8.2-src.tar.bz2.