Possible memory leak introduced in GHC 9.2?
Summary
We have a relatively simple web service running in Kubernetes, and it appears to leak memory when compiled with GHC 9.2.8 or 9.4.5, but not when compiled with 9.0.2:
The first part of the graph with continuously increasing memory use is how it looks when the application is compiled with either GHC 9.2.8 or 9.4.5. The last part with lower and constant memory use is after deploying a version compiled with GHC 9.0.2. "100%" on the vertical axis is 128 MB. In all cases the same commit on the release-23.05 branch of nixpkgs was used, so library dependency versions are as similar as possible.
The application processes 100-400 requests per second, depending on time of day, and uses Servant and Warp for HTTP, and Hasql for the database. If I enable heap profiling (-hc) when compiled with GHC 9.2.8 or 9.4.5, the memory use reported by Kubernetes increases faster, but the heap profile itself shows asymptotically constant memory use of less than 3 MB:
If I understand correctly, that means the memory use is not related to pure Haskell code. We suspect the problem is related to HTTP, because we have other applications that use Hasql without trouble, and they get the traffic from Kafka instead of HTTP.
Steps to reproduce
Unfortunately we don't yet have a minimal reproducible test case, as the application has to run with traffic for a couple of days before we can be sure that the memory use is increasing, and not just fluctuating. I realise that it probably means you can't pinpoint the issue, but for now I'm hoping for general advice on how to proceed. EDIT: This minimal Warp example seems to result in similar memory use characteristics with artificial traffic generated by Siege.
Did something change in GHC 9.2 that could cause this? Are there things I can try or measurements I can perform to rule out certain possibilities?
Expected behavior
Similar memory use with GHC 9.2.8 and 9.4.5 as with 9.0.2.
Environment
- GHC version used: 9.2.8 and 9.4.5
Optional:
- Operating System: Linux
- System Architecture: x86_64