We have:
- Kubernetes cluster with ingress points to
- Several nginx containers that proxy_pass to
- A Node application on a specific URI via location /app/
What we see:
After working for several days, at the same time all three nginx containers report upstream issues to the node app, that connection is unexpectedly closed by the client.
However, accessing the node app (direct ingress route), or curling from the nginx containers works. The issue does not seem to exist in the app itself, or we would expect to see a similar failure rate, for the same reasons as going via the nginx.
- CPU - well below the max
- Memory - well below the max
- 1024 sockets configured
- 100k file descriptors (hard and soft limits).
Our hypothesis is some form of resource is exhausted but not immediately clear what that resource is.
We are not maintaining keep-alive, but if socket / port exhaustion was the issue surely we would have seen the same behavior when logging into the container directly.
I am starting to run out of ideas - so any help is hugely appreciated.
Restarting the containers temporarily clears the issue.
The issue is believed to be nginx because of this - but very unclear as to where. Resources don't appear to be wildly different post-restart when compared with pre-restart - but it is the fact that a restart completely solves the issue, that the issue takes days to appear, that we feel it is somehow resource related.