Capturing Heap Dumps of stateless Kubernetes pods before container termination and export to AWS S3

Kubernetes supports executing commands before the container termination. So if we need to capture heap dump in the event of container termination to analyze an issue, we can use a Kubernetes feature called pre-stop hook. We can include the command that we need to execute before container termination to the pre-stop block inside the pod configuration.

lifecycle:
  preStop:
    exec:
      command: ["/scripts/shutdown.sh"]

A pre-stop hook is executed immediately before a container is terminated. When a container needs to be terminated, the Kubelet will run the pre-stop hook, and then send a SIGTERM signal to the main process. If the process doesn’t terminate gracefully a signal will be sent to kill the process.

Even if we collect the heap dump before the termination we have another problem. Because Kubernetes pods are stateless by default, therefore if a pod is crashed or terminated the contents inside the pod container will be lost. So if we need to collect heap dumps in java apps in the event of app crash, we need a way to persist the data we collected. In this blog I am going to discuss on how to implement a solution to overcome this.

1. How to capture heap-dump before pod termination

You can add a pre-stop hook as a lifecycle policy to your pod to collect heap-dump before container termination as follows and add a volume mount to store…