We have multiple, JVM11-based apps running on k8s. A couple of weeks ago Kubernetes killed all of the pods from the same deployment at the same time in our testing environment. The reason was OOMKilled.

The exciting part is that we have set the -Xmx to 3200m, and the pods have a memory request of 4Gi, limit of 5Gi. The heap size is correctly set, the app can run with the 3200Mi heap.

During the investigation the following looked interesting:

1. the memory usage of the pod increases during the morning, and never decreases. The pods starts with a new dataset every midnight, and the same behaviour happens every day. We have peak usage during the morning and during the afternoon hours, but the pod’s memory usage never decreases. Not even in the nonpeak time during the middle of the day.
   

(some new pods spin up during these peak hours, but we have a few pods that run all day. the picture is from one all-day-running pod)

2. it’s a spring boot application, and the micrometer framework is set up. On the dashboards I can see the following:

• the heap never extends 3200Mi, which is the expected behavior
• the max of the nonheap memory usage is way higher than the used/committed values
• the max of the JVM Total memory usage is equal to the pod’s memory usage. And it’s equal to sum(max-nonheap, max-heap).
  

According to the Prometheus metrics, the nonheap part of the JVM is allocated of (jvm_memory_used_bytes{container="<mycontainer>", area="nonheap"}):

• CodeHeap ‘non-nmethods’
• CodeHeap ‘non-profiled nmethods’
• CodeHeap ‘profiled nmethods’
• Compressed Class Space
• Metaspace

What’s interesting is, that the jvm_memory_max_bytes is reasonable for every nonheap part, except for the Compressed Class Space.

3. The Compressed Class Space never uses more than ~20-30 megabytes of RAM (jvm_memory_used_bytes), but as far as I understand, it always allocates the 1 gigabyte jvm_memory_max_bytes.

I’ve read a couple of docs, and all mention that the default value of the -XX:CompressedClassSpaceSize is 1 gigabyte.

Am I correctly assuming that the JVM allocates the whole gigabyte? If so, is it safe to set the -XX:CompressedClassSpaceSize to 256M for example? See the metrics below, we never use more than ~20 megabytes of memory on the CompressedClassSpaceSize part of the memory. Is there any downside for it?

Thanks,
Regards,