Skip to content
Prompt Words
History

OOM kill

Also called: out of memory, OOMKilled, exit code 137.

A container has a memory limit. When the program inside uses more than that, the Linux kernel kills it, and Kubernetes shows OOMKilled with exit code 137 and restarts it. For a Java app the memory is the heap (-Xmx) plus everything outside the heap, so a heap set too close to the limit gets the container killed.

A Java app in a pod with a 4 GiB memory limit. Its memory is the heap (-Xmx) plus 0.6 GiB outside the heap. Try -Xmx3g, then -Xmx3800m.

Heap: 0 GiB · Non-heap: 0 GiB · Total: 0 of 4 GiB · Restarts: 0

Nothing running yet. The container may use 4 GiB in all: heap and non-heap together.

    Idle

    Say it in a prompt

    Our Java service in Kubernetes keeps restarting with OOMKilled (exit code 137). The container limit is 4Gi and the JVM runs with -Xmx3800m. Replace -Xmx3800m with -XX:MaxRAMPercentage=75 (an explicit -Xmx wins over it), so the heap is about 3 GiB and threads, metaspace and buffers fit in the rest. Set requests.memory equal to the limit, and alert when container memory goes over 90% of the limit.

    Vague vs precise prompt

    Vague prompt

    the service keeps crashing, give it more memory

    Typical resultRaises -Xmx to match the new limit, so the heap plus threads and metaspace still go past it, and the pod is killed again a little later.

    Precise prompt

    The pod is OOMKilled (exit 137) at a 4Gi limit with -Xmx3800m. Replace -Xmx3800m with -XX:MaxRAMPercentage=75 (an explicit -Xmx wins over it) so the heap is about 3 GiB and the rest of the JVM fits, and alert above 90% of the limit.

    Typical resultThe heap stops at about 3 GiB and garbage collection works inside it; threads, metaspace and buffers fit in the last GiB, so the kernel no longer kills the container.

    Seen on

    • Kubernetes docs: Assign Memory Resources: a container that goes over its memory limit is terminated with reason OOMKilled and exit code 137, and the kubelet restarts it.
    • Kubernetes docs: Resource Management: memory limits are enforced by the kernel with out-of-memory (OOM) kills when a container uses more than its limit.
    • Java docs (the java command): -Xmx sets the maximum size of the heap. In its -XX:AllocateHeapAt entry, the page names the code heap, metaspace and thread stacks as separate memory structures.

    You might describe it as

    • the pod keeps restarting with exit code 137
    • the container dies with no error in the app's logs
    • memory goes over the container limit and it gets killed

    Not to be confused with

    • Streaming results

      An OOM kill is what happens when a program's memory goes past its container's limit; streaming results is one way to avoid it, by reading big data in small pieces.