If you set the JVM memory arguments Xmx and Xms, what are you actually doing? You're setting the heap size right? So, if I run 4 java processes using 5gb each on a machine with, say, 8gb, I've maxed the memory, right? Well, not necessarily.
This was the output from top for a Linux box running a dozen or so JVMs with Xmx4g:
top - 17:31:05 up 187 days, 12:24, 6 users, load average: 1.75, 1.20, 0.88
Tasks: 293 total, 1 running, 292 sleeping, 0 stopped, 0 zombie
Cpu(s): 5.6%us, 0.5%sy, 0.0%ni, 93.7%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 48274M total, 29096M used, 19178M free, 439M buffers
Swap: 8191M total, 990M used, 7201M free, 4889M cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28448 mds 17 0 5325m 4.8g 6600 S 1 10.1 502:39.47 java
29368 mds 17 0 4718m 1.8g 5668 S 0 3.8 74:00.60 java
16996 mds 17 0 4711m 1.7g 15m S 10 3.5 1:40.33 java
15366 mds 17 0 4661m 1.5g 15m S 0 3.3 0:59.42 java
15365 mds 17 0 4654m 1.5g 15m S 0 3.2 0:54.01 java
16997 mds 17 0 4711m 1.5g 15m S 0 3.1 1:01.36 java
.
.
The RESident space is for all but one less than 4gb. The VIRTual memory is over 4gb. Which do we trust?
First, what is RESident and VIRTual memory?
"A virtual memory scheme splits memory used by each program into small, fixed-size units called pages... At any one time, only some of the pages of a program need to be resident in physical memory page frames; these pages for the so-called resident set. Copies of the unused pages of a program are maintained in the swap area - a reserved area of disk space used to supplement the computer's RAM - and loaded into physical memory only as required." [1]
But this machine is barely using swap space. So, the missing memory is not there.
"When your JVM starts it reserves your maximum heap size as one continuous block of virtual memory on startup. However, only the pages actually used become resident pages (actual main memory). When you set the minimum size, it doesn't force the JVM to use that much memory if it doesn't need it." [2]
"Address space (ie. virtual memory in the process list) doesn't cost anything; it's not real. What's real is the RSS (RES) column, which is resident memory." [3]
So, we're setting something that is "not real". What's the point? Well, we're setting logical memory limitations, not physical. When a garbage collection is started and the size of the different areas of the garbage collector (eden, old etc) is a function of this parameter.
This is all became relevant for us this week because one of our boxes went wild and starting killing Java processes. Note: it is not necessarily the offender who is punished. The Linux OOM Killer [4] calculates who is the best process to kill using an algorithm that takes into account a number of variables (you can see who is most likely to die as the score is shown in the file /proc/PID/oom_score).
[1] The Linux Programming Interface, Kerrisk
[2] StackOverflow.
[3] ServerFault.
[4] Blog on the Linux OOM Killer.
This was the output from top for a Linux box running a dozen or so JVMs with Xmx4g:
top - 17:31:05 up 187 days, 12:24, 6 users, load average: 1.75, 1.20, 0.88
Tasks: 293 total, 1 running, 292 sleeping, 0 stopped, 0 zombie
Cpu(s): 5.6%us, 0.5%sy, 0.0%ni, 93.7%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 48274M total, 29096M used, 19178M free, 439M buffers
Swap: 8191M total, 990M used, 7201M free, 4889M cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28448 mds 17 0 5325m 4.8g 6600 S 1 10.1 502:39.47 java
29368 mds 17 0 4718m 1.8g 5668 S 0 3.8 74:00.60 java
16996 mds 17 0 4711m 1.7g 15m S 10 3.5 1:40.33 java
15366 mds 17 0 4661m 1.5g 15m S 0 3.3 0:59.42 java
15365 mds 17 0 4654m 1.5g 15m S 0 3.2 0:54.01 java
16997 mds 17 0 4711m 1.5g 15m S 0 3.1 1:01.36 java
.
.
The RESident space is for all but one less than 4gb. The VIRTual memory is over 4gb. Which do we trust?
First, what is RESident and VIRTual memory?
"A virtual memory scheme splits memory used by each program into small, fixed-size units called pages... At any one time, only some of the pages of a program need to be resident in physical memory page frames; these pages for the so-called resident set. Copies of the unused pages of a program are maintained in the swap area - a reserved area of disk space used to supplement the computer's RAM - and loaded into physical memory only as required." [1]
But this machine is barely using swap space. So, the missing memory is not there.
"When your JVM starts it reserves your maximum heap size as one continuous block of virtual memory on startup. However, only the pages actually used become resident pages (actual main memory). When you set the minimum size, it doesn't force the JVM to use that much memory if it doesn't need it." [2]
"Address space (ie. virtual memory in the process list) doesn't cost anything; it's not real. What's real is the RSS (RES) column, which is resident memory." [3]
So, we're setting something that is "not real". What's the point? Well, we're setting logical memory limitations, not physical. When a garbage collection is started and the size of the different areas of the garbage collector (eden, old etc) is a function of this parameter.
This is all became relevant for us this week because one of our boxes went wild and starting killing Java processes. Note: it is not necessarily the offender who is punished. The Linux OOM Killer [4] calculates who is the best process to kill using an algorithm that takes into account a number of variables (you can see who is most likely to die as the score is shown in the file /proc/PID/oom_score).
[1] The Linux Programming Interface, Kerrisk
[2] StackOverflow.
[3] ServerFault.
[4] Blog on the Linux OOM Killer.
No comments:
Post a Comment