Friday, June 7, 2013

Linux Server Memory Utilization Always Above 90%?

Oracle cloud services customers were facing a long time issue with production server memory utilization always being on the higher side(> 90% Memory usage). At sometime point that seemed to be something unusual and not realistic because if the memory usage is at that high level business would never be as usual for them, but customers were only worried about the memory graph that stays at 95% most of the time even when their business was as usual. My friend Neeraj Bhatia did some research on this issue and found out a Linux bug (From Oracle's perspective) and below is the explanation of the issue found.

 Oracle Enterprise Manager (OEM grid control) pulls memory usage information from SAR output saved in the Linux servers, Which used to mislead the Memory utilization for a while, Not sure about if Oracle EM 12C Which is the current production monitoring tool) has rectified the below mentioned Linux issue. If not, the memory usage is still misleading us.

Below is the explanation for the memory usage misleading issue.




## Here "kbmemfree","kbmemused","%memused","kbbuffers","kbcached" is shown where %memused is ##merely calculated by formula Memory utilization =  kbmemused/sum(kbmemused, kbmemfree)).                                                                                                                                              
## In order to get actual memory utilization, subtract kbbuffers and kbcached from kbmemused, and ##then used the above formula.                                                                      
## In this case, Memory utilization = (kbmemused-kbbuffers-kbcached)/sum(kbmemused,kbmemfree)                                                                                            
                                                                                                                                                                       
Kbmemused  kbmemfree kbbuffers  kbcached                                          %memused       
33180580      16252192    1591888    22660668    Incorrect Formula          0.671226368     = 67%                                                                                                                       With Correct Formula           0.180609414     = 18%  
                         
The reason behind this is Linux treats unused memory as a wasted resource and so uses as much RAM as it can to cache process/kernel information.                               
Here Buffers= amount of physical memory used as buffers for disk writes Cached = amount of physical memory used as cache for disk reads                                                         
                                                                                                                                                                       
Even if you use top utility to check the memory utilization, situation seems dreadful. It appears that almost all memory is being used. This is because of the way top display memory utilization information.          


 

                                                     


in KB
in MB
in GB
Total Physical Memory
49432772
48274.19141
47.14277
Memory Used
33167868
32390.49609
31.63134
Memory Free
16264904
15883.69531
15.51142

The top command reports that total 2065628k bytes (2017MB) of memory is configured on the system, 1195636k (1167MB) is used and 869992k (849MB) is free. Some of the used memory that the top command reports is actually being used for cache. In other words, it is available to be reassigned if another process requires it.                                  
                                                                                                                                                                       
To get a better picture of memory utilization, you should use free command. free is basically a c-program that reads output from /proc virtual file system.             



 
 The free command reports output in 1024-byte blocks. I used -m switch to report memory utilization in megabytes.

We can interpret free command output as follows. In the first line, it reports that this system has 48274MB memory configured, out of which 32581MB is used and 15693 is free. The shared memory column is obsolete and should be ignored. 1554MB reported under buffers column, being used as buffers for disk writes and 22106MB used as cache for disk read operations. It should be note that memory reported under buffers and cached, is available for use. Hence in second line, they are added to free memory. Thus in actual, system has 39353 MB free for use (~ =15693+1554+22106 MB). Thus it is important to note that, we should concentrate on second line “-/+ buffers/cache:” to get free memory in the system.


The third line reports swap utilization information. Here system has 1984MB of swap configured, swap is not being used and total swap is free for use. We can also get current memory utilization using vmstat (virtual memory statistics tool).  

Here you can download a quick tool(It shows the actual host memory utilization%) that was written with expectation of  OEM or Grid control to realize this Linux bug at least in future releases(EM 12c is out now!!) SARex (SAR data extract tool)       


HAPPY LEARNING!               

9 comments:

  1. You can compile this program in C

    http://it.slashdot.org/story/06/02/06/120210/understanding-memory-usage-on-linux

    ReplyDelete
  2. nice post for trapping sar flaws

    ReplyDelete
  3. good doc to understand sar utility which I was getting misunderstood from last two years

    ReplyDelete
  4. free gets the information from /proc/meminfo

    ReplyDelete
  5. Great post. I have been struggling to understand why OEM is showing only 40% memory utilization while sar is showing 99%. This post helped me to understand the difference.

    ReplyDelete

Thanks for you valuable comments !