Linux Server Memory Utilization Always Above 90%?

Oracle cloud services customers were facing a long time issue with production server memory utilization always being on the higher side(> 90% Memory usage). At sometime point that seemed to be something unusual and not realistic because if the memory usage is at that high level business would never be as usual for them, but customers were only worried about the memory graph that stays at 95% most of the time even when their business was as usual. My friend Neeraj Bhatia did some research on this issue and found out a Linux bug (From Oracle's perspective) and below is the explanation of the issue found.

 Oracle Enterprise Manager (OEM grid control) pulls memory usage information from SAR output saved in the Linux servers, Which used to mislead the Memory utilization for a while, Not sure about if Oracle EM 12C Which is the current production monitoring tool) has rectified the below mentioned Linux issue. If not, the memory usage is still misleading us.

Below is the explanation for the memory usage misleading issue.

## Here "kbmemfree","kbmemused","%memused","kbbuffers","kbcached" is shown where %memused is ##merely calculated by formula Memory utilization =  kbmemused/sum(kbmemused, kbmemfree)).                                                                                                                                              

## In order to get actual memory utilization, subtract kbbuffers and kbcached from kbmemused, and ##then used the above formula.                                                                      

## In this case, Memory utilization = (kbmemused-kbbuffers-kbcached)/sum(kbmemused,kbmemfree)                                                                                            

                                                                                                                                                                       

Kbmemused  kbmemfree kbbuffers  kbcached                                          %memused       

33180580      16252192    1591888    22660668    Incorrect Formula          0.671226368     = 67%                                                                                                                       With Correct Formula           0.180609414     = 18%  

                         

The reason behind this is Linux treats unused memory as a wasted resource and so uses as much RAM as it can to cache process/kernel information.                               

Here Buffers= amount of physical memory used as buffers for disk writes Cached = amount of physical memory used as cache for disk reads                                                         

                                                                                                                                                                       

Even if you use top utility to check the memory utilization, situation seems dreadful. It appears that almost all memory is being used. This is because of the way top display memory utilization information.          

 

                                                     

	in KB	in MB	in GB
Total Physical Memory	49432772	48274.19141	47.14277
Memory Used	33167868	32390.49609	31.63134
Memory Free	16264904	15883.69531	15.51142

The top command reports that total 2065628k bytes (2017MB) of memory is configured on the system, 1195636k (1167MB) is used and 869992k (849MB) is free. Some of the used memory that the top command reports is actually being used for cache. In other words, it is available to be reassigned if another process requires it.                                  

                                                                                                                                                                       

To get a better picture of memory utilization, you should use free command. free is basically a c-program that reads output from /proc virtual file system.             

 

 The free command reports output in 1024-byte blocks. I used -m switch to report memory utilization in megabytes.

We can interpret free command output as follows. In the first line, it reports that this system has 48274MB memory configured, out of which 32581MB is used and 15693 is free. The shared memory column is obsolete and should be ignored. 1554MB reported under buffers column, being used as buffers for disk writes and 22106MB used as cache for disk read operations. It should be note that memory reported under buffers and cached, is available for use. Hence in second line, they are added to free memory. Thus in actual, system has 39353 MB free for use (~ =15693+1554+22106 MB). Thus it is important to note that, we should concentrate on second line “-/+ buffers/cache:” to get free memory in the system.

The third line reports swap utilization information. Here system has 1984MB of swap configured, swap is not being used and total swap is free for use. We can also get current memory utilization using vmstat (virtual memory statistics tool).  

Here you can download a quick tool(It shows the actual host memory utilization%) that was written with expectation of  OEM or Grid control to realize this Linux bug at least in future releases(EM 12c is out now!!) SARex (SAR data extract tool)       

HAPPY LEARNING!