OOM Killer Issues Troubleshooting

OOM Killer Issues Troubleshooting

When receiving an OOM killer load alert notification via email, your next move should be to delve into what’s causing the issue and find a solution. To assist you, we have already examined the most common issues and proposed effective solutions to prevent their recurrence. Here are the three primary process groups we’ve identified:

  1. Common Cases
  2. Memory Leak Processes
  3. Non-Leaking Processes

Common Cases

Take a look at the table below to find solutions for frequently terminated processes identified by the OOM tool:

These are the background tasks that automatically run on any platform container and can be stopped by the OOM tool.

Process Resolution 
adt You might want to consider restarting the container to get the process back up and running smoothly.
cron You might want to consider restarting the container to get the process back up and running smoothly.
crond You might want to consider restarting the container to get the process back up and running smoothly.
expect It’s possible that this issue arises from activities like git fetch, git pull, or git gc processes. This might be due to the project’s large size or a slow internet connection.
git You can start the update from GIT by clicking on the button next to your project on the dashboard. If you prefer, you can also just wait for the next auto-deploy to run (if it’s turned on), and it will take care of restoring the git process for you.
git-remote-http It’s possible that this issue arises from activities like git fetch, git pull, or git gc processes. This might be due to the project’s large size or a slow internet connection.
gitlab-projects It’s possible that this issue arises from activities like git fetch, git pull, or git gc processes. This might be due to the project’s large size or a slow internet connection.
jem If you’ve been using the platform dashboard for tasks like deploying applications, installing SSL, or changing the engine lately, one of those operations might not have gone through successfully. Maybe you should give it another attempt.
nscd You might want to consider restarting the container to get the process back up and running smoothly.
ssh You might want to consider restarting the container to get the process back up and running smoothly.
sshd You might want to consider restarting the container to get the process back up and running smoothly.
systemd You might want to consider restarting the container to get the process back up and running smoothly.
systemd-journal You might want to consider restarting the container to get the process back up and running smoothly.
taskrunner You might want to consider restarting the container to get the process back up and running smoothly.

Processes with a High Risk of Memory Leak

We address potential memory leaks by taking specific actions or optimizing the application code. These issues are categorized depending on the stack type or programming language used. Each category offers tailored suggestions for general improvements and solutions to terminate particular processes.

Focus closely on the procedures within this group, as they are likely the primary reason for your out-of-memory problems. Sort them based on server role for better understanding.

Load Balancers

Common recommendations

Increase the amount of RAM assigned to the relevant node since the services it manages may simply need additional memory to function properly.

Related processes

Process Resolution
varnishd Increase the amount of memory allocated to a node – some of the services it’s managing might simply need more RAM to function properly.

Application Servers

Click on points within the list below for helpful tips on managing memory shortages, tailored to the programming language you’re using, along with effective solutions for handling intensive processes.

Common recommendationsTake a look at the primary settings for managing memory in your Java setup. If necessary, tweak them to suit your specific application requirements. For instance:

When you set up your Java environment, consider adjusting memory allocation using parameters like:

java -Xmx2048m -Xms256m

Here’s what these flags mean:

The Xmx flag determines the maximum heap memory the Java Virtual Machine (JVM) can use.

The Xms flag sets the initial memory allocation pool.

Tip: The platform also incorporates additional automated memory management for Java containers through Garbage Collector. You can tailor its settings to match the specifics of your application, helping to prevent out-of-memory (OOM) problems and optimize memory usage more effectively.Keep in mind that the JVM requires more memory than just what’s allocated for the heap. I recommend delving into the Java Memory Structure reference for a more thorough understanding.

Related processes

Process Resolution
java Make sure to review the xmx, xms, and xmn settings on your Java virtual machine, and adjust them based on the requirements of your application.
Common recommendations1. If you’re facing an issue with the httpd (httpd.itk) service, here’s what you can do to fix it:

  • Start by checking how much RAM each httpd instance is using on average.
  • Look into the /etc/httpd/httpd.conf file and remove any Jelastic autoconfiguration marks you find.
  • Adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%) / Average RAM.
Note: If you notice that the memory usage keeps going up steadily for each instance (like a leak), it’s a sign that you should lower the MaxRequestsPerChild setting to somewhere between 1000 and 5000.

2. To check the size of php-fpm instances within your nginx process, hop into your container through SSH and give a look using tools like ps or top. Here’s a simple guide:

  • If you notice that each php-fpm instance is gobbling up around 50-100Mb of RAM, it’s time to take action. Consider turning off auto-configuration and reducing the max_children parameter.
  • However, if you find that the sizes of instances are all over the place or surpassing 200-300Mb, there might be a leak somewhere in your process. It’s advisable to dig into your code, spot the issue, and optimize it. Alternatively, you can also opt to disable auto-configuration and trim down the max_requests_per_child parameter.

Related processes

Process Resolution 
httpd 1. First, find out how much RAM each httpd instance typically uses on average.

2. Next, locate and delete the Jelastic autoconfiguration tag in the /etc/httpd/httpd.conf file.

3. Then, adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%)/Average RAM.

lsyncd If a node is handling a lot of services, it might benefit from having extra RAM allocated to it. Sometimes, those services simply need more memory to run smoothly.
httpd.itk 1. First, find out how much RAM each httpd instance typically uses on average.

2. Next, locate and delete the Jelastic autoconfiguration tag in the /etc/httpd/httpd.conf file.

3. Then, adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%)/Average RAM.

nginx Turn off the automatic setup in Jelastic and fine-tune the settings to match the unique needs of your application.
php Turn off the automatic setup in Jelastic and fine-tune the settings to match the unique needs of your application.
php-fpm Turn off the automatic setup in Jelastic and fine-tune the settings to match the unique needs of your application.
php-fpm7.0 Turn off the automatic setup in Jelastic and fine-tune the settings to match the unique needs of your application.
php7.0 Turn off the automatic setup in Jelastic and fine-tune the settings to match the unique needs of your application.
Common recommendationsMemory leaks are something Ruby developers often encounter. Start by going over and optimising your code before taking on this. Another approach could be raising the RAM limit for your instance.

Related processes

Process Resolution 
httpd 1. First, find out how much RAM each httpd instance typically uses on average.

2. Next, locate and delete the Jelastic autoconfiguration tag in the /etc/httpd/httpd.conf file.

3. Then, adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%)/Average RAM.

httpd.itk 1. First, find out how much RAM each httpd instance typically uses on average.

2. Next, locate and delete the Jelastic autoconfiguration tag in the /etc/httpd/httpd.conf file.

3. Then, adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%)/Average RAM.

bundle If a node is handling a lot of services, it might benefit from having extra RAM allocated to it. Sometimes, those services simply need more memory to run smoothly.
gem If a node is handling a lot of services, it might benefit from having extra RAM allocated to it. Sometimes, those services simply need more memory to run smoothly.
ruby Maybe you should take a look at your code and see if there’s room for improvement, or you could try adding some more RAM to a node.
Common recommendations1. If you’re encountering issues with the httpd (httpd.itk) service, you can troubleshoot by adjusting the server’s memory management settings in the following manner:

  • First, check the average RAM consumption of each httpd instance.
  • Next, locate and remove the Jelastic autoconfiguration mark found in the /etc/httpd/httpd.conf file.
  • Then, tweak the ServerLimit and MaxClients values. You may compute these values using the following formula: (Total RAM – 5%) / Average RAM.
Note: If you’re seeing memory usage steadily climb for each instance, like there’s a leak, try dialing down the MaxRequestsPerChild value to somewhere between 1000 and 5000.

2. If not, you might want to consider giving the node more RAM. Sometimes, the main Python process simply needs extra memory to run smoothly.

Related processes

Process Resolution 
httpd 1. First, find out how much RAM each httpd instance typically uses on average.

2. Next, locate and delete the Jelastic autoconfiguration tag in the /etc/httpd/httpd.conf file.

3. Then, adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%)/Average RAM.

lsyncd Sometimes, you might need to give a node more RAM because the application it’s running simply needs extra memory to work smoothly.
httpd.itk 1. First, find out how much RAM each httpd instance typically uses on average.

2. Next, locate and delete the Jelastic autoconfiguration tag in the /etc/httpd/httpd.conf file.

3. Then, adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%)/Average RAM.

php Network problems can sometimes lead to downloads getting stuck, while on other occasions, it might be necessary to assign additional RAM to a node. In such cases, the application being used might simply need more memory to function properly.
python Sometimes, you might need to give a node more RAM because the application it’s running simply needs extra memory to work smoothly.
python2.7 Sometimes, you might need to give a node more RAM because the application it’s running simply needs extra memory to work smoothly.
Common recommendationsIf you’re having trouble with a process getting killed, try restarting the container. If the problem persists, adding more RAM to the node may assist. Sometimes, certain applications need more memory to run smoothly.

Related processes

Process Resolution 
lsyncd Sometimes, you might need to give a node more RAM because the application it’s running simply needs extra memory to work smoothly.
grunt Sometimes, you might need to give a node more RAM because the application it’s running simply needs extra memory to work smoothly.
node Sometimes, you might need to give a node more RAM because the application it’s running simply needs extra memory to work smoothly.
npm Sometimes, you might need to give a node more RAM because the application it’s running simply needs extra memory to work smoothly.
phantomjs Sometimes, you might need to give a node more RAM because the application it’s running simply needs extra memory to work smoothly.

Database Servers

Just select the relevant database stack from the list below to find helpful advice on managing out-of-memory problems and fixes for certain process terminations.

1. If you’re working with the InnoDB engine, which has been included since MySQL version 5.5, you’ll want to keep an eye on buffer sizes. This can be done by running the following command:SHOW ENGINE INNODB STATUS\G;

If you notice that the buffers are taking up a significant portion of your server’s RAM (over 80%), it’s a good idea to adjust the size of the buffer pool. You can do this by tweaking the innodb_buffer_pool_size parameter in the /etc/my.cnf file. Alternatively, if the buffers aren’t using up too much RAM, consider allocating more memory to your server.

2. Additionally, it’s wise to regularly check the MySQL logs for any warnings or recommendations.

Related processes

Process Resolution 
httpd 1. First, find out how much RAM each httpd instance typically uses on average.

2. Next, locate and delete the Jelastic autoconfiguration tag in the /etc/httpd/httpd.conf file.

3. Then, adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%)/Average RAM.

mysqld 1. If you’re working with the InnoDB engine (which is the default for MySQL versions 5.5 and above), you can assess buffer size by using the command SHOW ENGINE INNODB STATUS\G. If you notice that the buffers are taking up a large portion of your system’s memory (more than 80% of the total RAM), it’s advisable to adjust the size of the buffer pool by modifying the innodb_buffer_pool_size parameter in the /etc/my.cnf file.

2. Additionally, it’s a good practice to regularly review the MySQL logs for any warnings or recommendations that might indicate potential issues.

Common recommendationsIf you’re encountering an issue with the httpd service, you can try tweaking some server memory settings:

First off, take a look at how much RAM each httpd instance is gobbling up on average.

Next, go into the /etc/httpd/httpd.conf file and hunt down the Jelastic autoconfiguration mark. Get rid of it.

Lastly, decrease the ServerLimit and MaxClients values. A good rule of thumb is to use this formula: (Total_RAM – 5%) / Average_RAM to figure out the new values.

Note: If you see that the memory use keeps going up for each thing running (we call this a “leak”), try reducing the MaxRequestsPerChild setting to somewhere between 1000 and 5000.

Related processes

Process Resolution 
httpd 1. First, find out how much RAM each httpd instance typically uses on average.

2. Next, locate and delete the Jelastic autoconfiguration tag in the /etc/httpd/httpd.conf file.

3. Then, adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%)/Average RAM.

mongod If a node is handling a lot of services, it might benefit from having extra RAM allocated to it. Sometimes, those services simply need more memory to run smoothly.
Common recommendationsIncrease the amount of RAM assigned to the relevant node since the services it manages may simply need additional memory to function properly.

Related processes

Process Resolution 
httpd 1. First, find out how much RAM each httpd instance typically uses on average.

2. Next, locate and delete the Jelastic autoconfiguration tag in the /etc/httpd/httpd.conf file.

3. Then, adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%)/Average RAM.

postgres If a node is handling a lot of services, it might benefit from having extra RAM allocated to it. Sometimes, those services simply need more memory to run smoothly.
Common recommendationsIncrease the amount of RAM assigned to the relevant node since the services it manages may simply need additional memory to function properly.

Related processes

Process Resolution 
redis-server If a node is handling a lot of services, it might benefit from having extra RAM allocated to it. Sometimes, those services simply need more memory to run smoothly.

Common Processes for Different-Type Stacks

Common recommendations

In this part, you can start and stop processes across various types of nodes. So, how we handle out-of-memory (OOM) situations depends on the specific process. Take a look at the table below for the right guidance.

Related processes

Process Stack Resolution 
httpd PHP

Ruby

Python

MySQL

MongoDB

PostgreSQL

1. First, find out how much RAM each httpd instance typically uses on average.

2. Next, locate and delete the Jelastic autoconfiguration tag in the /etc/httpd/httpd.conf file.

3. Then, adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%)/Average RAM.

lsyncd PHP

NodeJS

Python

If a node is handling a lot of services, it might benefit from having extra RAM allocated to it. Sometimes, those services simply need more memory to run smoothly.
httpd.itk PHP

Ruby

Python

1. First, find out how much RAM each httpd instance typically uses on average.

2. Next, locate and delete the Jelastic autoconfiguration tag in the /etc/httpd/httpd.conf file.

3. Then, adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%)/Average RAM.

procmail Any You might want to consider restarting the container to get the process back up and running smoothly.
vsftpd Any You might want to consider restarting the container to get the process back up and running smoothly.
yum Any You might want to consider restarting the container to get the process back up and running smoothly.
cc1 3rd party If a node is handling a lot of services, it might benefit from having extra RAM allocated to it. Sometimes, those services simply need more memory to run smoothly.
clamd 3rd party If a node is handling a lot of services, it might benefit from having extra RAM allocated to it. Sometimes, those services simply need more memory to run smoothly.
ffmpeg 3rd party If a node is handling a lot of services, it might benefit from having extra RAM allocated to it. Sometimes, those services simply need more memory to run smoothly.
firefox 3rd party If a node is handling a lot of services, it might benefit from having extra RAM allocated to it. Sometimes, those services simply need more memory to run smoothly.
newrelic-daemon 3rd party Restart the primary stack service, which include nginx, tomcat, nodejs.

Non-Leaking Processes

Sometimes, certain operations might get cut off by the OOM killer, even if they’re not the main issue. A common solution to tackle these types of problems is to simply restart the container, which helps to revive the affected processes.

The processes mentioned here might get stopped by the OOM killer to clear up RAM for other tasks, but usually, they don’t create memory leak issues on their own.

Process Resolution
112×4 Restart container in order to restore the process
112×6 Restart container in order to restore the process
ahn Restart container in order to restore the process
anytermd Restart container in order to restore the process
apache2 Restart container in order to restore the process
apparmor-profil Restart container in order to restore the process
beam.smp Restart container in order to restore the process
dbus-daemon Restart container in order to restore the process
firewall.sh Restart container in order to restore the process
freshclam Restart container in order to restore the process
getty Restart container in order to restore the process
inimuxsock Restart container in order to restore the process
install-info Restart container in order to restore the process
iojs Restart container in order to restore the process
iptables-save Restart container in order to restore the process
kindlegen Restart container in order to restore the process
Lxz6 Restart container in order to restore the process
mda Restart container in order to restore the process
mingetty Restart container in order to restore the process
named Restart container in order to restore the process
nano Restart container in order to restore the process
nheqminer_cpu Restart container in order to restore the process
nrsysmond Restart container in order to restore the process
pidproxy Restart container in order to restore the process
PM2 Restart container in order to restore the process
postmaster Restart container in order to restore the process
rnd Restart container in order to restore the process
rpc.statd Restart container in order to restore the process
rpcbind Restart container in order to restore the process
rpm Restart container in order to restore the process
rsmain Restart container in order to restore the process
rsyslogd Restart container in order to restore the process
run-parts Restart container in order to restore the process
running-in-cont Restart container in order to restore the process
saslauthd Restart container in order to restore the process
sed Restart container in order to restore the process
semodule Restart container in order to restore the process
sendmail Restart container in order to restore the process
service Restart container in order to restore the process
sh Restart container in order to restore the process
simplehttp.py Restart container in order to restore the process
sleep Restart container in order to restore the process
startpar-upstar Restart container in order to restore the process
status Restart container in order to restore the process
stress Restart container in order to restore the process
stunnel Restart container in order to restore the process
supervisord Restart container in order to restore the process
syslog-ng Restart container in order to restore the process
tesseract Restart container in order to restore the process
upstart-file-br Restart container in order to restore the process
upstart-socket- Restart container in order to restore the process
upstart-udev-br Restart container in order to restore the process
valhalla_run_ro Restart container in order to restore the process
xinetd Restart container in order to restore the process

Start by finding the name of the process mentioned in the email. The solution you need can be found by looking through the documents linked above.