OOM Killer Issues Troubleshooting
When receiving an OOM killer load alert notification via email, your next move should be to delve into what’s causing the issue and find a solution. To assist you, we have already examined the most common issues and proposed effective solutions to prevent their recurrence. Here are the three primary process groups we’ve identified:
Common Cases
Take a look at the table below to find solutions for frequently terminated processes identified by the OOM tool:
These are the background tasks that automatically run on any platform container and can be stopped by the OOM tool.
| Process | Resolution |
| adt | You might want to consider restarting the container to get the process back up and running smoothly. |
| cron | You might want to consider restarting the container to get the process back up and running smoothly. |
| crond | You might want to consider restarting the container to get the process back up and running smoothly. |
| expect | It’s possible that this issue arises from activities like git fetch, git pull, or git gc processes. This might be due to the project’s large size or a slow internet connection. |
| git | You can start the update from GIT by clicking on the button next to your project on the dashboard. If you prefer, you can also just wait for the next auto-deploy to run (if it’s turned on), and it will take care of restoring the git process for you. |
| git-remote-http | It’s possible that this issue arises from activities like git fetch, git pull, or git gc processes. This might be due to the project’s large size or a slow internet connection. |
| gitlab-projects | It’s possible that this issue arises from activities like git fetch, git pull, or git gc processes. This might be due to the project’s large size or a slow internet connection. |
| jem | If you’ve been using the platform dashboard for tasks like deploying applications, installing SSL, or changing the engine lately, one of those operations might not have gone through successfully. Maybe you should give it another attempt. |
| nscd | You might want to consider restarting the container to get the process back up and running smoothly. |
| ssh | You might want to consider restarting the container to get the process back up and running smoothly. |
| sshd | You might want to consider restarting the container to get the process back up and running smoothly. |
| systemd | You might want to consider restarting the container to get the process back up and running smoothly. |
| systemd-journal | You might want to consider restarting the container to get the process back up and running smoothly. |
| taskrunner | You might want to consider restarting the container to get the process back up and running smoothly. |
Processes with a High Risk of Memory Leak
We address potential memory leaks by taking specific actions or optimizing the application code. These issues are categorized depending on the stack type or programming language used. Each category offers tailored suggestions for general improvements and solutions to terminate particular processes.
Focus closely on the procedures within this group, as they are likely the primary reason for your out-of-memory problems. Sort them based on server role for better understanding.
Load Balancers
Common recommendations
Increase the amount of RAM assigned to the relevant node since the services it manages may simply need additional memory to function properly.
Related processes
| Process | Resolution |
| varnishd | Increase the amount of memory allocated to a node – some of the services it’s managing might simply need more RAM to function properly. |
Application Servers
Click on points within the list below for helpful tips on managing memory shortages, tailored to the programming language you’re using, along with effective solutions for handling intensive processes.
When you set up your Java environment, consider adjusting memory allocation using parameters like:
java -Xmx2048m -Xms256m
Here’s what these flags mean:
The Xmx flag determines the maximum heap memory the Java Virtual Machine (JVM) can use.
The Xms flag sets the initial memory allocation pool.
Tip: The platform also incorporates additional automated memory management for Java containers through Garbage Collector. You can tailor its settings to match the specifics of your application, helping to prevent out-of-memory (OOM) problems and optimize memory usage more effectively.Keep in mind that the JVM requires more memory than just what’s allocated for the heap. I recommend delving into the Java Memory Structure reference for a more thorough understanding.
Related processes
| Process | Resolution |
| java | Make sure to review the xmx, xms, and xmn settings on your Java virtual machine, and adjust them based on the requirements of your application. |
- Start by checking how much RAM each httpd instance is using on average.
- Look into the /etc/httpd/httpd.conf file and remove any Jelastic autoconfiguration marks you find.
- Adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%) / Average RAM.
2. To check the size of php-fpm instances within your nginx process, hop into your container through SSH and give a look using tools like ps or top. Here’s a simple guide:
- If you notice that each php-fpm instance is gobbling up around 50-100Mb of RAM, it’s time to take action. Consider turning off auto-configuration and reducing the max_children parameter.
- However, if you find that the sizes of instances are all over the place or surpassing 200-300Mb, there might be a leak somewhere in your process. It’s advisable to dig into your code, spot the issue, and optimize it. Alternatively, you can also opt to disable auto-configuration and trim down the max_requests_per_child parameter.
Related processes
| Process | Resolution |
| httpd | 1. First, find out how much RAM each httpd instance typically uses on average.
2. Next, locate and delete the Jelastic autoconfiguration tag in the /etc/httpd/httpd.conf file. 3. Then, adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%)/Average RAM. |
| lsyncd | If a node is handling a lot of services, it might benefit from having extra RAM allocated to it. Sometimes, those services simply need more memory to run smoothly. |
| httpd.itk | 1. First, find out how much RAM each httpd instance typically uses on average.
2. Next, locate and delete the Jelastic autoconfiguration tag in the /etc/httpd/httpd.conf file. 3. Then, adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%)/Average RAM. |
| nginx | Turn off the automatic setup in Jelastic and fine-tune the settings to match the unique needs of your application. |
| php | Turn off the automatic setup in Jelastic and fine-tune the settings to match the unique needs of your application. |
| php-fpm | Turn off the automatic setup in Jelastic and fine-tune the settings to match the unique needs of your application. |
| php-fpm7.0 | Turn off the automatic setup in Jelastic and fine-tune the settings to match the unique needs of your application. |
| php7.0 | Turn off the automatic setup in Jelastic and fine-tune the settings to match the unique needs of your application. |
Related processes
| Process | Resolution |
| httpd | 1. First, find out how much RAM each httpd instance typically uses on average.
2. Next, locate and delete the Jelastic autoconfiguration tag in the /etc/httpd/httpd.conf file. 3. Then, adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%)/Average RAM. |
| httpd.itk | 1. First, find out how much RAM each httpd instance typically uses on average.
2. Next, locate and delete the Jelastic autoconfiguration tag in the /etc/httpd/httpd.conf file. 3. Then, adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%)/Average RAM. |
| bundle | If a node is handling a lot of services, it might benefit from having extra RAM allocated to it. Sometimes, those services simply need more memory to run smoothly. |
| gem | If a node is handling a lot of services, it might benefit from having extra RAM allocated to it. Sometimes, those services simply need more memory to run smoothly. |
| ruby | Maybe you should take a look at your code and see if there’s room for improvement, or you could try adding some more RAM to a node. |
- First, check the average RAM consumption of each httpd instance.
- Next, locate and remove the Jelastic autoconfiguration mark found in the /etc/httpd/httpd.conf file.
- Then, tweak the ServerLimit and MaxClients values. You may compute these values using the following formula: (Total RAM – 5%) / Average RAM.
2. If not, you might want to consider giving the node more RAM. Sometimes, the main Python process simply needs extra memory to run smoothly.
Related processes
| Process | Resolution |
| httpd | 1. First, find out how much RAM each httpd instance typically uses on average.
2. Next, locate and delete the Jelastic autoconfiguration tag in the /etc/httpd/httpd.conf file. 3. Then, adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%)/Average RAM. |
| lsyncd | Sometimes, you might need to give a node more RAM because the application it’s running simply needs extra memory to work smoothly. |
| httpd.itk | 1. First, find out how much RAM each httpd instance typically uses on average.
2. Next, locate and delete the Jelastic autoconfiguration tag in the /etc/httpd/httpd.conf file. 3. Then, adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%)/Average RAM. |
| php | Network problems can sometimes lead to downloads getting stuck, while on other occasions, it might be necessary to assign additional RAM to a node. In such cases, the application being used might simply need more memory to function properly. |
| python | Sometimes, you might need to give a node more RAM because the application it’s running simply needs extra memory to work smoothly. |
| python2.7 | Sometimes, you might need to give a node more RAM because the application it’s running simply needs extra memory to work smoothly. |
Related processes
| Process | Resolution |
| lsyncd | Sometimes, you might need to give a node more RAM because the application it’s running simply needs extra memory to work smoothly. |
| grunt | Sometimes, you might need to give a node more RAM because the application it’s running simply needs extra memory to work smoothly. |
| node | Sometimes, you might need to give a node more RAM because the application it’s running simply needs extra memory to work smoothly. |
| npm | Sometimes, you might need to give a node more RAM because the application it’s running simply needs extra memory to work smoothly. |
| phantomjs | Sometimes, you might need to give a node more RAM because the application it’s running simply needs extra memory to work smoothly. |
Database Servers
Just select the relevant database stack from the list below to find helpful advice on managing out-of-memory problems and fixes for certain process terminations.
If you notice that the buffers are taking up a significant portion of your server’s RAM (over 80%), it’s a good idea to adjust the size of the buffer pool. You can do this by tweaking the innodb_buffer_pool_size parameter in the /etc/my.cnf file. Alternatively, if the buffers aren’t using up too much RAM, consider allocating more memory to your server.
2. Additionally, it’s wise to regularly check the MySQL logs for any warnings or recommendations.
Related processes
| Process | Resolution |
| httpd | 1. First, find out how much RAM each httpd instance typically uses on average.
2. Next, locate and delete the Jelastic autoconfiguration tag in the /etc/httpd/httpd.conf file. 3. Then, adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%)/Average RAM. |
| mysqld | 1. If you’re working with the InnoDB engine (which is the default for MySQL versions 5.5 and above), you can assess buffer size by using the command SHOW ENGINE INNODB STATUS\G. If you notice that the buffers are taking up a large portion of your system’s memory (more than 80% of the total RAM), it’s advisable to adjust the size of the buffer pool by modifying the innodb_buffer_pool_size parameter in the /etc/my.cnf file.
2. Additionally, it’s a good practice to regularly review the MySQL logs for any warnings or recommendations that might indicate potential issues. |
First off, take a look at how much RAM each httpd instance is gobbling up on average.
Next, go into the /etc/httpd/httpd.conf file and hunt down the Jelastic autoconfiguration mark. Get rid of it.
Lastly, decrease the ServerLimit and MaxClients values. A good rule of thumb is to use this formula: (Total_RAM – 5%) / Average_RAM to figure out the new values.
Note: If you see that the memory use keeps going up for each thing running (we call this a “leak”), try reducing the MaxRequestsPerChild setting to somewhere between 1000 and 5000.
Related processes
| Process | Resolution |
| httpd | 1. First, find out how much RAM each httpd instance typically uses on average.
2. Next, locate and delete the Jelastic autoconfiguration tag in the /etc/httpd/httpd.conf file. 3. Then, adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%)/Average RAM. |
| mongod | If a node is handling a lot of services, it might benefit from having extra RAM allocated to it. Sometimes, those services simply need more memory to run smoothly. |
Related processes
| Process | Resolution |
| httpd | 1. First, find out how much RAM each httpd instance typically uses on average.
2. Next, locate and delete the Jelastic autoconfiguration tag in the /etc/httpd/httpd.conf file. 3. Then, adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%)/Average RAM. |
| postgres | If a node is handling a lot of services, it might benefit from having extra RAM allocated to it. Sometimes, those services simply need more memory to run smoothly. |
Related processes
| Process | Resolution |
| redis-server | If a node is handling a lot of services, it might benefit from having extra RAM allocated to it. Sometimes, those services simply need more memory to run smoothly. |
Common Processes for Different-Type Stacks
Common recommendations
In this part, you can start and stop processes across various types of nodes. So, how we handle out-of-memory (OOM) situations depends on the specific process. Take a look at the table below for the right guidance.
Related processes
| Process | Stack | Resolution |
| httpd | PHP
Ruby Python MySQL MongoDB PostgreSQL |
1. First, find out how much RAM each httpd instance typically uses on average.
2. Next, locate and delete the Jelastic autoconfiguration tag in the /etc/httpd/httpd.conf file. 3. Then, adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%)/Average RAM. |
| lsyncd | PHP
NodeJS Python |
If a node is handling a lot of services, it might benefit from having extra RAM allocated to it. Sometimes, those services simply need more memory to run smoothly. |
| httpd.itk | PHP
Ruby Python |
1. First, find out how much RAM each httpd instance typically uses on average.
2. Next, locate and delete the Jelastic autoconfiguration tag in the /etc/httpd/httpd.conf file. 3. Then, adjust the ServerLimit and MaxClients values using this formula: (Total RAM – 5%)/Average RAM. |
| procmail | Any | You might want to consider restarting the container to get the process back up and running smoothly. |
| vsftpd | Any | You might want to consider restarting the container to get the process back up and running smoothly. |
| yum | Any | You might want to consider restarting the container to get the process back up and running smoothly. |
| cc1 | 3rd party | If a node is handling a lot of services, it might benefit from having extra RAM allocated to it. Sometimes, those services simply need more memory to run smoothly. |
| clamd | 3rd party | If a node is handling a lot of services, it might benefit from having extra RAM allocated to it. Sometimes, those services simply need more memory to run smoothly. |
| ffmpeg | 3rd party | If a node is handling a lot of services, it might benefit from having extra RAM allocated to it. Sometimes, those services simply need more memory to run smoothly. |
| firefox | 3rd party | If a node is handling a lot of services, it might benefit from having extra RAM allocated to it. Sometimes, those services simply need more memory to run smoothly. |
| newrelic-daemon | 3rd party | Restart the primary stack service, which include nginx, tomcat, nodejs. |
Non-Leaking Processes
Sometimes, certain operations might get cut off by the OOM killer, even if they’re not the main issue. A common solution to tackle these types of problems is to simply restart the container, which helps to revive the affected processes.
The processes mentioned here might get stopped by the OOM killer to clear up RAM for other tasks, but usually, they don’t create memory leak issues on their own.
| Process | Resolution |
| 112×4 | Restart container in order to restore the process |
| 112×6 | Restart container in order to restore the process |
| ahn | Restart container in order to restore the process |
| anytermd | Restart container in order to restore the process |
| apache2 | Restart container in order to restore the process |
| apparmor-profil | Restart container in order to restore the process |
| beam.smp | Restart container in order to restore the process |
| dbus-daemon | Restart container in order to restore the process |
| firewall.sh | Restart container in order to restore the process |
| freshclam | Restart container in order to restore the process |
| getty | Restart container in order to restore the process |
| inimuxsock | Restart container in order to restore the process |
| install-info | Restart container in order to restore the process |
| iojs | Restart container in order to restore the process |
| iptables-save | Restart container in order to restore the process |
| kindlegen | Restart container in order to restore the process |
| Lxz6 | Restart container in order to restore the process |
| mda | Restart container in order to restore the process |
| mingetty | Restart container in order to restore the process |
| named | Restart container in order to restore the process |
| nano | Restart container in order to restore the process |
| nheqminer_cpu | Restart container in order to restore the process |
| nrsysmond | Restart container in order to restore the process |
| pidproxy | Restart container in order to restore the process |
| PM2 | Restart container in order to restore the process |
| postmaster | Restart container in order to restore the process |
| rnd | Restart container in order to restore the process |
| rpc.statd | Restart container in order to restore the process |
| rpcbind | Restart container in order to restore the process |
| rpm | Restart container in order to restore the process |
| rsmain | Restart container in order to restore the process |
| rsyslogd | Restart container in order to restore the process |
| run-parts | Restart container in order to restore the process |
| running-in-cont | Restart container in order to restore the process |
| saslauthd | Restart container in order to restore the process |
| sed | Restart container in order to restore the process |
| semodule | Restart container in order to restore the process |
| sendmail | Restart container in order to restore the process |
| service | Restart container in order to restore the process |
| sh | Restart container in order to restore the process |
| simplehttp.py | Restart container in order to restore the process |
| sleep | Restart container in order to restore the process |
| startpar-upstar | Restart container in order to restore the process |
| status | Restart container in order to restore the process |
| stress | Restart container in order to restore the process |
| stunnel | Restart container in order to restore the process |
| supervisord | Restart container in order to restore the process |
| syslog-ng | Restart container in order to restore the process |
| tesseract | Restart container in order to restore the process |
| upstart-file-br | Restart container in order to restore the process |
| upstart-socket- | Restart container in order to restore the process |
| upstart-udev-br | Restart container in order to restore the process |
| valhalla_run_ro | Restart container in order to restore the process |
| xinetd | Restart container in order to restore the process |
Start by finding the name of the process mentioned in the email. The solution you need can be found by looking through the documents linked above.