IP Office: 100% CPU utilization causing users not able to login, not able to transfer and other issues.

Doc ID    SOLN321366
Version:    1.0
Status:    Published
Published date:    05 Feb 2018
Author:   
ralagao
 

Details

IPOSE Primary and Secondary 9.1.11

I verified that the system is at 100% utilization. They have rebooted the server last week but the problem remained. BP also told me that he wasn't able to login to root from cli.

We reset the root password from WebControl. From top command, I determined that the CPU resource was being used by YAM service (It is not YUM). This is not IPO application and is not part of the IPO installation. I verified also that the Secondary server doesn't have this.

I found that the YAM service dir was under /opt/dev0. Upon digging, I found that the dir has details about mining altcoins (These are the coins aside from bitcoin). It is clear that someone must have accessed the system and copied these files here and ran the YAM service.

Problem Clarification

They were having issues where in phones are not able to login, not able to transfer and other related issues. They have noticed that the CPU utilization of the Primary server is staying at 100% utilization.

Cause

There was a service called YAM (Not the linux YUM) which was consuming almost all the CPU resource. It is non-IPO related service. Someone must have copied the files there and ran it.

Solution

I killed the service and deleted the dir. Right after killing the service, the CPU utilization moved down to normal. I asked BP to schedule restart of the system to verify the service hasn't come back. I also recommended him to change all the password.

*If you have similar issue where CPU utilization is high, you may run TOP command in cli. Verify which service is using the most CPU resource and move forward from there.


Avaya -- Proprietary. Use pursuant to the terms of your signed agreement or Avaya policy