AEP: Mpp process is down post reboot

Doc ID    SOLN360351
Version:    1.0
Status:    Published
Published date:    03 Mar 2021
Author:   
kkanthboppan
 

Details

Avaya Aura Experience Portal.
Version 7.2.0
Single Server with AEP & MPP on it

Problem Clarification

Mpp service doesn't start automaticly after Reboot of the system.
It seems that the mpp process is stuck and not restarted post reboot.

Cause

From the logs, we see that the System Manager process is exceeding the resource limits of Max stack size from the logs.

@2020-12-17 18:57:09,588|PSYSM00044|INFO|SysMgr|8101|FileName=mppsysmgr.cpp,LineNumber=577|MPP System Manager starting with resource limits Max stack size : soft=4096K, hard=4096K :: Max open files : soft=16384, hard=16384 : PrimaryEPM=(IPV4)172.16.101.33(count=0) : MPP base=MPPEnv:/opt/Avaya/ExperiencePortal/MPP : AAEP base=AvayaEnv:/opt/Avaya/ExperiencePortal|S1100pvo479####
@2020-12-17 18:57:31,513|PSYSM00055|INFO|SysMgr|8202|FileName=mppsysmgr.cpp,LineNumber=1412|Primary EPM resumed polling|S1100pvo479####
@2020-12-17 18:57:33,674|PSYSM00036|INFO|SysMgr|8202|FileName=mppsysmgr.cpp,LineNumber=3183|New MPP command MMS_ConfigureMpp in state Stopped - new state = Stopped|S1100pvo479####

This is stack full is causing the MPP process to go down and not allow it to come up post reboot.

@2020-12-17 18:57:54,054|PSYSM00036|INFO|SysMgr|8202|FileName=mppsysmgr.cpp,LineNumber=3183|New MPP command MMS_StartMpp in state Stopped - new state = Starting|S1100pvo479####
@2020-12-17 18:58:09,602|PSYSM00056|ERROR|SysMgr|8101|FileName=mppsysmon.cpp,LineNumber=695|MPP filesystem / disk use has reached 67 percent, exceeding the configured threshold of 60 percent.|S1100pvo479####
@2020-12-17 18:58:10,090|PSYSM00036|INFO|SysMgr|8202|FileName=mppsysmgr.cpp,LineNumber=3183|New MPP command MMS_ConfigureMpp in state Running - new state = Running|S1100pvo479####
@2020-12-17 18:58:10,126|PSYSM00036|INFO|SysMgr|8202|FileName=mppsysmgr.cpp,LineNumber=3183|New MPP command MMS_AddStations in state Running - new state = Running|S1100pvo479####
@2020-12-17 18:58:21,428|PSYSM00036|INFO|SysMgr|8202|FileName=mppsysmgr.cpp,LineNumber=3228|New MPP command OCWS_QueryResources in state Running - new state = Running|S1100pvo479####
@2020-12-17 18:58:51,467|PSYSM00036|INFO|SysMgr|8202|FileName=mppsysmgr.cpp,LineNumber=3228|New MPP command OCWS_QueryResources in state Running - new state = Running|S1100pvo479####
@2020-12-17 18:59:21,496|PSYSM00036|INFO|SysMgr|8202|FileName=mppsysmgr.cpp,LineNumber=3228|New MPP command OCWS_QueryResources in state Running - new state = Running|S1100pvo479####
@2020-12-17 18:59:51,525|PSYSM00036|INFO|SysMgr|8202|FileName=mppsysmgr.cpp,LineNumber=3228|New MPP command OCWS_QueryResources in state Running - new state = Running|S1100pvo479####
@2020-12-17 19:00:21,552|PSYSM00036|INFO|SysMgr|8202|FileName=mppsysmgr.cpp,LineNumber=3228|New MPP command OCWS_QueryResources in state Running - new state = Running|S1100pvo479####
@2020-12-17 19:00:51,580|PSYSM00036|INFO|SysMgr|8202|FileName=mppsysmgr.cpp,LineNumber=3228|New MPP command OCWS_QueryResources in state Running - new state = Running|S1100pvo479####
@2020-12-17 19:01:21,610|PSYSM00036|INFO|SysMgr|8202|FileName=mppsysmgr.cpp,LineNumber=3228|New MPP command OCWS_QueryResources in state Running - new state = Running|S1100pvo479####
@2020-12-17 19:25:18,306|PSYSM00044|INFO|SysMgr|25567|FileName=mppsysmgr.cpp,LineNumber=577|MPP System Manager starting: soft=4096K, hard=4096K :: Max open files : soft=16384, hard=16384 : PrimaryEPM=(IPV4)172.16.101.33(count=0) : MPP base=MPPEnv:/opt/Avaya/ExperiencePortal/MPP : AAEP base=AvayaEnv:/opt/Avaya/ExperiencePortal|S1100pvo479####
@2020-12-17 19:25:20,536|PSYSM00055|INFO|SysMgr|25669|FileName=mppsysmgr.cpp,LineNumber=1412|Primary EPM resumed polling|S1100pvo479####
@2020-12-17 19:25:22,706|PSYSM00036|INFO|SysMgr|25669|FileName=mppsysmgr.cpp,LineNumber=3183|New MPP command MMS_ConfigureMpp in state Stopped - new state = Stopped|S1100pvo479####

Solution

 Suggest below steps:
  1. Go to folder /opt/Avaya/ExperiencePortal/MPP/config - and vi mpp script change the ULIMIT_MAX_OPEN_FILE_DESCRIPTORS=30000 (This would be by default ULIMIT_MAX_OPEN_FILE_DESCRIPTORS=16384)
  2. Go to /opt/Avaya/ExperiencePortal/MPP folder and vi mppmon.conf change the number of retries of restarting the mpp to desired number.
  • mpp : noforce :5 --> make it to 10 and see if that helps

Avaya -- Proprietary. Use pursuant to the terms of your signed agreement or Avaya policy