Current analysis:
For MPP1
In below report, MPP station are invoking the application and ‘end details’ points towards an application error.
Start Time
|
Voice Portal
|
MPP
|
Application
|
Originating #
|
Destination #
|
Duration
|
End Type
|
End Details
|
11/1/12 11:37:19 AM
|
VoicePortal
|
MPP1
|
IVR GEN 2 Production
|
|
|
14
|
Near End Disconnect
|
NearEndDisconnect : Application exited. [ERRORS]
|
11/1/12 11:36:02 AM
|
VoicePortal
|
MPP1
|
IVR GEN 2 Production
|
AA Inbound P
|
|
30
|
Near End Disconnect
|
NearEndDisconnect : Application exited. [ERRORS]
|
11/1/12 11:35:49 AM
|
VoicePortal
|
MPP1
|
IVR GEN 2 Production
|
|
|
2
|
Near End Disconnect
|
NearEndDisconnect : Application exited. [ERRORS]
|
11/1/12 11:35:37 AM
|
VoicePortal
|
MPP1
|
Prod Call
|
|
|
2
|
Near End Disconnect
|
NearEndDisconnect : Application exited. [ERRORS]
|
11/1/12 11:35:36 AM
|
VoicePortal
|
MPP1
|
IVR GEN 2 Production
|
|
|
2
|
Near End Disconnect
|
NearEndDisconnect : Application exited. [ERRORS]
|
11/1/12 11:35:05 AM
|
VoicePortal
|
MPP1
|
IVR GEN 2 Production
|
|
|
62
|
Near End Disconnect
|
NearEndDisconnect : Application exited. [ERRORS]
|
11/1/12 11:35:05 AM
|
VoicePortal
|
MPP1
|
IVR GEN 2 Production
|
|
|
35
|
Near End Disconnect
|
NearEndDisconnect : Application exited. [ERRORS]
|
Even in alarm report we can see many application related errors are reported
11/1/12 11:34:44 AM
|
UNACK
|
MPP1
|
VP Session Manager
|
Minor
|
QSESM00070
|
PSESM00070
|
QSESM00070: Application errors occurred for an application
|
11/1/12 11:07:12 AM
|
UNACK
|
MPP1
|
VP Session Manager
|
Minor
|
QSESM00070
|
PSESM00070
|
QSESM00070: Application errors occurred for an application
|
From the available log it doesn’t seems that MPP was having any issue.
MPP logging was not enabled to any level,so no information have been captured.
For MPP5
MPP logs of MPP5 are showing ASR errors. No other error are seen.
ASR errors:
@2012-11-09 10:21:56,169|PASR_00001|WARN|ASR|31168|FileName=asr/MrcpASRProvider.cpp,LineNumber=185|Could not find an available ASR server.|AAAAAA00100815####
@2012-11-09 10:21:56,169|PASR_00001|WARN|ASR|31168|FileName=asr/MrcpASRProvider.cpp,LineNumber=185|Could not find an available ASR server.|AAAAAA00100815####
@2012-11-09 10:21:56,169|PASR_00001|WARN|ASR|31168|FileName=asr/MrcpASRProvider.cpp,LineNumber=185|Could not find an available ASR server.|AAAAAA00100815####
@2012-11-09 10:21:56,169|PASR_00003|WARN|ASR|31168|FileName=asr/MrcpASRProvider.cpp,LineNumber=303|Failed to get an ASR port!|AAAAAA00100815####
@2012-11-09 10:21:56,169|PSESM00011|INFO|SessMgr|31168|FileName=SpeechCtrl.cpp,LineNumber=1046|ASR connect to server failed to initialize with error 'no ASR resource available'|AAAAAA00100815####
@2012-11-09 10:21:56,220|PASR_00001|WARN|ASR|31168|FileName=asr/MrcpASRProvider.cpp,LineNumber=185|Could not find an available ASR server.|AAAAAA00100815####
@2012-11-09 10:21:56,220|PASR_00001|WARN|ASR|31168|FileName=asr/MrcpASRProvider.cpp,LineNumber=185|Could not find an available ASR server.|AAAAAA00100815####
@2012-11-09 10:21:56,220|PASR_00001|WARN|ASR|31168|FileName=asr/MrcpASRProvider.cpp,LineNumber=185|Could not find an available ASR server.|AAAAAA00100815####
@2012-11-09 10:21:56,220|PASR_00003|WARN|ASR|31168|FileName=asr/MrcpASRProvider.cpp,LineNumber=303|Failed to get an ASR port!|AAAAAA00100815####
@2012-11-09 10:21:56,220|PSESM00011|INFO|SessMgr|31168|FileName=SpeechCtrl.cpp,LineNumber=1046|ASR connect to server failed to initialize with error 'no ASR resource available'|AAAAAA00100815####
This was probably the time when MPP was stopped:
@2012-11-12 11:27:45,767|PSESM00059|INFO|SessMgr|2517|FileName=SessionMgr.cpp,LineNumber=1775|Session Manager is shutting down with reason ProcessStop and grace period 300|AAAAAA00100815####
@2012-11-12 11:32:15,966|PSESM00058|INFO|SessMgr|26607|FileName=SessionMgr.cpp,LineNumber=846|Session Manager is initialized and ready...|AAAAAA00100815####
@2012-11-12 11:32:36,853|PMRCP00009|INFO|MRCP|26667|FileName=asr/MrcpASRPort.cpp,LineNumber=1008|Established connection to the server [Nuance Production OSR]. Port:1 State:IDLE MRCPSess: VPSess: Server:aaaaaa00100915.r3.money.ge.com/media/speechrecognizer File:asr/MrcpASRPort.cpp Line:1008|AAAAAA00100815####
@2012-11-12 11:32:36,853|PMRCP00009|INFO|MRCP|26667|FileName=asr/MrcpASRPort.cpp,LineNumber=1008|Established connection to the server [Nuance Production OSR 3]. Port:3 State:IDLE MRCPSess: VPSess: Server:aaaaaa00101117.r3.money.ge.com/media/speechrecognizer File:asr/MrcpASRPort.cpp Line:1008|AAAAAA00100815####
@2012-11-12 11:32:36,854|PMRCP00009|INFO|MRCP|26667|FileName=tts/MrcpTTSPort.cpp,LineNumber=970|Established connection to the server [Production TTS server]. Port:1 State:IDLE MRCPSess: VPSess: Server:aaaaaa00100916.r3.money.ge.com/media/speechsynthesizer File:tts/MrcpTTSPort.cpp Line:970|AAAAAA00100815####
VPMS logs User datacom stopped/started the MPP
@2012-11-12 11:27:40,294 EST|POMS_00203:messages.vpmsMsgCode|INFO|OMS|VoicePortal|TP-Processor22|User=datacom|Stop requested for MPPs: MPP5|String:MPP5||||||vpms|1234567890####
@2012-11-12 11:27:40,297 EST|POMS_00208:messages.vpmsMsgCode|INFO|OMS|VoicePortal|TP-Processor22|User=datacom|Stopping MPPs: MPP5|String:MPP5|String:Stop|||||vpms|1234567890####
@2012-11-12 11:27:40,297 EST|POMS_00208:messages.vpmsMsgCode|INFO|OMS|VoicePortal|TP-Processor22|User=datacom|Stopping MPPs: MPP5|String:MPP5|String:Stop|||||vpms|1234567890####
@2012-11-12 11:32:14,443 EST|PADMN01115:messages.vpmsMsgCode|INFO|Administration|VoicePortal|TP-Processor28|User=datacom|Starting MPPs MPP5.|String:MPP5||||||vpms|1234567890####
@2012-11-12 11:32:14,443 EST|PADMN01115:messages.vpmsMsgCode|INFO|Administration|VoicePortal|TP-Processor28|User=datacom|Starting MPPs MPP5.|String:MPP5||||||vpms|1234567890####
@2012-11-12 11:32:14,809 EST|POMS_00004:messages.vpmsMsgCode|INFO|OMS|VoicePortal|VP-MPP-Interface||Start command sent successfully to MPP5|String:MPP5||||||vpms|1234567890####
: @2012-11-12 11:32:14,809 EST|POMS_00004:messages.vpmsMsgCode|INFO|OMS|VoicePortal|VP-MPP-Interface||Start command sent successfully to MPP5|String:MPP5||||||vpms|1234567890####
Recommendation:
· if such issue occurs again, please enable the MPP logging to finest level ( except for Telephony: set it to Finer) and make some test calls. This will populate into MPP logs the information which can help in finding the root cause. Then the MPPs can be restarted. And logging set back to Off or Fine.
· It is also recommended to reboot the Voice Portal servers at least every 6 months.