cms link alarm caused by CM Maintenance

Doc ID    SOLN318232
Version:    7.0
Status:    Published
Published date:    25 Mar 2019
Created Date:    15 Nov 2017
Author:   
Ming Jiang
 

Details

CMS All version

(Alarm Name) : ACDLINK , (Alarm String) : 11/21:25,ACT|CMS,r13.1da.g,3,13,ACDLINK3,MAJ
Tue Nov 14 18:54:48 [20] SESS ST8camack timer expired, link is not responding...
Tue Nov 14 18:55:27 [00] PBX DATAX 20:17:00 yday 318 of 17 ==============
Tue Nov 14 18:56:27 [00] PBX DATAX 20:18:00 yday 318 of 17 ==============
Tue Nov 14 18:57:27 [00] PBX DATAX 20:19:00 yday 318 of 17 ==============
Tue Nov 14 18:58:31 [00] PBX DATAX 20:20:00 yday 318 of 17 ==============
Tue Nov 14 18:59:26 [55] PBX DATAX timer expired
Tue Nov 14 18:59:26 [55] TCP CLIENT DISCONNECTED
Tue Nov 14 18:59:31 [55] TCP client connect timed out
Tue Nov 14 18:59:31 [55] TCP OUT OF ORDER
Tue Nov 14 18:59:36 [55] TCP client connect timed out
Tue Nov 14 18:59:36 [55] PBX link or protocol failure
Tue Nov 14 18:59:36 [55] PBX 4 calls ignored
Tue Nov 14 18:59:36 [55] PBX call ignored


Tue Nov 14 19:33:30 [57] SESS DATAX
Tue Nov 14 19:33:30 [57] PBX RTCS
Tue Nov 14 19:33:31 [06] PBX BUSYOUT
Tue Nov 14 19:33:31 [06] PBX BUSY-OUT
Tue Nov 14 19:33:41 [17] PBX RELBUSY
Tue Nov 14 19:33:41 [17] PBX RTCS
Tue Nov 14 19:33:42 [17] PBX requesting logon translations
Tue Nov 14 19:33:42 [17] PBX PUMP-UP
Tue Nov 14 19:33:42 [17] PBX PUMP-UP 20:55:17 11/14/17 00000 calls ======
Tue Nov 14 19:33:43 [17] PBX PUMP-UP 20:55:18 11/14/17 00000 calls ======

Link alarm  caused by reboot

Problem Clarification

Tue Nov 14 18:59:36 [55] PBX call ignored
Tue Nov 14 18:59:36 [55] PBX ApplQ sess down note
Tue Nov 14 18:59:36 [55] PBX WSA
Tue Nov 14 18:59:36 [55] PBX ApplQ sess down note
Tue Nov 14 18:59:39 [00] PBX WSA 20:21:00 yday 318 of 17 ==============
Tue Nov 14 18:59:54 [14] SESS WSA timeout, retrying...
Tue Nov 14 18:59:59 [14] TCP client connect timed out
Tue Nov 14 19:00:18 [34] SESS WSA timeout, retrying...
Tue Nov 14 19:00:23 [34] TCP client connect timed out
Tue Nov 14 19:00:42 [54] SESS WSA timeout, retrying...
Tue Nov 14 19:00:47 [54] TCP client connect timed out
Tue Nov 14 19:00:51 [00] PBX WSA 20:22:00 yday 318 of 17 ==============
Tue Nov 14 19:01:06 [14] SESS WSA timeout, retrying...
Tue Nov 14 19:01:11 [14] TCP client connect timed out
Tue Nov 14 19:01:30 [34] SESS WSA timeout, retrying...
Tue Nov 14 19:01:35 [34] TCP client connect timed out
Tue Nov 14 19:01:54 [54] SESS WSA timeout, retrying...
Tue Nov 14 19:01:59 [54] TCP client connect timed out
Tue Nov 14 19:02:03 [00] PBX WSA 20:23:00 yday 318 of 17 ==============
Tue Nov 14 19:02:18 [14] SESS WSA timeout, retrying...
Tue Nov 14 19:02:23 [14] TCP client connect timed out
Tue Nov 14 19:02:42 [34] SESS WSA timeout, retrying...

Cause

CM Maintenance

So the outage is a bit more than half an hour from 18:59 to 19:33. And the reason is CM does not respond to CMS connection request (client connect timed out, timer expired). Mostly this CM ACD3, was under maintenance. So after maintenance  the mis link from CM was released.

Solution

(SECONDARY)=# ./active_alarms
[21328:New Connection (r3xxxx,IT_daemon,*,root,pid=399,optimised) ]
[21328:New IIOP Connection (r3xxxx:4001) ]
[21328:New IIOP Connection (xx.xxx.xxx.xxx:4012) ]
Number of alarm event records = 1
Product Alarm Severity Date Time
CMS ACDLINK3 WARNING 11/14/2017 19:02:36


(SECONDARY)=# ./alarm_resolve
[21946:New Connection (r3xxxx,IT_daemon,*,root,pid=399,optimised) ]
[21946:New IIOP Connection (r3xxxx:4001) ]
[21946:New IIOP Connection (xx.xxx.xxx.xxx:4012) ]
Command executed successfully

In r16.3 and earlier alarm_resolve is run by the following command:
/cms/aom/bin/alarm_resolve

In r17 and later run it like this:
/opt/cc/aot/bin/alarm_resolve

or

cd /cms/aom/bin and run ./alarm_resolve

Clear alarms and close the SR.


Avaya -- Proprietary. Use pursuant to the terms of your signed agreement or Avaya policy