Customer was looking for a cause for the outage.
Followed these steps to find the cause:
A. Checked Active and Resolved alarms:
ID MO Source On Bd Lvl Ack Date
1 A CMG 26 WRN N Sun Jun 25 09:56:31 EDT 2017
2 A CMG 26 WRN N Sun Jun 25 09:56:31 EDT 2017
3 A CMG 26 WRN N Sun Jun 25 09:56:31 EDT 2017
4 A CMG 26 WRN N Sun Jun 25 09:56:30 EDT 2017
5 A CMG 26 WRN N Sun Jun 25 09:56:30 EDT 2017
6 A CMG 26 WRN N Sun Jun 25 09:56:30 EDT 2017
7 A CMG 26 WRN N Sun Jun 25 09:56:30 EDT 2017
8 A CMG 26 WRN N Sun Jun 25 09:56:31 EDT 2017
9 A CMG 26 WRN N Sun Jun 25 09:56:30 EDT 2017
10 A CMG 26 WRN N Sun Jun 25 09:56:30 EDT 2017
11 A CMG 26 WRN N Sun Jun 25 09:56:29 EDT 2017
12 A CMG 26 WRN N Sun Jun 25 09:56:29 EDT 2017
13 A CMG 26 WRN N Sun Jun 25 09:56:29 EDT 2017
14 A CMG 26 WRN N Sun Jun 25 09:56:29 EDT 2017
15 A CMG 26 WRN N Sun Jun 25 09:56:30 EDT 2017
16 A CMG 26 WRN N Sun Jun 25 09:56:30 EDT 2017
17 A CMG 26 WRN N Sun Jun 25 09:56:30 EDT 2017
18 A CMG 26 WRN N Sun Jun 25 09:56:29 EDT 2017
19 A CMG 26 WRN N Sun Jun 25 09:56:29 EDT 2017
20 A CMG 26 WRN N Sun Jun 25 09:56:29 EDT 2017
21 A CMG 26 WRN N Sun Jun 25 09:56:28 EDT 2017
22 A CMG 26 WRN N Sun Jun 25 09:56:28 EDT 2017
23 A CMG 26 WRN N Sun Jun 25 09:56:28 EDT 2017
24 A CMG 26 WRN N Sun Jun 25 09:56:28 EDT 2017
B. Checked for interchange and/or restarts:
display initcauses
INITIALIZATION CAUSES
Cause Action Escalated Mode Time
Internal Request 1 no Standby 08/21 06:29
Internal Request 1 no Standby 08/21 06:29
Initialized 4 no Standby 11/06 06:12
Internal Request 1 no Standby 11/06 06:12
Internal Request 1 no Standby 11/06 06:12
Interchange-Craft 1 no Active 11/06 06:15
Interchange-Craft 4 no Standby 02/05 06:17
Internal Request 1 no Standby 02/05 06:17
Internal Request 1 no Standby 02/05 06:17
Initialized 4 no Standby 02/05 06:30
Internal Request 1 no Standby 02/05 06:30
Internal Request 1 no Standby 02/05 06:30
Initialized 4 no Standby 05/06 06:19
Internal Request 1 no Standby 05/06 06:19
Internal Request 1 no Standby 05/06 06:19
Interchange-Craft 1 no Active 05/06 06:24
SERVER STATUS
Cluster ID: 001
Duplication: sw
Standby Busied? no
Standby Refreshed? yes
Standby Shadowing: on
Duplication Link: up
Elapsed Time since Init/Interchange: 52d 05:44:02
cm02a cm02b
ID: 001 (1) ID: 002 (2)
Mode: Active Mode: Standby
Major Alarms: no Major Alarms: no
Minor Alarms: no Minor Alarms: no
Control Network: 0 / 0 / 0 Control Network: 0 / 0 / 0
Processor Ethernet: up Processor Ethernet: up
PE Priority: high PE Priority: high
Server Hardware: okay Server Hardware: okay
Processes: okay Processes: okay
C. Reviewed logs in the Active and Standby server. Found no issues or data logged by the standby server at this time which matches the fact that an interchange did not take place. See the attached log for the findings from the Active server. The only thing showing in the log to indicate an issue is
20170625:071507826:5091980:capro(9226):MED:[ DENYEVT ERR event=2055 d1=000b d2=713be578]
20170625:071507826:5091981:hmm(9221):MED:[ MTCEVT ERR type=0321 lname=5400 pn1=00000807 pn2=0000000b aux=713be578 rc=0]
20170625:071507826:5091982:hmm(9221):MED:[ MTCEVT ERR type=0301 lname=1424 pn1=00000000 pn2=0000000b aux=00000000 rc=0]
20170625:071507826:5091983:capro(9226):MED:[ConnClosed: MG=#11 disconnected: socket closure, moved to link-bounce staten: near_ipaddr = 10.115.xx.xx, far_ipaddr = 10.101.x.xx actIntf 0]
Denial Event 2055 - call process report that there's issues on the H.248 link
Next move - disconnected the socket connection to several Gateways to reset the link.
No hardware issues found. No software issues found. The Gateways showed that they lost communication with the CM then the H.248 link was down - all at the same time the CM reported an issue on the link as well. The Gateways reported the issue from the far end, the CM side. Per the logs, the outage lasted less than 60 seconds. Not all Gateway links were bounced.
