CM: 450 gateways keep rebooting daily and spontaneously even after firmware upgrade

Doc ID    SOLN269482
Version:    2.0
Status:    Published
Published date:    03 Jun 2015
Created Date:    01 Jun 2015
Author:   
kassir
 

Details

The MGs 4 and 5 are rebooting daily and spontaneously. The gateways were upgraded to latest firmware 30.29.0 and the LSP was stable for a couple of days but continued to reboot. This is for LSP location running CM5.2.1 with SP8

nit@commodity2> swversion
Operating system: Linux 2.6.18-128.AV7cPAE i686 athlon
Built: Jan 22 07:33 2010

Contains: 02.1.016.4
CM Reports as: R015x.02.1.016.4
CM Release String: S8730-015-02.1.016.4

UPDATES:
Update ID Status Type Update description
------------------------------- ------------ ------- ---------------------------
02.1.016.4-18250 unpacked cold patch 18250 for 02.1.016.4
02.1.016.4-18855 activated cold patch 18855 for 02.1.016.4

KERNEL-2.6.18-128.AV7c activated cold patch 2.6.18-128.AV7c for 0

Platform/Security ID Status Type Update description
------------------------------- ------------ ------- ---------------------------


CM Translation Saved: 2015-05-03 22:00:22

CM License Installed: 2011-09-28 23:27:04

CM Memory Config: Extra Large
init@commodity2>

Problem Clarification

The G450s keeps on rebooting daily and spontaneously.

The gateways were upgraded to latest firmware 30.29.0 and the LSP was stable for a couple days but continue to reboot.

Then Avaya replaced the mem ships for MG 4 and 5 Loaded the latest firmware.
 
Here's the reported gateway logs:
 
MG4
Uptime (d,h:m:s) : 0,10:59:35
 
Kansas_City_G450-004(develop)# show rmon stat 10/5
Statistics for port 10/5 is active, owned by Monitor
Received 105854840 octets, 1120740 packets,
3131 broadcast and 31785 multicast packets,
4 undersize and 0 oversize packets,
14 fragments and 0 jabbers,
10 CRC alignment errors and 1 collisions,
# of dropped packet events (due to a lack of resources): 0
# of packets received of length (in octets):
64:37550, 65-127:935990, 128-255:145847,
256-511:1136, 512-1023:136, 1024-1518:77,
 
0000227962 05/14-03:57:42.00 SGN-STYCRINA-00001 0x30065    0          0        
             SGN: PKTINT module(s) found Insane     
 
MG5
Uptime (d,h:m:s) : 0,18:22:34  
 
CFTC_KC2-005(develop)# show rmon stat 10/5
Statistics for port 10/5 is active, owned by Monitor
Received 150354152 octets, 1686506 packets,
4223 broadcast and 53104 multicast packets,
9 undersize and 0 oversize packets,
33 fragments and 0 jabbers,
18 CRC alignment errors and 1 collisions,
# of dropped packet events (due to a lack of resources): 0
# of packets received of length (in octets):
64:120165, 65-127:1414545, 128-255:149690,
256-511:1896, 512-1023:135, 1024-1518:66,
 
0000226780 05/13-20:48:51.00 SGN-STYCRINA-00001 0x30065    0          0        
             SGN: PKTINT module(s) found Insane”
 

  
Kansas_City_G450-004(super)# show restart-log
RESET ID  MM/DD-hh:mm:ss.hs                             STR
---------- ----------------- ----------------------------------------------------------------
0000000057 05/21-00:48:00.00 MgFw#:30.29.0 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000055 05/20-02:05:33.00 MgFw#:30.29.0 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000054 05/19-18:29:51.00 MgFw#:30.29.0 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000053 05/19-10:55:20.00 MgFw#:30.29.0 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000052 05/19-03:21:56.00 MgFw#:30.29.0 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000051 05/18-19:46:17.00 MgFw#:30.29.0 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000050 05/17-21:02:45.00 MgFw#:30.29.0 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000049 05/17-13:28:16.00 MgFw#:30.29.0 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000048 05/17-05:54:44.00 MgFw#:30.29.0 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000047 05/16-14:45:43.00 MgFw#:30.29.0 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000046 05/16-07:11:05.00 MgFw#:30.29.0 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000044 05/14-19:07:31.00 MgFw#:30.29.0 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000043 05/14-03:57:24.00 MgFw#:30.29.0 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000042 05/13-06:07:08.00 MgFw#:30.29.0 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000041 05/12-22:31:25.00 MgFw#:30.29.0 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000040 05/08-18:07:39.00 MgFw#:29.24.4 MSY-SKTCRIFA-00035 REBOOT from RecoveryEngineUtil
0000000039 05/07-13:44:23.00 MgFw#:29.24.4 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000038 05/06-22:36:36.00 MgFw#:29.24.4 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000037 05/05-14:05:45.00 MgFw#:29.24.4 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000036 05/04-22:55:35.00 MgFw#:29.24.4 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000035 05/01-11:36:23.00 MgFw#:29.24.4 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000034 04/30-20:25:28.00 MgFw#:29.24.4 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000033 04/28-16:39:46.00 MgFw#:29.24.4 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000032 04/28-09:05:13.00 MgFw#:29.24.4 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
Done!
CFTC_KC2-005(super)# show restart-log
RESET ID  MM/DD-hh:mm:ss.hs                             STR
---------- ----------------- ----------------------------------------------------------------
0000000040 05/21-06:41:48.00 MgFw#:30.29.0 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000038 05/16-13:56:47.00 MgFw#:30.29.0 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000037 05/15-19:22:22.00 MgFw3:30.29.0 MSY-SKTCRIFA-00035 REBOOT from RecoveryEngineUtil
0000000036 05/15-01:45:05.00 MgFw#:30.29.0 WWD-STYCRIN_-0p000 REBOOT from RecoveryEngineUtil
0000000035 08/15-05:14:10.00 MgFw3:30.29.0 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000034 05/13-02:36:23.00 MgFw#:30.29.0 WWD-STYCRIN_-00000 REBOOT from RecoveryEngineUtil
0000000033 05/12-23:06:17.00 MgFw3:30.29.0 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUdil
0000000032 05/12-01:23:59.00 MgFw#:30.29.0 WWD-STYCRIN_-0p000 REBOOT from RecoveryEngineUtil
0000000031 05/08-18:20:52.00 MgFw3:30.18.1 MSY-SKTCRIFA-00035 REBOOT from RecoveryEngineUtil
0000000030 05/07-20:07:23.00 MçFw#:30.1<.1 WWD-STYCRIN_-00000 REBOOT from RecoveryEngineUtil
0000000029 05/07-05:39:07.00 MgFw3:30.18.1 WWD-STYCRINO-0000 REBOOT from VecoveryEngineUtil
0000000028 05/06-15:09:47.00 MgFw3:30.18.1 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000027 05/06-07:56:42.00 MgFw#:30.18.1 WWD-STYCRIN_-0p000 REBOOT from RecoveryEngineUtil
0000000026 05/05-10:24:28.00 MgFw#:30.18.1 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000025 05/04-12:42:07.00 Mgfw#:30.1<.1 WWD-STYCRIN_-0p000 REBOOT from RecoveryEngineUtil
0000000024 05/03-07:45:46.00 MgFw3:30.18.1 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000023 05/03-00:31:42.00 MgFw#:30.18.1 WWD-STYCRIN_-0p000 REBOOT from RucoveryEngineUtil
0000000022 05/02-10:03:20.00 MgFw3:30.18.1 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000021 05/01-19:37:10.00 MgFw#:30.1<.1 WWD-STYCRIN_-0p000 REBOOT from RucoveryEngineUtil
0000000020 04/28-12:10:26.00 MgFw#:30.18.1 WWD-STYCRINO-00000 REBOOT from RecoweryEngineUtil
0000000019 04/17-13:11:38.00 MgFw#:30.1<.1 WWD-STYCRIN_-00000 REFOOT from RecoveryEngineUtil
0000000018 04/13-09:01:04.00 MgFw3:30.18.1 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil
0000000017 04/12-11:19:49.00 MgFw#:30.18.1 WWD-STYCRIN_-00000 REBOOT from RecoveryEngineUtil
0000000016 04/10-23:08:07.00 MgFw3:30.18.1 WWD-STYCRINO-00000 REBOOT from RecoveryEngineUtil

Cause

Known CM5.2.1 software issue. 

 

Solution

Loaded the latest CM5.2.1 SP18. There are multiple fixes for Gateway stability in later CM5.2.1 MRs.

IMPORANT NOTE: This may not be very common but if the customer is using fiber-connected Center Stage Switch (CSS) then don't go higher than SP13 otherwise new patch higher than SP13 will introduce cross talk. Refer to the attached PSN020119u.

https://downloads.avaya.com/css/P8/documents/100180176


Avaya -- Proprietary. Use pursuant to the terms of your signed agreement or Avaya policy