CM 6.2 : RCA : Customer had an outage today on one of multiple PN's PN. Need to know why there was an outage.

Doc ID    SOLN286931
Version:    6.0
Status:    Published
Published date:    20 May 2022
Created Date:    06 Apr 2016
Author:   
townsd
 

Details

swversion
     Operating system:  Linux 2.6.18-238.AV2axen i686 i686
                Built:  Mar 23 17:55 2012

             Contains:  02.0.823.0
        CM Reports as:  R016x.02.0.823.0
    CM Release String:  vcm-016-02.0.823.0
     Publication Date:  27 December 2011

UPDATES:
Update ID                       Status       Type    Update description
------------------------------- ------------ ------- ---------------------------
02.0.823.0-21388                activated    cold    patch 21388 for 02.0.823.0


KERNEL-2.6.18-238.AV2a          activated    cold    kernel patch KERNEL-2.6.18-

Platform/Security ID            Status       Type    Update description
------------------------------- ------------ ------- ---------------------------

Messaging ID                    Status       Type    Update description
------------------------------- ------------ ------- ---------------------------


 CM Translation Saved:   2016-07-05 02 :01:09

 CM License Installed:   2016-01-30 01 :01:21

     CM Memory Config:   Large

Problem Clarification

- Customer had an outage in Port Network 2, Port network 19 and Port network 30

 

Here is an example of checking for sanity failures in /var/log/ecs:

grep checkSlot 2017-1226*

 

Cause

- Network failure;

20160705:080613064:152481907:pcd(4819):MED:[[18:1] errorSocket: socket error - E_ERROR (err code 2)]

20160705:080613064:152481908:pcd(4819):MED:[[18:1] errorSocket: recovering socket (stage 21)]

20160705:080613084:152481909:pcd(4818):MED:[[18:1] Pcd_wfatal: FATAL reported (0x10a) -> FATAL]

20160705:080613084:152481910:pcd(4818):MED:[[18:1] Pcd_hw_rep: HW report to mtce (code 1, aux 266)]

20160705:080613084:152481911:pcd(4819):MED:[[18:1] updFatal: FATAL set for PKTINT already in Pcd_out_srv]

 

 

20160705:080613284:152481915:pcd(4819):MED:[[29:0] errorSocket: socket error - E_ERROR (err code 2)]

20160705:080613284:152481916:pcd(4819):MED:[[29:0] errorSocket: recovering socket (stage 21)]

20160705:080613304:152481917:pcd(4819):MED:[[29:0] updFatal: FATAL set for PKTINT but cannot report to mtce]

20160705:080613304:152481918:pcd(4818):MED:[[29:0] Pcd_wfatal: FATAL reported (0x10a) -> FATAL]

 

 

20160705:080624365:152482062:hmm(4830):MED:[CM6_proc_err: pro=7186,err=203,seq=20914,da1=1074987008(0x40130000),da2=1(0x1)]

20160705:080624633:152482063:pcd(4819):MED:[[1:1] IPSI: 36 03:33:30.125 (interrupt): Spitfire:  out of buffers!]

20160705:080624783:152482064:pcd(4819):MED:[[1:1] IPSI: 36 03:33:30.275 (interrupt): Spitfire:  out of buffers!]

20160705:080624825:152482065:hmm(4830):MED:[CM6_proc_err: pro=7186,err=203,seq=20914,da1=1074659328(0x400e0000),da2=1(0x1)]

20160705:080624848:152482066:pcd(4819):MED:[[1:1] IPSI: 36 03:33:30.340 (interrupt): Spitfire:  out of buffers!]

20160705:080624849:152482067:pcd(4819):MED:[[1:1] IPSI: 36 03:33:30.340 (interrupt): Spitfire:  out of buffers!]

20160705:080624932:152482068:pcd(4819):MED:[[1:0] IPSI: 36 03:31:01.500 (AASched): ERROR:(msgRetrans) no ack to AA from angel 0x42 after retrans, dropping]

20160705:080624933:152482069:pcd(4819):MED:[[1:1] IPSI: 36 03:33:30.425 (interrupt): Spitfire:  out of buffers!]

20160705:080624936:152482070:pcd(4819):MED:[[1:1] IPSI: 36 03:33:30.430 (interrupt): Spitfire:  out of buffers!]

20160705:080625082:152482071:pcd(4819):MED:[[1:0] IPSI: 36 03:31:01.650 (AASched): ERROR:(msgRetrans) no ack to AA from angel 0x42 after retrans, dropping]

20160705:080625083:152482072:pcd(4819):MED:[[1:0] IPSI: 36 03:31:01.650 (AASched): ERROR:(msgRetrans) no ack to AA from angel 0x42 after retrans, dropping]

 

Solution

- Since the customer had failover IPSI's the network was up and running.

- Checked that the failover IPSI's in PN2, PN 19 and PN 30. Informed the customer on the same.


Avaya -- Proprietary. Use pursuant to the terms of your signed agreement or Avaya policy