Remote PN (G650) lose connectivity with the main server therefore CLANs go uncontrolled by the CM software as indicated in the ECS logs:
PN13 (IPSI 13A & 13B) WENT OOS:
20160810:183534256:194135914:pcd(9262):MED:[[12:1] checkSlot: too many sanity failures (15)]
20160810:183534256:194135915:pcd(9262):MED:[[12:0] checkSlot: too many sanity failures (15)]
20160810:183534256:194135916:pcd(9263):MED:[[12:0] errorSocket: socket error - E_ERROR (err code 4)]
20160810:183534256:194135917:pcd(9263):MED:[[12:0] errorSocket: recovering socket (stage 0)]
20160810:183534256:194135918:pcd(9263):MED:[[12:1] errorSocket: socket error - E_ERROR (err code 4)]
20160810:183534256:194135919:pcd(9263):MED:[[12:1] errorSocket: recovering socket (stage 0)]
20160810:183534256:194135920:pcd(9262):MED:[[12:1] Pcd_wfatal: FATAL reported (0x10a) -> FATAL]
20160810:183534256:194135921:pcd(9262):MED:[[12:1] Pcd_hw_rep: HW report to mtce (code 1, aux 266)]
20160810:183534256:194135922:hmm(9278):MED:[ MTCEVT ERR type=0101 lname=1029 pn1=00000000 pn2=0c006100 aux=0000010a rc=0]
20160810:183534257:194135923:fastmap(30884):MED:[IPSI_SOHEVL ipsi_pn2=0x0c006100 flt_cls=2 soh_chg=1 soh_sts=1 A=0.0.0.0 B=0.1.0.0]
20160810:183534257:194135924:fastmap(30884):MED:[IPSI_CNTRL: complete for pnn 12, rc 0]
20160810:183534257:194135925:pcd(9262):MED:[[12:1] Pcd_hdl_mtc: mtce requesting PCD_RESET]
20160810:183534257:194135926:pcd(9262):MED:[[12:1] Pcd_reset: reset failure on un-connected PKTINT]
20160810:183534258:194135927:pcd(9263):MED:[[12:1] updFatal: FATAL set for PKTINT but cannot report to mtce]
20160810:183534276:194135928:pcd(9263):MED:[[12:1] updFatal: FATAL set for PKTINT but cannot report to mtce]
20160810:183534276:194135929:pcd(9262):MED:[[12:1] Pcd_wfatal: FATAL reported (0x10a) -> FATAL]
20160810:183534276:194135930:pcd(9262):MED:[[12:1] Pcd_hw_rep: HW report to mtce (code 1, aux 266)]
20160810:183534276:194135931:pcd(9262):MED:[[12:1] Pcd_hw_rep: PCD_RESET failure on PKTINT]
20160810:183619464:194137772:hmm(9278):MED:[PORT-NETWORK 13 UNAVAILABLE (WARM PENDING).]
20160810:183619465:194137774:pcd(9263):MED:[[12:0] errorSocket: socket error - E_SOCKET_DOWN]
20160810:183619465:194137775:pcd(9263):MED:[[12:0] errorSocket: recovering socket (stage 0)]
20160810:183619466:194137781:gip(9950):MED:[clanSesUpdt: ETHERNET link 73 is down/busy! actionIn 0]
20160810:183619467:194137785:gip(9950):MED:[clanSesUpdt: ETHERNET link 70 is down/busy! actionIn 0]
20160810:183619467:194137787:gip(9950):MED:[clanSesUpdt: ETHERNET link 71 is down/busy! actionIn 0]
20160810:183619468:194137789:gip(9950):MED:[clanSesUpdt: ETHERNET link 72 is down/busy! actionIn 0]
20160810:183619482:194137802:pcd(9262):MED:[[12:1] Pcd_hdl_mtc: mtce requesting PCD_RESET]
20160810:183619482:194137803:pcd(9262):MED:[[12:1] Pcd_reset: reset failure on un-connected PKTINT]
20160810:183619482:194137804:pcd(9263):MED:[[12:1] updFatal: FATAL set for PKTINT but cannot report to mtce]
20160810:183619483:194137805:pcd(9262):MED:[[12:0] Pcd_hdl_mtc: mtce requesting PCD_RESET]
20160810:183619483:194137806:pcd(9262):MED:[[12:0] Pcd_reset: reset failure on un-connected PKTINT]
20160810:183619483:194137807:pcd(9263):MED:[[12:0] updFatal: FATAL set for PKTINT but cannot report to mtce]
20160810:183619494:194137848:pcd(9262):MED:[[12:1] Pcd_hdl_mtc: mtce requesting PCD_OUT_SERV]
20160810:183619495:194137849:pcd(9262):MED:[[12:0] Pcd_hdl_mtc: mtce requesting PCD_OUT_SERV]
PN13 RESTORED:
20160810:183619603:194137878:pcd(9262):MED:[[12:1] proSmsg: IPSI connection is up]
20160810:183619604:194137879:pcd(9262):MED:[[12:0] proSmsg: IPSI connection is up]
20160810:183620443:194137896:hmm(9278):MED:[PORT-NETWORK 13 UNAVAILABLE (COLD PENDING).]
20160810:183724941:194141642:fg_mapa(16068):MED:[RESET PORT-NETWORK 13 LEVEL 2 (COLD) PERFORMED.]
20160810:183724941:194141643:hmm(9278):MED:[PORT-NETWORK 13 AVAILABLE.]
The above log file entries show that CLAN(s) in PN13 were out of service and unable to serve out registration request and other signaling related functions. To avoid such issues the network link between the main CM server(s) and the remote IPSIs has to be stabile without transport delays or link drops.
