Communication Manager: IPSI keeps bouncing.

Doc ID    SOLN267110
Version:    8.0
Status:    Published
Published date:    16 Jan 2019
Created Date:    17 Apr 2015
Author:   
rhondari
 

Details

Any Communication Manager version

Problem Clarification

IPSI sanity and socket errors or inability to add them to the community after that hardware had been replaced.

Alarm Data: CMMINOR-CMMAJOR(POW-SUP, SYS-LINK 38A0101 38A15),MAJ:

Both the a and b of a specific IPSI were impacted. Boards were reseated. Busy, reset and released.

Logs revealed constant sanity and socket errors as well as the IPSI's going in and out of service. The associated boards were replaced after which the sanity and socket errors ceased however neither could be joined to the cluster.

ESS Alarm: CL 000 ESS 513 04/10/01:25 2 2 0 a y

Error 513 means that the ess is unable to contact all of the IPSI it is programmed to see (in this case 38a/38b).

The following two commands mean that its internal firewall isn’t blocking ports.

root@ess-server2> grep '5010|5011|5012' /etc/opt/ecs/ip_fw_cn.conf
CM -A OUTPUT -p tcp --dport 5010 -j ACCEPT
CM -A OUTPUT -p tcp --dport 5011 -j ACCEPT
CM -A OUTPUT -p tcp --dport 5012 -j ACCEPT

root@ess-server2> grep '5010|5011|5012' /etc/opt/ecs/ip_fw_admin.conf
CM pcd-ipsi 5010/tcp
CM ipsivsn 5011/tcp
CM ipsilic 5012/tcp

Neither restarting the internal firewall nor the IPSI has resolved the issue.

Wireshark traces were necessary to determine root cause: Duplicate IP Addresses were noted for both IPSI's in question.

Cause

Possible Causes:

  • Network: In this case, duplicate IP addresses
  • During a migration to a new platform the original server could still be up and in-service

Solution

Possible Solutions:

  • Wireshark traces should be taken from the both the ESS and the IPSI fro review when this sort of problem is encountered.
    • In this case, IPSI 38a was conflicting with the ESS ETH0 IP Address and IPSI 38b was conflicting with the ETH3 IP Address. 
    • The trace showed a duplicate IP address and the two associated MAC addresses.  
    • Duplicate IP addresses should be resolved.
  • If the original server is still in-service perform a shutdown on it to remove it from the network to allow the new platform to take control
    • After shutting down the original server then power it off
    • Perform a reset system 4 on the new platform server

Note: Service impact will depend upon which device has an incorrect address and will be changed.

 

Review CM ECS Logs to see if there are Duplicate IP Address messages in there.

Additional Relevant Phrases

IPSI going in and out of service IPSI switching from active to standby randomly IPSI's are showing the wrong IP address for the correct Call Server

Avaya -- Proprietary. Use pursuant to the terms of your signed agreement or Avaya policy