ERS5520: Incrementing Packet Indiscard at switch interface connected to IBM server and skewed behavior of packet counters.

Doc ID    SOLN235570
Version:    2.0
Status:    Published
Published date:    27 Dec 2017
Created Date:    27 Aug 2013
Author:   
sharma68
 

Details

ERS5520: Customer reported that they are seeing packet drops on switch interfaces which are connected to an IBM Server. Upgrade to the latest software load did not resolved the issue.

Customer reported that the Indiscard counter is incrementing continuously while the interfaces on the switch were configured not to drop any packet what-so-ever.

Problem Clarification

Packet captures were taken from the switch interface which was connected to the server, and after analyzing the Wireshark Outputs that were captured during the remote session and in the RX Only PCAPs we have found some packets that were ‘malformed’ which will cause the in-discards counter to increment. These packets are being sourced by the server which is plugged into port 2/12 mac-address 34:40:b5:a5:53:ca. The packets have both the source and destination address set to the same address, 34:40:b5:a5:53:ca. Since this mac address is learned on this port, any packets with a destination of this mac address that arrives on the same port will be discarded by the switch and hence the counter will increment.

It was tested and verified in lab setup, the fact that the packets are addressed to itself (source and destination mac addresses are the same) is the reason the counter is incrementing. When the destination address in one packet was modified and played back for 4000 times, the counter did not increment which proves that these malformed packets are being dropped at the switch interface causing the counter to increment. Also, It was found that these packets hits the switch in batch of three (we noticed such packets with three consecutive frame numbers in the PCAP) and that explains why the counter was incrementing by three during the remote session.

Cause

Malformed packets from the IBM server was causing the Indiscard Counter to increment at the switch interface.

Further, it was also established that in a lab test that the behavior of Indiscard Counter is skewed when the port is mirrored on that switch, it may or may not increment at that time. This explains why we have to wait for about 27 minutes to see the first packet drop during the maintenance window. More information about the test is as follows:

Test scenario: 4000 Malformed Packets with same Source and Destination MACs were pumped into the switch, while the port was mirrored. During this test, the Indiscard counter did not incremented. This process was repeated two times with the same result. When the Port mirroring was removed and similar 4000 packets were pumped intot he switch, the Indiscard counter incremented by 4000, one for each malformed packet injected.

Result: It was established from this test that, the Indiscrad counter performs unexpectedly when the port is mirrored. When, the port mirroring is removed the Indiscard Counter works as expected.

Solution

Customer shall check with the IBM/Server team so as why such malformed packets are being sent by the server.


Avaya -- Proprietary. Use pursuant to the terms of your signed agreement or Avaya policy