ERS5530 stack and linux bonding for high throughput

[im12]

Hi

We've got 2 5530s in a stack with 10 dell r610 running debian squeeze with xen connected to them via 4x 1gig links each. 2 links in each switch of the stack.

sysDescr: Ethernet Routing Switch 5530-24TFD
HW:35 FW:6.0.0.6 SW:v6.1.1.017

Initially this was setup with bonding mode 0 ( round robin distribution between the 4 nics ). The throughput was fine, we could get speeds both tx/rx of just above 2gbit with nfs3. Meaning there is a single ip in both ends with just one connection.

However, allthough the throughput was good the stability was not. We'd loose connectivity to the nas randomly from either 1 server or all of them. As expected (I suppose ) you'd see the mac address of the bond interface jumping between all 4 connected ports when doing show mac-address-table. My initial thought was "thats not good" and when we shut 3 of the 4 connected switchport the stability issues disappeared.

So the question is, how do we set this up for best possible performance. From what I can understand, both MLT and LACP configs will hash based on mac or ip and in both cases that will result in only 1 nic getting used.

I really hope that someone on here can point us in a useful direction in regards to what needs to be done on the switches to support something like the bond mode 0 on the server side.

Regards and thanks in advance

[im12]

bond config :
auto bond1
iface bond1 inet static
address 10.0.20.23
netmask 255.255.255.0
network 10.0.20.0
slaves eth2 eth3 eth4 eth5
bond_mode balance-rr
bond_arp_ip_target 10.0.20.12
bond_arp_interval 2000
mtu 8122

[im12]

With MLT:
As the bond mode is balance-rr it seems we can utilize more than 1 link on outbound traffic:
root@x3:~# dd if=/dev/zero bs=8192 of=/mnt/xen_convirt/tmp/test3.img count=2500000
2500000+0 records in
2500000+0 records out
20480000000 bytes (20 GB) copied, 110.856 s, 185 MB/s

Inbound will still be limited by the hashing employed by the MLT, and I'm not even able to max out 1 link:
root@x3:~# dd if=/mnt/xen_convirt/tmp/test.img bs=8192 > /dev/null
1250000+0 records in
1250000+0 records out
10240000000 bytes (10 GB) copied, 185.452 s, 55.2 MB/s

With no LAG:
With "wrong" switch config, these are the results we get:
outbound:
root@x3:~# dd if=/dev/zero bs=8192 of=/mnt/xen_convirt/tmp/test3.img count=2500000
2500000+0 records in
2500000+0 records out
20480000000 bytes (20 GB) copied, 114.935 s, 178 MB/s

inbound:
root@x3:~# dd if=/mnt/xen_convirt/tmp/test.img bs=8192 > /dev/null
1250000+0 records in
1250000+0 records out
10240000000 bytes (10 GB) copied, 44.8598 s, 228 MB/s


So I'm not really looking to implement LACP or MLT with the restrictions you mention. If I can get around those restrictions somehow, then fine

But is there nothing I can do to make the switch behave like the balance-rr bonding mode? As you can see the throughput is sooo much better. But the catch is that the bond mac addr is jumping between the 4 switch ports and its causing (seemingly) some harm to the switch.

I'd really like to figure out what exactly is happening to the switch in this setup that is causing the timeouts. And maybe we can mitigate that somehow and just continue using balance-rr ?