Brocade DCX webtools authentication problem
datestamp, [FW-1342], 23858, SLOT 6 | FID id, WARNING, vFabricName, Sec Login Violation, is above high boundary(High=2, Low=1). Current value is 6 Violation(s)/minute. datestamp, [SEC-1193], 23859, SLOT 6 | FID id, INFO, vFabricName, Security violation: Login failure attempt via HTTP. IP Addr: violating ipaddress datestamp, [SEC-1193], 23860, SLOT 6 | FID id, INFO, vFabricName, Security violation: Login failure attempt via HTTP. IP Addr: violating ipaddress datestamp, [SEC-1193], 23861, SLOT 6 | FID id, INFO, vFabricName, Security violation: Login failure attempt via HTTP. IP Addr: violating ipaddress datestamp, [SEC-1193], 23862, SLOT 6 | FID id, INFO, vFabricName, Security violation: Login failure attempt via HTTP. IP Addr: violating ipaddress
The violating ipaddress was the address of the DCFM server. So I fired up a browser from my workstation and connected to webtools. Webtools showed up fine with the authentication screen:
So we use the user which is defined for DCFM, and it starts authenticating:
And after a second or two we get an invalid password error:
This happens starting Webtools from DCFM server, from workstations, from every vlan, and with every user we tried. We’re not using RADIUS authentication, this is normal local switch authentication. All of the tried users are working if you use them logging in through SSH. My guess at that time was that the authentication between the http server on the directors and the local switch database was broken, due to a bug. Contacted a Brocade engineer directly, he made some calls but no one had ever seen this strange behaviour. Logged a call with IBM support (directors are OEM’d by IBM) and then the hassle of logs and dumps sending and answering your standard L1 questions all came by. They purely focussed on DCFM losing it’s password in the discovery setup screen. To me it was obvious why it was gone there, DCFM was told it was an invalid user, so it clears the field. IBM L1 support however was persisting this was where the problem was. After persuading them to dispatch the call to Brocade, things sped up a bit.
First bullet on the action plan was to upgrade our Java plugin. For FOS 6.3.0 the plugin should be at least at 1.6.0.13 or later. Of course that didn’t work because it already was at 1.6.0.13. After telling them the complete story, apparently things got dropped in the conversation between IBM and Brocade, they came up with a HA failover. Effectively rebooting the CTP’s.
So we did:
SwitchName:vFabricName:username> hashow Local CP (Slot 6, CP0): Active, Warm Recovered Remote CP (Slot 7, CP1): Standby, Healthy HA enabled, Heartbeat Up, HA State synchronized SwitchName:vFabricName:username> hafailover Local CP (Slot 6, CP0): Active, Warm Recovered Remote CP (Slot 7, CP1): Standby, Healthy HA enabled, Heartbeat Up, HA State synchronized Warning: This command is being run on a redundant control processor(CP) system, and this operation will cause the active CP to reset. Therefore all existing telnet sessions are required to be restarted. Are you sure you want to fail over to the standby CP [y/n]? y Forcing Failover ... SwitchName:vFabricName:username> hashow Local CP (Slot 7, CP1): Active, Warm Recovered Remote CP (Slot 6, CP0): Non-Redundant SwitchName:vFabricName:username> hashow Local CP (Slot 7, CP1): Active, Warm Recovered Remote CP (Slot 6, CP0): Standby, Healthy HA enabled, Heartbeat Up, HA State not in sync SwitchName:vFabricName:username> hashow Local CP (Slot 7, CP1): Active, Warm Recovered Remote CP (Slot 6, CP0): Standby, Healthy HA enabled, Heartbeat Up, HA State synchronized SwitchName:vFabricName:username>
After this we tried Webtools, and it worked. DCFM picked it up immediately, without any changes it discovered the failed over DCX. Interesting to see however was the fact in discovery setup the password was still blanked for this DCX, although it was re-discovered automatically. Just filled the password in there again, and it accepted it. Field is now filled.
Problem solved. Although there’s no answer from Brocade yet explaining why this happened. Expecting a note in upcoming releasenotes somewhere
FIY:
This happened on:
Brocade DCX running FOS code 6.3.0b
DCFM 10.3.3 build 11



