free bootstrap template



How to… Cut your own legs off

It's just a click away...

In band management of devices is a great way of keeping costs down for a service management organisation. It essentially means that the customer’s own network is used to transport the management traffic. It’s no biggie for the customer, the amount of traffic is tiny compared to what they are transferring and this is very common practice.

There is however a problem and that is that humans are involved. Humans do daft things, sometimes in the most well-meaning of ways but that doesn’t mean that there is not a customer impact.

This circumstance made me laugh out loud, I probably shouldn’t have done but what can you do when you hear someone fail really badly. We are after all English and love to cheer when crockery smashes or glasses are dropped.

This is a small site connected to a large network via an SHDS circuit. There is no router on site, the connectivity is Ethernet based and allows the remote site switch to connect on one of it’s ports and the remote core switch appears to be right next door, regardless of distance (within limitations).

Port 1 in this switch is the circuit. My colleague is remotely looking at it in preparation for some work later that day. He moves around the interface, as any engineer would do, grabbing information that he needs. To do this he uses a specific command “show”. He is also a human and lazy, so he abbreviates this command to “sh”. Nothing to go wrong there really.

Except he does something that I never really got a good explanation for. He moves to configuration mode and into the interface of port 1 – this basically means he can now issue configuration commands. Now he is human, therefore fallible, and he forgets he is in this configuration mode and wants to know something, so he types “sh” and presses enter to look for the command he needs.

In configuration mode show doesn’t exist. “sh” does exist though, as a shortcut for shutdown. Thus the device did as it was told and shut down the interface to which the circuit was connected through which he was also connected and his session is lost, the site is now down and the resultant cursing is near unprintable.


Learning point for everyone:

Humans can be idiots, protecting against and predicting the complete landscape of idiocy is not really possible. There are basics though that can always be implemented – in this case I wonder if actually the root cause is poor user interface design by the manufacturer, or is it that there was no change control mechanism that escalated the rights of that engineer when, and only when, a change was approved?

We all take risk and indeed should do. Calculated risk based upon our own capability and the capability of others. That engineer is someone that I still trust to this day because he learnt. That is not a mistake he will ever make again.


Onwards to next epic fail

Business details

Registered company no. 11869849 
VAT number 317720513