free portfolio website templates



How to... Take down an entire business

Default routes are helpful.

I am at this point going to assume that you have an idea about IP and how routing works, at a high level, low level or somewhere in-between. If not, search the Internet and come back in a few hours.

So here we are, in the comms room. The aforementioned remote access system has a priority 1 fault and I have been called out to fix it. Looking remotely I was flummoxed. You need to put in your mind that in these days there was no remote console for Windows, my visibility was limited to what I could TELNET to, yet my personal familiarity was physical hands on Windows, the Atari 8/16bit computing set, the Sega Master system and the PlayStation (1). Handy.

I could probably, with some thought, have connected remotely to the Cisco terminal server which I knew little about and the DEC VAX, which may as well have been a food processor to me.

Instead, what I did was go to work and go to the comms room – there’s a P1 and I am paid to sort it, sort it I will! I am forever the hero engineer! More on this later.

So I poke around the RAS server, it all seems fine. The modems have had a bad evening with a lot of lock-ups, so a bit of rebooting there clears a few alarms but the report says no access out of the local building, why on earth would this be? So I start to teach myself IP routing on the fly during a fault condition. What could go wrong?

The server points to the default gateway, ok well that presumably means all traffic it doesn’t know where to send will go here. That must be all out of building traffic. Well that’s easy. Let’s see what that device is. Oh it’s the Cisco terminal server, I wonder what that understands?

At this point, I remind you that I knew nothing about Cisco devices, or IP, or routing. In fact I was great at buying sandwiches and making tea but we all start somewhere.

So I get a console on the router/terminal server and notice it has no default gateway, which seems odd as it won’t be able to route traffic out of the building. How strange. Nothing useful in this routing table thingy that I have found either. Now I know the default gateway for the building is x.x.x.1 so why don’t I just force the issue? A bit of poking around the interface and it’s pretty easy. And I do. Nice simple static default route addition. And it works. Everything is hunky dory the world is a wonderful place.

However. What I didn’t know was that there was a P1 running on the building connectivity as a whole. This was resolved about the same time as my “fixes”. On restoration, not only did this new default route exist in my poor little 2924 but I also advertised an incredibly low cost default route to the building, to the locality and actually to the whole business. This was a big international business. I had actually sucked all unknown traffic into my little 2924. It hurt. A lot.


Learning point for everyone:

Do not make changes to anything. It worked before so something else has changed. Find what changed, do not change anything in your world. Put your hands in your pockets, step away from the keyboard. Do not do anything. Watch, listen and if necessary coordinate. By all means verify normality but be clear on what normality is. Document this for others – these are the standard processes running in normal operation, this is the routing table, this is the performance characteristics under normal operation and much more. Dull but invaluable for day to day operations to succeed, not fail. Quality starts with understanding the normality. Fault fixing is finding out what you don’t know.

Minimal access rights for everyone. Prove greater rights are needed and the people know how to use them responsibly. No changes permitted. At all. Ever. Without change control. That will keep your business running as it does today.

It won’t however allow for fast-paced digital transformation of an existing business, there are other approaches needed for that.

Business details

Registered company no. 11869849 
VAT number 317720513