I am in a factory, a food one and a brand new one. It has been a long build but watching a state of the art factory build is fascinating. Conveyor belts and spinning things and robots and self driving vehicles are all in action to deliver a minimal human intervention factory.

I am delivering a fairly generic set of infrastructure capability for this, including WAN, LAN, WiFi, VoIP and security.

We are a few weeks prior to final opening of the factory and it’s in a test phase, with lots of experimentation taking place, the occasional disaster as timings along lines are altered and for me it’s the highlight of that year, I have engineered myself a day on site to look around and just watch it all come to life.

It turns out I have picked a great day. I start by running a tool I wrote in my own time across the network. It picks up in far more detail than anything else I have ever used, what is actually connected. It has been a little annoyance of mine in the process actually. I like to design from the edge inwards. What’s connected to a network defines what features and capabilities that network needs. In this case the client wouldn’t delve into that but just wanted 200 ports, “Can’t you just provide 200 ports? I don’t see what it matters what is connected.” Hmm.

My tool tells me lots of standard stuff and highlights a few more interesting things. One is some connected switches we have not provided – not a problem but now I know they are there I can protect against bad designs and spanning-tree failures that would otherwise out of our control. There are also a couple of things hanging of WiFi that I just can’t get to the bottom of and it’s bugging me.

My main problem here is that I know we haven’t so much designed WiFi as put it in. We have done that in an empty factory that will shortly be filling with lots of metal cans, liquids and end product. I know this will impact coverage significantly so just want to be sure what in the factory is relying on connectivity that may disappear.

I therefore approach the customer for help and after a little trauma, we find the devices in question. They turn out to be two forklift trucks, particularly interesting ones in fact, as they have no drivers.

I have to now ask the question, “what’s the WiFi connection for?”. The answer chills me a little as it is providing the picking and dropping instructions to the fork lifts. The WiFi is not specifically resilient by design, so I have to ask the next question “what happens if they lose signal?. The customer doesn’t know so I have little choice but to find out.

Fortunately being in this test phase, we can do pretty much what we like, so whilst the factory is in full swing and the fork lifts are busy doing stuff, I turn off the WiFi and watch.

Both of the forklifts stop. That is good. I don’t want these things operating without instruction, that’s when crashes occur and people get hurt. On the downside we now have a single point of failure in the factory that shouldn’t be there. Particularly as if this were a real hardware fault, we have to get 3 stories up in the middle of the factory floor, requiring at minimum a tower lift, but possibly scaffolding looking at the equipment now deployed all over the factory floor. That’s a day or two shutdown, not very appealing.

Then things go bad. One of the forklifts sets off doing stuff. Strange but I figure out pretty quick it has reattached to another WiFi access point. After a while it heads off to do something going directly towards the other forklift. In fact directly into the other forklift. Then having knocked it flying into a warehouse shelf stops as if it were a pet having just destroyed a prize ornament but pretending otherwise.

That wasn’t the only issue. There were some portable phones no-one mentioned that a grumpy person came shouting over about. One of the robotic arms stopped too. Essentially the whole resilient factory was coming to a standstill because an £800 device had ‘failed’.