Some time ago we all lived in fear of complete technical breakdown due to a little problem called the millennium bug. In simple terms it was because some software simply had not accounted for the fact that the next century (21st) would happen. Some software had only two digits to store the year in, so when the clock ticked from 1999 to 2000, these clocks went from 99 to 00, causing problems as it did so.

Many look back and say the IT world over egged the problem and in some cases I have no doubt that is true however there were also cases when the software would indeed fail and could be demonstrated to do so. This story relates to one of these real world situations.

The system itself is fairly irrelevant but testing had shown it to be impacted by the bug and fairly catastrophically. Therefore we needed a plan to sort this out. Simply put we needed to do some patching on the operating system (Solaris) and some upgrades on the software package itself. Failure to do so would result in no remote access for staff.

To me this was an eminently simple task, 10 systems around the world operating in resilient pairs, the only slightly tricky bit was the size of the software, which really meant that sending it electronically all over the world to the right locations was not practical. A new plan was needed. Whilst I pondered the right approach and tested the upgrade process, a strange thing happened, two project managers from the central Y2k team suddenly appeared to “make sure it all happened”.

Immediately the task became more complicated. As I highlighted the risks and problems that I was tackling there was a distinct nervousness growing amongst the PMs and they disappeared to do some PM type things. Over the next few days I figured out a plan to operate with zero downtime and a proposed schedule to work to, all I needed was a local contact in each physical location to put a CD into the machines ahead of migration time. Simples.

The PMs re-appeared a few days later with their plan. Odd how they had come up with a plan without talking to me, the main sysadmin. Their plan was simple, I would fly to Paris and do an upgrade on the primary system first the evening of arrival and then the secondary system the following evening, flying back that night. Next it would be a flight across to New York. Deal with the primary then to Washington to do the secondary and back to UK. Then a bit of a longer trip to Tokyo for 1 day to do both systems during the day as they were yet to be formally live, then that night fly to Sydney to land, go straight to site, do the primary then the secondary the following day before heading back.

There were a few flaws with this plan which I felt the need to point out. Firstly it was unnecessary to go to site, it could all be done remotely. Secondly the systems needed to be upgraded in a very specific order and manner or the database would basically break, trashing the whole thing. I also had a few words to say about their planning of my time and concepts such as jetlag, timezones and that it was all just so unnecessary. This all came as a bit of a surprise to the PMs so they disappeared for more PM stuff to happen.

The next regrouping occurred a week or so later, the new plan that had been hatched was more in line with the correct approach yet still had an insistence that the risk levels on this were so high that a physical presence was needed on site and that needed to be me. At this point, I was young, felt that a little travel could be fun and if someone wanted to plan it out and pay me to do that, so be it. Let the grand tour begin!

And a grand tour it indeed was, however the lack of decent planning really started to come unstuck as the task progressed but in good news, that gives me a few more epic fails!