How many times has one of your network projects come to a screeching halt (probably at 2am) because you didn’t have the right power plug or patch connector? Seems like such a trivial thing, but millions of dollars of equipment won’t do much more than look pretty in the racks until it’s all powered up and connected together.
Almost two years ago I was the lead network engineer on a rip and replace project swapping out some Catalyst 6509s for some 6880s, adding in some 4500Xs, replacing old PIXes with a pair of new ASAs, and setting up some Nexus 5K pairs with a line of top-of-rack 2Ks. After the core cutover there would be a big LAN switch refresh, a Prime/ACS upgrade, and a migration to a new UCS B series. It was a several million dollar project.
I worked with the storage engineer to get some things straightened out with migrating VMs. I worked out the schedule with my project manager and the customer. I even ran the plan by the most senior network engineer at my company. Because of some configuration requirements there was no good way to run the old and new core/edge in parallel, so it was going to be a hot cutover. We discussed spanning-tree. We discussed EIGRP convergence. We discussed VMotion (well the storage guy did), and we discussed rollback strategies. Everyone liked the plan just fine, and we moved forward.
The cutover was scheduled for 11pm Friday night (naturally). I started the pre-work at 7pm and touched up my scripts. I made sure the boxes labeled “power cables” and “fiber patch cables” were nearby. I didn’t open them up though.
I think you know where I’m going with this.
I got the initial configs on the gear before it was delivered, and now it was all racked and about half of it booted up. It was maybe 7:45 when I started opening boxes looking for the Twinax cables. Found ’em. Grabbed some aqua patches with LC connectors and started getting the 5548s cabled up.
The customer had the campus core in a room across the building, so now it was time to home these 5548s from this room back to the core using the fiber patch panel….but wait….are those ST connectors? I thought? Weren’t they? …..oh no……
I had all LC-LC. The patch panel was ST.
Yes, it’s initially the solutions architect’s and PM’s responsibility to have that figured out ahead of time, but the project was handed off to me long before the cutover. I had no one to blame. Hundreds of thousands of dollars of equipment sat disconnected just prior to an overnight cutover because I had the wrong connectors. Lovely. The customer wasn’t thrilled but at least we could get the core switches and firewalls cutover and leave the Nexuses (Nexi?) out until the next change window. I grabbed the power cables box and headed across the building to the network MDF. The core switches were racked and the firewalls in place. They were still unplugged so I tore open the box I was holding.
My heart sank. All C13 plugs. PDUs were all C14, of course.
That night we got little accomplished. Thankfully getting the plugs and connectors was no big deal, and the actual cutover (rescheduled for a week later) went very well. After taking the customer out to lunch, all was just fine. I learned a hard lesson. I planned out a clean cutover and expertly configured some advanced networking technologies, but in my haste to be the hero of the day I left out some of the most basic considerations like power and layer 1 connectivity. Today I typically start with those areas and take as many pictures as I can.
I’ve heard it said that an expert is someone who’s made all the possible mistakes that can be made in a particular field. Well, scratch that mistake off the to-do list.