How to Plan for a Network Cutover

A network cutover is often the culminating event for networking projects. All of the planning, staging, testing, and configuration leads to this brief yet critical change window. Though there are a variety of cutover types and activities, I’ve found that there are some fundamental principles that apply to all of them.

Discovery

For a cutover to be successful, you need to start with network discovery. The level and depth of discovery will depend on how well you already know the network, what sort of cutover it is, and how well you understand the technology.

Remember that your level of discovery is commensurate with the complexity of the cutover. For example, replacing a stack of 4 IDF switches won’t need nearly the amount of discovery effort as ripping and replacing a data center core or a pair of WAN firewalls.

During the discovery phase it’s important to collect as much information as you can so you’re not scrounging for it at the last minute, or worse, during the cutover. Collect information like VLANs, usernames and passwords, SNMP strings, where ACLs are located, routing configuration, IP address of important resources like DHCP servers, DNS servers, and so on.

That’s a fraction of the information I’d collect, but it gives you an idea of what you should do to get started. On my team, we use spreadsheets which we review with customers at the project kickoff. Each spreadsheet is focused on a particular technology and is a way for engineers to collect as much information as possible quickly and efficiently.

Planning and Testing

Once you have the info you need and have studied the topology, it’s time to plan what you’re going to actually do. During this phase, I try to build the topology in a lab environment.

For some cutovers this is unnecessary or just impossible. For example, I don’t have Cisco Firepower appliances in a lab at work or at home, so it’s just not possible for me to lab a Firepower install.

However, there are times when I’m making significant changes to routing in which case I’d create the customer’s topology as accurately as I can in GNS3. That way, I can test changes in a lab environment.

When cutting over a production network to a new wired Cisco ISE solution, I’ll spin up an ISE VM, domain controller, etc, and set up a switch with a few different types of endpoints. This allows me to test scenarios and plan out exactly what I’m going to deploy in prod.

I also like to write out what I’m going to do step-by-step. Within that workflow I include code snippets, important phone numbers, and anything else I don’t want to hunt for during the actual cutover.

This workflow varies from project to project, but generally what I like to do is create a numbered list of ordered tasks. Even if you don’t refer to it much during the cutover, writing out a workflow ahead of time forces you to think through the steps carefully and avoids winging it during the change window.

Staging

I also try to stage devices and/or configuration. This is actually pretty easy to do most of the time even with complex cutovers, and it has a couple benefits.

Staging devices and configuration means there’s less to do during the change window, and it also means network downtime can be much shorter.

For example, I’ve ripped and replaced my fair share of Cisco 6500 core switches in my day, and that’s usually not a trivial task. A fully loaded 6513 has tons of connections to distribution layer switches, firewalls, virtual machine hosts, and other network appliances like load balancers, wireless LAN controllers, and so on.

Taking the entire chassis down in order to physically disconnect everything, un-rack it (remember that they’re the size of Volkswagons), rack the new core, cable it, etc is likely an unacceptably long downtime and an outage that just has too many moving parts for my taste.

Now imagine imagine replacing a pair of 6513s in a VSS along with all the cabling as well. A cutover can be so impactful to the organization that staging gear and configs ahead of time is more than just a good idea – it’s the right way to do do it.

An example using the scenario above would be spinning up the new core switches in an adjacent rack to the prod switches. Then, I would carefully trunk the old switches to the new switches allocating plenty of bandwidth. Now I can slowly migrate all the distribution switches, VM hosts, etc over to the new switches carefully over several days or even weeks.

That way, you get all the devices moved over slowly but leave all the layer 3 configuration, routing, and ACLs on the old core until you’re ready. Then, you can cut routing over and VLANs one at a time during the change window. Running the staged environment in parallel to the production network is pretty much how I approach all my cutovers if I can.

If something like this isn’t an option, I would at least make sure that the new devices are configured, racked, and ready to go with the simple move of a cable. That way you give yourself time for troubleshoot and rollback in seconds if necessary.

For example, when I’m swapping out Cisco ASAs for a new pair of FTD appliances, I would do the best I can to make room in the rack to mount the new Firepower boxes immediately above or below the old ASAs. Then, when the time comes, all I need to do is swap cables.

Backups

Before pulling cables and hacking away at the CLI, make sure to take a fresh backup of all the devices you’re working with. Hopefully you got them during the discovery phase, but I would do it again just before the cutover.

In addition to straight ‘show run’ outputs, I’d also capture live information like routing tables, MAC address tables, VLAN database information, and that sort of thing. The specific info you need will depend on the type of cutover.

Rollback Plan

Lastly, I’d never start a cutover without a rollback plan. Sometimes the rollback plan is simple and obvious, such as moving the cable back to the old device. Sometimes, especially if the cutover is entirely configuration and not new hardwrae, the rollback plan requires a step-by-step workflow to undo the changes that were made.

—

I’ve spent most of my professional life working for VARs which means I’ve basically gone from cutover to cutover my entire career. In fact, most of the networking battle scars I have are from cutovers, so this is a topic close to my heart.

Regardless of how complex your network cutover, remember the fundamental principles of Discovery, Planning, Testing, Backups, and Rollback. And remember that none of us is so experienced we can skip the peer review.

Thanks,

Phil