Cisco Nexus Switches Part 2: Basic Configuration

This week’s post will cover basic information gathering and configuration of Cisco Nexus switches. I’ll be using the 5500 series as my example and covering the basics without getting into features such as fibre channel, VSANs and that sort of thing.

But before hacking away at the CLI, the first step should be to gather all the relevant information needed for a successful deployment. The goal is to use whatever awesome new gear you’re installing to meet some technical and business need, so getting it right the first time is paramount. Set up a spreadsheet, set something up in Google Docs, start an Evernote account, or just use a checklist with blanks to fill out. Really, the key is to gather the information you need early on so you can configure and deploy the equipment correctly, efficiently, and professionally.

When deploying Cisco Nexus switches, you’ll need some specific information about the network already in place, and you’ll need to make some configuration decisions ahead of time. You should spend some time running the hardware, recording serial numbers and meeting with your team or with the customer. Here is just an overview of some things to do and consider.

1. Start off by unboxing the new gear and powering everything up. Let the new switches run for a few days just so you know you don’t have any DOA devices to RMA. I try to do this whether the switches are for internal use or for a customer. If you’re on site with a customer, you may not be able to do run them at all before racking them, but the key is letting them run for a while at least before putting them into production.
2. Make a list of the following preliminary information:
   • Hostname for each device
   • Management IP addresses, subnet mask and default gateway
   • Local user accounts
   • Features to enable such as vPC, FCoE, DHCP, FEX, VTP, LACP, etc
   • Role of switch (end-of-row, top-of-rack, core)
   • All VLANs needed on the Nexus switches
   • Rack location, type of cage nuts to use
   • vPC number(s) (just a unique identifier you’ll need to set up vPCs later on)
   • Uplink trunk ports to data center/LAN core
   • DHCP relay information
   • Any VLAN interfaces that will used on the switches for your design
   • A list of all the devices that will connect to the Nexus switches
3. Check that you have the correct power cables for the PDUs, correct SFPs (1/10 Gbps ethernet, 8 Gbps fibre channel) and appropriate storage connectivity.
4. Identify the hot and cold aisles and plan to install the switches accordingly. Default airflow on the 5500 series is front-to-back, for example, the back being where all the ports are located. Airflow on the switches can be ordered in either direction, so this is an important thing to check.

I like to gather the Nexus specific information before getting into mounting hardware or configuring anything at all. In my experience, sitting down with the team or with the customer before doing anything whatsoever is the best way to ensure a smooth project. Below is a simplified version of a spreadsheet I’ve used to gather relevant information. It’s a variation of something I used when working for a Cisco partner a few years ago and should be part of a larger spreadsheet in which you should capture DNS and RADIUS server addresses, SmartNet contract numbers, serial numbers, asset tag information, rack and data center location, and all that sort of thing. You can download the spreadsheet here.

 

 

Now let’s get into the initial configuration wizard.

1. Power up the new Nexus switch and connect to the console port using a serial cable. The switch will take several minutes to boot.
2. The initial configuration wizard starts automatically. Use the information you worked out with your team or with the customer to complete the wizard. These settings can be changed later. The Nexus 7000 series initial configuration is almost the same, but it will prompt you for additional information about the default virtual device context.

Next we’ll upgrade the firmware.

Upgrading the firmware requires a reboot, so make sure to do this before moving forward with any significant configuration and of course before putting the switch into production. There are several methods for moving files around, but I prefer using a USB stick because it’s fast, straightforward, and reliable.

1. Download the latest recommended firmware code version for your specific switch from Cisco’s download page website (you’ll need to log in) and save it to your USB stick.
2. Insert the USB stick into the USB port of the switch and run the following commands:

 

After the firmware is upgraded, we can start the configuration.

I prefer to start with the more basic elements of the configuration and move to the more complex. Every deployment is different, so keep in mind that there is more than one way to accomplish some of these tasks. Typically Nexus 5500 series switches are configured in pairs, so make sure to repeat the below configuration on the second switch. Also, make sure you’re saving your config periodically as you go.

 

1. Now configure basic Spanning Tree.

NEXUS5K-A#config term

NEXUS5K-A(config)#spanning-tree  port type network default

NEXUS5K-A(config)#spanning-tree  port type edge bpduguard default

 

2. Now enable all the features you’ll need for this implementation. Below is just an example of common features. It’s typically best practice not to enable features you don’t need.

NEXUS5K-A(config)#feature lacp

NEXUS5K-A(config)#feature fex

NEXUS5K-A(config)#feature interface-vlan

NEXUS5K-A(config)#feature vpc

NEXUS5K-A(config)#feature lldp

 

3. Typical IP storage traffic requires the switch to accommodate jumbo frames, but by default the switch is configured to process 1500 byte ethernet frames. Configure a QoS policy to accommodate 9000 byte ethernet frames.

NEXUS5K-A(config)#policy-map  type network-qos jumbo

NEXUS5K-A(config-pmap)#class  type network-qos class-default

NEXUS5K-A(config-pmap-nq-c)#mtu  9216system  qos

NEXUS5K-A(config-pmap-nq-c)#system  qos

NEXUS5K-A(config-sys-qos)#service-policy type network-qos jumbo

NEXUS5K-A(config-sys-qos)#end

 

4. Next configure the VLANs needed for this deployment. In a large network with a lot of VLANs I’ve used VTP in client mode to quickly get all the VLANs onto the switch, but generally I don’t recommend doing that. If you choose to use VTP, you’ll need to enable the feature and make sure you configure VTP in client mode. Afterward you can disable the protocol and the feature.

NEXUS5K-A#config term

NEXUS5K-A(config)#vlan 10

NEXUS5K-A(config-vlan)#name iSCSI

NEXUS5K-A(config)#vlan 20

NEXUS5K-A(config-vlan)#name vMOTION

NEXUS5K-A(config)#vlan 30

NEXUS5K-A(config-vlan)#name VM_MANAGEMENT

NEXUS5K-A(config)#vlan 40

NEXUS5K-A(config-vlan)#name NFS

NEXUS5K-A(config-if)#exit

 

5.  Now configure the virtual port channel (vPC). Configuring a vPC requires a peer link, vPC domain ID, and the appropriate interface configuration. The example below has two 10 Gbps ports in a port channel, though I typically configure four ports if I know they will be available. The channel-group mode must be active in order to utilize LACP.

NEXUS5K-A(config)#vpc domain 10

NEXUS5K-A(config-vpc)#peer-keepalive destination [IP address of switch B] source [IP address of switch A]

NEXUS5K-A(config-vpc)#interface  e1/5-6

NEXUS5K-A(config-if)#channel-group  10 mode active

NEXUS5K-A(config-if)#interface  po 10

NEXUS5K-A(config-if)#description  vpc peer link

NEXUS5K-A(config-if)#switchport  mode trunk

NEXUS5K-A(config-if)#switchport  trunk allowed vlan 1, vlan 10, vlan 20, [include additional necessary vlans]

NEXUS5K-A(config-if)#spanning-tree  port type network

NEXUS5K-A(config-if)#vpc peer link

NEXUS5K-A(config-if)#no shut

NEXUS5K-A(config-if)#exit

 

The NX-OS operating system chooses the primary and secondary switch priorities automatically, but the role priority command can be used to manually configure which is which. The lower priority value sets the switch as primary. You can also add the delay restore [time in seconds] command to manually control how long it takes before the vPC comes back up on the peer switch after a reload. There are a variety of other commands you can use to control more precisely the behavior of the vPC, but for this exercise I’ve kept the configuration simple.

 

6. Configure the uplink trunk ports to the core switch. The upstream switch will likely be the data center core (Nexus 7009/7010) or the LAN core. The config below is for a Nexus 7k upstream switch.

NEXUS5K-A(config)#interface e1/1-2

NEXUS5K-A(config-if)#description  TRUNK_TO_CORE

NEXUS5K-A(config-if)#switchport

NEXUS5K-A(config-if)#switchport  mode trunk

NEXUS5K-A(config-if)#spanning-tree  port type network

NEXUS5K-A(config-if)#end

 

Notice above that in order to configure a range of ports on a Nexus switch it isn’t necessary to use the interface range command you may be used to from configuring Catalyst switches. Also note the interface command spanning-tree port type network. This is extremely important to use on interfaces connecting to other Nexus switches. When connecting to an IP storage controller use the interface command spanning-tree port type edge trunk. This command is used when connecting to end hosts that carry multiple VLANs. When connecting to non-Nexus switches such as a Catalyst 6500 series switch use the spanning-tree port type normal command. If you have redundant core switches, you should use a vPC for the uplink(s).

 

7. Configure the access ports.

NEXUS5K-A#config t

NEXUS5K-A(config)#interface e1/15

NEXUS5K-A(config-if)#description  UCS-FI-A Port e1/15

NEXUS5K-A(config-if)#switchport

NEXUS5K-A(config-if)#switchport  mode access

NEXUS5K-A(config-if)#switchport  access vlan 200

NEXUS5K-A(config-if)#end

 

8. Configure the fabric extenders. Each FEX will have a unique identifier which will also end up being the prefix on the interface number. In the example below, the first FEX is assigned the identifier 101, so the interfaces will appear as 101/1/1. A new vPC also needs to be created for each FEX which means each Nexus 5548/5596 will have two additional vPCs configured: one for each FEX. The example below is for one. Use a port channel to each FEX so you have link redundancy as well as switch redundancy.

NEXUS5K-A#conf t

NEXUS5K-A(config)#interface e1/10-11

NEXUS5K-A(config-if)#switchport  mode fex-fabric

NEXUS5K-A(config-if)#fex associate 101

NEXUS5K-A(config-if)#channel-group  101

NEXUS5K-A(config-if)#no shutdown

NEXUS5K-A(config-if)#interface  po 101

NEXUS5K-A(config-if)#switchport  mode fex-fabric

NEXUS5K-A(config-if)#fex associate 101

NEXUS5K-A(config-if)#vpc 101

NEXUS5K-A(config-if)#description  DUAL_HOMED_NX2248

NEXUS5K-A(config-if)#end

NEXUS5K-A#copy run start

 

 

You’ll still need to fine tune your configuration including configuring your vty lines, SNMP, VRFs, RADIUS servers, and whatever features and optimizations you prefer to use. You may also want to employ a function called configuration synchronization (config-sync). Also, I don’t typically like to route on Nexus 5k switches so that they can focus on doing what they do best: switching frames super fast at layer 2. You can take a look at a basic configuration used in production here.

This basic overview of configuring Cisco Nexus switches is meant to get you started. There are a variety of additional features and nuances that you can look into on Cisco’s website and all over the internet. For now, stay tuned for my next posts on Nexus switching, network engineering and professional development.