1. Manuals
  2. Brands
  3. Cisco Manuals
  4. Software
  5. Nexus 7000 Series
  6. Design manual

Cisco Nexus 7000 Design Manual

On flexpod with cisco nexus 7000 using fcoe
Hide thumbs Also See for Cisco Nexus 7000:

Quick Links

See also: Reference Manual, Configuration Manual
FlexPod Data Center with VMware vSphere 5. 1 and
Cisco Nexus 7000 Design Guide
Last Updated: November 22, 2013
Building Architectures to Solve Business Problems
loading

Related Manuals for Cisco Cisco Nexus 7000

Summary of Contents for Cisco Cisco Nexus 7000

  • Page 1 FlexPod Data Center with VMware vSphere 5. 1 and Cisco Nexus 7000 Design Guide Last Updated: November 22, 2013 Building Architectures to Solve Business Problems...
  • Page 2 Cisco Validated Design...
  • Page 3 Haseeb Niazi, Technical Marketing Engineer, Server Access Virtualization Business Unit, Cisco Systems Haseeb has over 13 years of experience at Cisco dealing in Data Center, Security, and WAN Optimization related technologies. As a member of various solution teams and advanced services, Haseeb has helped many enterprise and service provider customers evaluate and deploy a wide range of Cisco solutions.
  • Page 4 Chris O'Brien, Technical Marketing Manager, Server Access Virtualization Business Unit, Cisco Systems Chris O'Brien is currently focused on developing infrastructure best practices and solutions that are designed, tested, and documented to facilitate and improve customer deployments. Previously, O'Brien was an application developer and has worked in the IT industry for more than 15 years.
  • Page 5 Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries. All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other com- pany. (0809R) ©...
  • Page 6 About Cisco Validated Design (CVD) Program...
  • Page 7 This document describes the Cisco and NetApp® FlexPod® solution, which is a validated approach for deploying Cisco and NetApp technologies as a shared cloud infrastructure.
  • Page 8 Cisco and NetApp have developed a solution called VMware vSphere® on FlexPod. FlexPod® is a predesigned, best practice data center architecture that is built on the Cisco UCS, the Cisco Nexus® family of switches, and NetApp Fabric Attached storage (FAS) or V-Series systems.
  • Page 9 FlexPod Program Benefits Cisco and NetApp have thoroughly validated and verified the FlexPod solution architecture and its many use cases while creating a portfolio of detailed documentation, information, and references to assist customers in transforming their data centers to this shared infrastructure model. This portfolio includes,...
  • Page 10 LANs (VLANs), Quality of Service (QoS), and the Cisco Nexus 1000v virtual distributed switch. This capability enables the full utilization of FlexPod while maintaining consistent application and security policy enforcement across the stack even with workload mobility.

  • Page 11: System Overview

    NetApp Fabric-Attached Storage (FAS) systems • These components are connected and configured according to best practices of both Cisco and NetApp and provide the ideal platform for running a variety of enterprise workloads with confidence. FlexPod can scale up for greater performance and capacity (adding compute, network, or storage resources individually as needed), or it can scale out for environments that need multiple consistent deployments (rolling out additional FlexPod stacks).
  • Page 12 In addition, the Cisco Nexus 7000 Series Switches feature virtual PortChannel (vPC) capabilities. The vPC allows links that are physically connected to two different Cisco Nexus 7000 series devices to appear as a single “logical” port channel to a third device, essentially offering device fault tolerance.
  • Page 13 This dedicated uplink design leverages FCoE-capable NetApp FAS controllers. From a storage traffic perspective, both standard LACP and Cisco vPC link aggregation technologies play an important role in FlexPod distinct uplink design.
  • Page 14 ONTAP operating in 7-Mode. As shown in Figure 4, the Cisco UCS and Cisco Nexus components do not require any modifications. These layers of the stack are essentially unaware of the storage controllers’ mode of operation. The differences occur within the NetApp domain of FlexPod configuration.
  • Page 15 FlexPod Design Details Figure 4 FlexPod Model Comparison It is a fundamental design decision to leverage clustered Data ONTAP or 7-Mode, because these cannot be run simultaneously on the same controller, and the choice will influence hardware requirements, the logical construction of FlexPod stack, and ultimately the operational practices of the enterprise. Organizations having the following requirements should consider adopting clustered Data ONTAP: Large to midsize enterprises that are seeking scalable, shared IT solutions for nondisruptive •...
  • Page 16 Managed as a single system whether it has one server or 160 servers with thousands of virtual machines, the Cisco UCS decouples scale from complexity. The Cisco UCS accelerates the delivery of new services simply, reliably, and securely through end-to-end provisioning and migration support for both virtualized and non-virtualized systems.
  • Page 17 It offers aggregation, high density, and end-of-row and top-of-rack server connectivity. For campus core deployments, it provides a scalable, highly resilient, high-performance solution. The Cisco Nexus 7000 Series platform runs on Cisco NX-OS Software. It was specifically designed for the most mission-critical deployments in the data center and on campus.
  • Page 18 Manager, via the Cisco Nexus 2232 Fabric Extender platform, the CIMC does not manage the server anymore. Instead it is managed with the Cisco UCS Manager software. The server is managed using the Cisco UCS Manager GUI or Cisco UCS Manager CLI. The Cisco Nexus 2232 Fabric Extender provides data and control traffic support for the integrated C-Series servers.
  • Page 19 VM. The Cisco VM-FEX unifies the virtual and physical networking infrastructure by allowing a switch ASIC to perform switching in hardware, not on a software based virtual switch. The Cisco VM-FEX is off loads the ESXi hypervisor, which may improve the performance of any hosted VM applications.
  • Page 20 FlexPod Design Details If storage requirements change over time, the NetApp storage offers the flexibility to change quickly, as needed and without expensive and disruptive “forklift” upgrades. For example, a LUN can be changed from FC access to iSCSI access without moving or copying the data. Only a simple dismount of the FC LUN and a mount of the same LUN using iSCSI would be required.
  • Page 21 FlexPod Design Details Data ONTAP operating in 7-Mode is deployed on an HA pair of controllers that is discrete from any other storage systems in the environment and is managed as such. For this reason, the scalability with clustered Data ONTAP is superior to that of 7-Mode, which is further discussed in “Clustered Data ONTAP”...

  • Page 22: Vmware Vsphere

    VMware vSphere ESXi hypervisor providing efficient memory, storage and compute abstraction for virtual machines • Network – VMware vSphere supports third party virtual distributed switches such as the Cisco Nexus 1000v providing a resilient and fully integrated virtualized network access layer. Storage • –...
  • Page 23 Domain and Element Management This section of the document provides general descriptions of the domain and element managers used during the validation effort. The following managers were used: Cisco UCS Manager • NetApp OnCommand® • VMware vSphere 5.1 on FlexPod with Nexus 7000 Using FCoE...
  • Page 24 GUI, a command-line interface (CLI), or an XML API. The Cisco UCS Manager resides on a pair of Cisco UCS 6200 Series Fabric Interconnects using a clustered, active-standby configuration for high availability. The software gives administrators a single interface for performing server provisioning, device discovery, inventory, configuration, diagnostics, monitoring, fault detection, auditing, and statistics collection.
  • Page 25 FlexPod Design Details Figure 6 NetApp OnCommand System Manager Example VMware vCenter Server VMware vCenter Server is the simplest and most efficient way to manage VMware vSphere, irrespective of the number of VMs you have. It provides unified management of all hosts and VMs from a single console and aggregates performance monitoring of clusters, hosts, and VMs.
  • Page 26 VMware vSphere Web Client enables you to connect to a VMware vCenter Server system to manage a Cisco Nexus 1000V through a browser. The vCenter plug-in is installed as a new tab in the Cisco Nexus 1000V as part of the user interface in vSphere Web Client.
  • Page 27 It is important to note that Cisco, NetApp, and VMware have interoperability matrixes that should be referenced to determine support for any specific implementation of FlexPod. For more information on Interoperability Matrix, see the following links: •...
  • Page 28 Device Image Comments Compute Cisco UCS Fabric 2.1(1e) Embedded management Interconnects 6200 Series Cisco UCS B-200 M2 2.1(1e) Software bundle release Cisco UCS B-200 M3 2.1(1e) Software bundle release Cisco UCS C-220 M2 2.1(1e) Software bundle release Cisco UCS C-220 M3 2.1(1e)
  • Page 29 The design is physically redundant across the stack, addressing Layer 1 high-availability requirements, but there are additional Cisco and NetApp technologies and features that make for an even more effective solution. This section of the document discusses the logical configuration validated for FlexPod.
  • Page 30 FlexPod Implementation and Design Figure 9 FlexPod Distinct Uplink Design with Clustered Data ONTAP VMware vSphere 5.1 on FlexPod with Nexus 7000 Using FCoE...
  • Page 31 Figure 10 the components of the Cisco UCS system offer physical redundancy and a set of logical structures to deliver a very resilient FlexPod compute domain. In this validation effort, multiple UCS B-Series servers’ service profiles are SAN booted over FCoE as VMware ESXi nodes. The ESXi nodes consisted of Cisco UCS B200-M3 Series Blades with Cisco UCS 1240 VIC adapters.
  • Page 32 FlexPod continues this practice. One key design decision in the Cisco UCS domain is the selection of I/O components. There are numerous combinations of I/O adapter, fabric extenders and fabric interconnects available; so, it is important to understand the impact of these selections on the overall flexibility, scalability and resiliency of the fabric.
  • Page 33 Mezzanine Slot card • The Cisco UCS 2208XP Series Fabric Extenders, installed in each blade chassis, have eight 10 Gigabit Ethernet, FCoE-capable, Enhanced Small Form-Factor Pluggable (SFP+) ports that connect the blade chassis to the fabric interconnect. The Cisco UCS 2204 has four external ports with identical characteristics to connect to the fabric interconnect.
  • Page 34 Cisco UCS B200-M3 backplane connections validated for the FlexPod. The Cisco UCS B200M3 server uses a VIC 1240 in the mLOM slot with an empty mezzanine slot. The Cisco UCS Fabric Extender 2204XP enables 2 KR lanes to the half-width blade while the global discovery policy dictates the formation of a fabric port channel.
  • Page 35 This passive device provides connectivity for the unused ports on the Cisco UCS VIC 1240, essentially enabling the 40-Gb potential of the mLOM card. Beyond the raw capacity improvements is the creation of two more automatic port channels between the FEX and the server.
  • Page 36 Figure 14 Validated UCS Backplane Configuration Using VIC 1240 with Port Extender Note “Appendix: Cisco UCS Fabric Interconnect and IOM Connectivity Diagrams” section on page 54 for additional combinations of UCS second-generation hardware and the connectivity options they provide. Figure 15 describes the availability and performance aspects of the second-generation Cisco UCS I/O gear.
  • Page 37 FlexPod Implementation and Design Figure 15 Cisco UCS VIC Availability for FEX 2204XP and 2208XP Options Cisco UCS VIC Availability for FEX 2204XP and 2208XP assumes the presence of Cisco UCS 6200 Note • Series Fabric Interconnects. Fabric failover is not required for deployments using the Cisco Nexus 1000v. For more information •...
  • Page 38 Cisco UCS 5108 blade chassis. Each 10 Gigabit Ethernet VIC port connects to fabric A or B through FEX. The Cisco UCS FEX and FIs form port channels automatically based on the chassis discovery policy providing a link resiliency to the Cisco UCS C-Series Server. This is identical to the behavior of the FEX to fabric interconnect connectivity.
  • Page 39 PortChannel (vPC) allowing links that are physically connected to two different Cisco Nexus 7000 Series devices to appear as a single PortChannel to a third device. In the FlexPod topology both Cisco UCS Fabric Interconnects and NetApp storage systems are connected to the Cisco Nexus 7000 switches through vPC.
  • Page 40 18. It is crucial that the fabrics do not mix, maintaining SAN A/B isolation best practices. The vPC links facing the Cisco UCS Fabric Interconnects, Po 13 and Po14, do not carry any FCoE traffic. Make sure not to define any FCoE VLANs on these links.
  • Page 41 7000 Series switch. This allows complete SAN A/SAN B isolation within the FlexPod environment. In each of the Storage VDCs a port channel, Po2, is dedicated to FCoE and connected to the Cisco UCS Fabric Interconnects. Each discrete port channel supports a single VLAN associated with Fabric A or Fabric B.
  • Page 42 The Cisco Nexus 7000 Series switch in the FlexPod design provides Fibre Channel over Ethernet services to the UCS and NetApp FAS platforms. Internally the Cisco Nexus 7000 platforms need to be configured to support FCoE zoning to enforce access policy between Cisco UCS-based initiators and NetApp FAS-based targets.
  • Page 43 Bridge assurance is only enabled on the vPC Peer Link. – Ports facing the NetApp storage controller and UCS are defined as “edge” trunk ports. – For information on Cisco Nexus 7000 Series Switches configuration details, see: http://www.cisco.com/en/US/partner/products/ps9402/products_installation_and_configuration_guides _list.html VMware vCenter and vSphere VMware vSphere 5.1 provides a platform for virtualization and comprises multiple components and...
  • Page 44 Cisco Nexus 1000v Architecture Figure 20 shows a single ESXi node with a VEM registered to the Cisco Nexus 1000v VSM. The ESXi vmnics are presented as Ethernet interfaces in the Cisco Nexus 1000v. In this example, the ESXi node is the third module in the virtual distributed switch as the Ethernet interfaces are labeled as module/interface #.
  • Page 45 FlexPod Discrete Uplink Design - Cisco Nexus 1000v Focus The Cisco Nexus 1000v supports port profiles. Port profiles are logical templates that can be applied to the Ethernet and virtual Ethernet interfaces available on the Nexus 1000v. In FlexPod architecture, the Cisco Nexus 1000v aggregates the Ethernet uplinks into a single port channel named the “System-Uplink”...
  • Page 46 IT organization. The Cisco UCS Virtual Interface Card (VIC) offers each VM a virtual Ethernet interface or vNIC. This vNIC provides direct access to the Cisco UCS Fabric Interconnects and Cisco Nexus 5500 Series Switches where forwarding decision can be made for each VM using Cisco VM-FEX interface.
  • Page 47 Fabric A and B. The red line indicates the current active fabric path in this example B. The Cisco UCS Manager allows administrators to assign a preferred active path (A or VMware vSphere 5.1 on FlexPod with Nexus 7000 Using FCoE...
  • Page 48 B) or assign no preference allowing the Cisco UCS Manager to provision active dynamic vNICs equally between fabric interconnects. The maximum number of Virtual Interfaces (VIF) that can be defined on a Cisco VIC Adapter depends on the following criteria and must be considered in any VM-FEX design: The presence of jumbo frames •...
  • Page 49 FlexPod Implementation and Design NetApp Storage Controllers NetApp Clustered Data ONTAP allows one or more storage HA pairs that are interconnected to be managed as a single system or pool of storage resources. Figure 22 details the logical configuration of the clustered Data ONTAP environment used during validation.
  • Page 50 FlexPod Implementation and Design Figure 22 FlexPod Discrete Uplink Design - NetApp Storage Controller Focus Nodes 1 and 2 form a two-node storage failover pair through the HA interconnect direct connection. The FlexPod design uses the following port and interface assignments: Port 3a and 4a on each node support FCoE data traffic that is accessible through an FC LIF assigned •...
  • Page 51 7-Mode does not scale beyond a single pair of controllers as compared to the clustered Data ONTAP. From a design perspective, the Cisco Nexus and Cisco UCS component configurations are identical to the previously defined FlexPod configuration with clustered Data ONTAP.
  • Page 52 (ifgrp). Each port of the ifgrp is connected to one of the upstream switches, allowing multiple active paths by utilizing the Cisco Nexus vPC feature. Interface group is a mechanism that allows the aggregation of a network interface into one logical unit. Combining links aids in network availability and bandwidth.
  • Page 53 FlexPod Implementation and Design Figure 23 FlexPod Discrete Uplink Design with Data ONTAP Operating in 7-Mode VMware vSphere 5.1 on FlexPod with Nexus 7000 Using FCoE...
  • Page 54 FlexPod is the optimal shared infrastructure foundation on which to deploy a variety of IT workloads. Cisco and NetApp have created a platform that is both flexible and scalable for multiple use cases and applications. One common use case is to deploy VMware vSphere as the virtualization solution, as described in this document.
  • Page 55 Appendix: Cisco UCS Fabric Interconnect and IOM Connectivity Diagrams Figure 25 shows two 10 Gbps active connections between the Cisco UCS VIC 1240 and 2104 Fabric Extenders. Figure 25 Connectivity Illustration with FEX model 2104 and VIC 1240 Figure 26 shows two 10 Gbps active connections between the Cisco UCS VIC 1240 and 2204 Fabric Extenders.
  • Page 56 20 Gbps port channels created by adding a port expander card to the mezzanine slot of the blade server using a Cisco UCS VIC 1240 card in the mLOM slot in combination with the Cisco UCS 2204 Fabric Extenders.
  • Page 57 B200 M3 Connectivity Illustration with FEX model 2204, VIC 1240 and Port Expander Figure 28 shows a Cisco UCS B200 M3 half-width blade server with both the Cisco UCS VIC 1240 and VIC 1280 adapters using the Cisco UCS 2204 Fabric Extenders. This results in 4 independent 10 Gbps connections.
  • Page 58 B200 M3 Connectivity Illustration with FEX model 2204, VIC 1240 and VIC 1280 Figure 29 shows a Cisco UCS VIC 1240 connecting to a pair of Cisco UCS 2208 Fabric Extenders forming two 20 Gbps port channels from the Cisco UCS B200 M3 half-width blade server.
  • Page 59 Connectivity Illustration with FEX model 2208 and VIC 1240 Figure 30 shows a Cisco UCS VIC 1240 with port expander card connecting to a pair of Cisco UCS 2208 Fabric Extenders. This configuration allows for two 40 Gbps capable port channels.
  • Page 60 B-200 M3 Illustration with FEX model 2208, VIC 1240 and Port Expander Figure 31 shows a half-width blade using the Cisco UCS VIC 1240 and 1280 in the mLOM and mezzanine slot respectively. This configuration combined with the Cisco UCS 2208 Fabric Extender results in four 20 Gbps aggregate port channels.
  • Page 61 This section provides references that can be helpful during FlexPod design implementation: Cisco Unified Computing System: • http://www.cisco.com/en/US/products/ps10265/index.html Cisco UCS 6200 Series Fabric Interconnects: • http://www.cisco.com/en/US/products/ps11544/index.html Cisco UCS 5100 Series Blade Server Chassis: • http://www.cisco.com/en/US/products/ps10279/index.html • Cisco UCS B-Series Blade Servers: http://www.cisco.com/en/US/partner/products/ps10280/index.html •...
  • Page 62 • http://www.vmware.com/products/datacenter-virtualization/vsphere/index.html Interoperability Matrix: • NetApp Interoperability Matrix Tool – http://support.netapp.com/matrix/ – Cisco UCS Hardware and Software Interoperability Tool http://www.cisco.com/web/techdoc/ucs/interoperability/matrix/matrix.html VMware Compatibility Guide – http://www.vmware.com/resources/compatibility/search.php NetApp Data ONTAP 8 Operating System • http://www.netapp.com/us/products/platform-os/data-ontap-8/index.aspx VMware vSphere 5.1 on FlexPod with Nexus 7000 Using FCoE...

This manual is also suitable for:

Vmware vsphere 5.1