1. Manuals
  2. Brands
  3. Cisco Manuals
  4. Network Router
  5. 10000 Series
  6. Troubleshooting manual

Cisco 10000 Series Troubleshooting Manual

Hide thumbs Also See for 10000 Series:
Table of Contents

Quick Links

See also: Configuration Manual, Installation Manual
Cisco 10000 Series Router
Troubleshooting Guide
Corporate Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: 408 526-4000
800 553-NETS (6387)
Fax: 408 526-4100
Text Part Number: OL-0439-02
Table of Contents
loading

Related Manuals for Cisco 10000 Series

Summary of Contents for Cisco 10000 Series

  • Page 1 Cisco 10000 Series Router Troubleshooting Guide Corporate Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 526-4100 Text Part Number: OL-0439-02...
  • Page 2 OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. CCVP, the Cisco Logo, and the Cisco Square Bridge logo are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn is a service mark of Cisco Systems, Inc.;...

  • Page 3: Table Of Contents

    Basic Troubleshooting Tasks and Startup Issues C H A P T E R Basic Troubleshooting Tasks Confirming the Hardware Installation Displaying Cisco IOS Software Version and System Environment Information Using the show version Command Using the show environment Command for Basic Troubleshooting Hardware Troubleshooting Flowchart...
  • Page 4 Troubleshooting the OC-12 Packet-Over-SONET Line Card Troubleshooting the Channelized T3 Line Card Troubleshooting the Gigabit Ethernet Line Card Cisco 10000 Series Router Alarms and Error Messages C H A P T E R OC-12 POS Line Card Alarms and Error Messages...
  • Page 5 Testing with Digital Multimeters and Cable Testers Testing with TDRs and OTDRs Testing with TDRs Testing with OTDRs Testing with Breakout Boxes, Fox Boxes, and BERTs/BLERTs Testing with Network Monitors Testing with Network Analyzers N D E X Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...
  • Page 6 Contents Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...
  • Page 7 “Related Documentation” section on page viii. The Cisco 10000 series router supports the aggregation of leased lines and many high-density and Layer 3 services. You configure, manage, and maintain the router by using the Cisco IOS command line interface (CLI). Audience To benefit from this guide, you must be experienced using Cisco IOS and have some responsibility for installing, configuring, or operating the Cisco 10000 series router.

  • Page 8: About This Guide

    Line Card Configuration Guide—Contain detailed information on line card configuration and administration. For additional information about the Cisco 10000 series router, refer to the Cisco 10000 Series Router Hardware Installation Guide. For more information about the Cisco IOS software that runs on the router, refer to the Cisco IOS command reference books and configuration guides.

  • Page 9: Obtaining Documentation

    Obtaining Documentation Cisco documentation and additional literature are available on Cisco.com. Cisco also provides several ways to obtain technical assistance and other technical resources. These sections explain how to obtain technical information from Cisco Systems.

  • Page 10: Product Documentation Dvd

    The DVD enables you to access multiple versions of installation, configuration, and command guides for Cisco hardware and software products. With the DVD, you have access to the same HTML documentation that is found on the Cisco website without being connected to the Internet.

  • Page 11: Reporting Security Problems In Cisco Products

    We encourage you to use Pretty Good Privacy (PGP) or a compatible product (for example, GnuPG) to encrypt any sensitive information that you send to Cisco. PSIRT can work with information that has been encrypted with PGP versions 2.x through 9.x.

  • Page 12: Cisco Technical Support & Documentation Website

    Cisco engineer. The TAC Service Request Tool is located at this URL: http://www.cisco.com/techsupport/servicerequest For S1 or S2 service requests, or if you do not have Internet access, contact the Cisco TAC by telephone. (S1 or S2 service requests are those in which your production network is down or severely degraded.) Cisco engineers are assigned immediately to S1 and S2 service requests to help keep your business operations running smoothly.

  • Page 13: Obtaining Additional Publications And Information

    Cisco products that are sold through channel partners. It is updated twice a year and includes the latest Cisco offerings. To order and find out more about the Cisco Product Quick Reference Guide, go to this URL: http://www.cisco.com/go/guide...
  • Page 14 You can access the Internet Protocol Journal at this URL: http://www.cisco.com/ipj Networking products offered by Cisco Systems, as well as customer support services, can be • obtained at this URL: http://www.cisco.com/en/US/products/index.html...

  • Page 15: Chapter 1 Basic Troubleshooting Tasks And Startup Issues

    Basic Troubleshooting Tasks and Startup Issues This chapter describes the basic procedures that users should perform before undertaking a detailed troubleshooting analysis of the Cisco 10000 series router or logging a case with the Cisco Technical Assistance Center (TAC). These basic troubleshooting checks are organized as follows: Basic Troubleshooting Tasks, page 1-1 •...

  • Page 16: Displaying Cisco Ios Software Version And System Environment Information

    If the problem persists, consult the Cisco 10000 series router release notes to confirm that this is not an existing problem. Finally, you should consider replacing the component.

  • Page 17: Using The Show Environment Command For Basic Troubleshooting

    Power Entry Module 0 type DC status: OK Power Entry Module 1 type AC status: Failure The Cisco 10000 series router default temperature alarm thresholds are programmed to assert at approximately these conditions: Minor inlet and core alarms will assert when the ambient temperature is within a few degrees of 40 •...

  • Page 18: Hardware Troubleshooting Flowchart

    Cisco 10000 series router is malfunctioning. Figure 1-1 describes a series of hardware dependent startup events that must take place for a Cisco 10000 series router to allow the passage of IP traffic. At each main point of the flowchart, there are pointers to the chapters in this guide that describe how to troubleshoot individual pieces of hardware.

  • Page 19: Cisco 10000 Series Router System Startup Sequence

    Cisco 10000 Series Router System Startup Sequence Cisco 10000 Series Router System Startup Sequence Table 1-1 describes the visible sequence of events that occur during a typical Cisco 10000 series router power up. Table 1-1 Cisco 10000 Series Router System Startup Sequence...
  • Page 20 Chapter 1 Basic Troubleshooting Tasks and Startup Issues Cisco 10000 Series Router System Startup Sequence Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...

  • Page 21: Chapter 2 Pem Faults And Blower Failures

    C H A P T E R PEM Faults and Blower Failures The following sections provide methods for troubleshooting faults involving the Cisco 10000 series router Power Entry Modules (PEMs) and blower modules. This chapter contains the following major sections: DC PEM Faults, page 2-1 •...
  • Page 22 If the MISWIRE LED is on, the power cables are reversed. Power off the PEM and the external power source and reconnect the wires correctly. See the Cisco 10000 Series Router Hardware Installation Guide. Cisco 10000 Series Router Troubleshooting Guide...

  • Page 23: Ac Pem Faults

    AC PEM LEDs located at the bottom of the front panel. Table 2-3 AC PEM LEDs Status Condition POWER Steady green Power is available. FAULT Steady yellow PEM has failed or is turned off. Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...

  • Page 24: Other Electrical Problems

    Call the Cisco TAC. Other Electrical Problems If the electrical problem cannot be traced to a PEM, check the unit for: Improper power cable connections to the Cisco 10000 series router • Improper installation of other field-replaceable units (FRUs) •...

  • Page 25: Blower Module Faults

    Blower Module Faults Blower Module Faults The blower module is critical to the operation of the Cisco 10000 series router. Severe overheating can result in system failure. The blower module contains a redundant array of fans in a single shelf unit.
  • Page 26 Chapter 2 PEM Faults and Blower Failures Blower Module Faults Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...

  • Page 27: Chapter 3 Troubleshooting Pres

    PRE fault states, the management Ethernet port, and the serial port. PRE Module Faults The PRE provides the IP routing and forwarding functionality in the Cisco 10000 series router. Thus, in a nonredundant PRE configuration, a PRE failure is a system failure. A redundant PRE configuration is recommended.
  • Page 28 System is booting. yellow ALARMS Green PRE is active (primary). Flashing PRE is standby (secondary). green No power to PRE. Yellow Fail A major failure has disabled the PRE. The PRE is operating properly. Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...

  • Page 29: Pre Module Faults

    Return the card to its slot and screw it firmly into place. Replace the card. If the problem persists, contact the Cisco TAC. FAIL LED is yellow, Reinsert the PRE.

  • Page 30: Console Port Serial Connection Problems

    “Obtaining Technical Assistance” section on page -xi instructions on contacting the Cisco TAC. Console Port Serial Connection Problems If the console screen connected to a Cisco 10000 series router console port appears frozen or fails to work properly • Refer to the “Cisco 10000 Series Router System Startup Sequence”...
  • Page 31 If the cable, connections, power, and terminal settings all check out and you still cannot connect to the console port on the module, replace the module in question. If the problem persists, contact the Cisco TAC for further assistance.
  • Page 32 Chapter 3 Troubleshooting PREs Troubleshooting PREs Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...

  • Page 33: Troubleshooting The Oc-12 Packet-Over-Sonet Line Card

    C H A P T E R Troubleshooting Line Cards This chapter discusses troubleshooting faults on the following Cisco 10000 series router line cards: Troubleshooting the OC-12 Packet-Over-SONET Line Card, page 4-1 • • Troubleshooting the Channelized T3 Line Card, page 4-4 Troubleshooting the Gigabit Ethernet Line Card, page 4-6 •...
  • Page 34 A major failure has disabled the line card. The line card is operating properly. Rx (receive) Green Receiving traffic. Not receiving traffic. Tx (transmit) Green Transmitting traffic. Not transmitting traffic. Carrier Green Carrier detected. Carrier not detected. Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...

  • Page 35: Chapter 4 Troubleshooting Line Card

    If there are no bent pins, replace with a new line card. Try inserting the line card in a different slot. If the card works in a different slot, the Cisco 10000 series router Card seems to be passing backplane may be defective. Contact the Cisco TAC.

  • Page 36: Troubleshooting The Channelized T3 Line Card

    DS1 OOF, DS3 OOF, DS3 AIS, or DS3 FERF 1 level. No alarm condition. Loop (active loop) Yellow Port is in a loopback state and not enabled for data traffic. Port is not in a loopback state. Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...
  • Page 37 Check the cable connections at the remote equipment. Replace the card. Try the card in a different slot. If the card works in a different slot, you may have a damaged backplane. Call the Cisco TAC. Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...

  • Page 38: Troubleshooting The Gigabit Ethernet Line Card

    LEDs on the gigabit Ethernet line card faceplate to help you verify correct operation. Make sure that the gigabit Ethernet Interface Converter (GBIC) type on the Cisco 10000 series router matches the GBIC type at the other end of the fiber optic cable.
  • Page 39 Card seems to be passing traffic (Tx/Rx lights), but cannot communicate with the PRE If the card works in a different slot, the Cisco 10000 series router backplane may be defective. Call the Cisco TAC. Fail LED blinks steadily This is a user correctable problem. The steadily blinking LED indicates a transmit failure.
  • Page 40 Chapter 4 Troubleshooting Line Cards Troubleshooting the Gigabit Ethernet Line Card Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...

  • Page 41: Chapter 5 Cisco 10000 Series Router Alarms And Error Messages

    Cisco 10000 Series Router Alarms and Error Messages This chapter describes the alarms and error messages that can be raised on the Cisco 10000 series router. The Cisco 10000 series router interface, environment, and system alarms provide feedback on the functional status of a chassis, PRE, and line cards.
  • Page 42 Chapter 5 Cisco 10000 Series Router Alarms and Error Messages OC-12 POS Line Card Alarms and Error Messages Table 5-1 SONET Alarms Alarm Type and Severity Alarm Symptoms Recommendation SLOS Alarm messages appear in the Check the fiber optic cable to CLI and logs.

  • Page 43: Oc-12 Pos Line Card Bit Error Rate Alarms

    Chapter 5 Cisco 10000 Series Router Alarms and Error Messages OC-12 POS Line Card Alarms and Error Messages Table 5-1 SONET Alarms (continued) Alarm Type and Severity Alarm Symptoms Recommendation PLOP Alarm messages appear in the CLI This is an error in the data. As with and logs.

  • Page 44: Oc-12 Pos Line Card Error Messages

    Chapter 5 Cisco 10000 Series Router Alarms and Error Messages OC-12 POS Line Card Alarms and Error Messages Table 5-2 BER Alarms (continued) Alarm Type and Severity Alarm Symptoms Recommendation BER_SF BER_SF and BER_SD alarms In both cases, test the quality of result in APS cutovers.
  • Page 45 Chapter 5 Cisco 10000 Series Router Alarms and Error Messages OC-12 POS Line Card Alarms and Error Messages Table 5-4 describes the software-related error messages that the OC-12 POS line card might generate. It also defines the problem causing the error message and suggests a corrective action.

  • Page 46: Channelized T3 Line Card Alarms And Error Messages

    Chapter 5 Cisco 10000 Series Router Alarms and Error Messages Channelized T3 Line Card Alarms and Error Messages Channelized T3 Line Card Alarms and Error Messages This section describes alarm and error messages that can be generated by the channelized T3 line card.

  • Page 47: Channelized T3 Line Card Error Messages

    Chapter 5 Cisco 10000 Series Router Alarms and Error Messages Channelized T3 Line Card Alarms and Error Messages Channelized T3 Line Card Error Messages Table 5-6 lists the error messages that the channelized T3 line card can generate. Table 5-6...

  • Page 48: Gigabit Ethernet Line Card Error Messages

    Chapter 5 Cisco 10000 Series Router Alarms and Error Messages Gigabit Ethernet Line Card Error Messages Gigabit Ethernet Line Card Error Messages Table 5-7 describes the error messages associated with the gigabit Ethernet (GE) line card. Where possible, the source of the error message is explained and a corrective action is suggested.

  • Page 49: Chapter 6 Troubleshooting Serial Lines

    C H A P T E R Troubleshooting Serial Lines This chapter describes methods for detecting and correcting data errors on the Cisco 10000 series router serial interfaces. Optical Signal Input/Output Problems Signal input and output problems can occur at any point in the network and can be caused by mechanical defects in cables or fiber, poor connections, or loss of signal caused by other equipment failures.

  • Page 50: Evaluating The Power Budget

    Chapter 6 Troubleshooting Serial Lines Optical Signal Input/Output Problems Table 6-1 Optical Fiber Types for Cisco 10000 Series Router Line Cards Card Type Appropriate Fiber Type OC-12 Packet Over SONET line card Single mode. Gigabit Ethernet line card The appropriate fiber type for the gigabit Ethernet line card is dependent upon the installed GBIC.

  • Page 51: Using Bit Error Rate Tests

    Use a loopback at a location of your choice in the link or network. • • Configure remote testing equipment to transmit the same BER test pattern at the same time. Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...

  • Page 52: Sending A Ber Test Pattern On A T1 Line

    The example that follows is for a T1, numbered 10, on a CT3 line card in slot 1: Router(config)# controller T3 1/0/0 Router(config-controller)# t1 10 bert pattern 1s interval 14400 unframed Note You can terminate a BER test during the specified test period with the no t1 bert command. Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...

  • Page 53: Viewing The Results Of A Ber Test

    Indicates the test pattern you selected for the test (2^11), the Test Pattern : 2^11, Status : Sync, Sync Detected : 1 current synchronization state (sync), and the number of times synchronization has been detected during this test (1). Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...

  • Page 54: Terminating A Ber Test

    Configuring a T3 Controller to Respond to Remote Loopback Commands, page 6-7 • Configuring the Loopback Mode for a Gigabit Ethernet Interface, page 6-8 • Configuring the Loopback Mode for an OC-12 POS Interface, page 6-8 • Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...

  • Page 55: Configuring The Loopback Mode For A T3 Controller

    Router(config-controller)# equipment customer loopback To prevent a controller from responding to remote loopback commands, type: Router(config)# controller T3 1/0/0 Router(config-controller)# equipment network loopback Remote loopbacks are only available when you use c-bit parity framing. Note Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...

  • Page 56: Configuring The Loopback Mode For A Gigabit Ethernet Interface

    Use the no form of the command to stop the loopback test. In the following example, a loopback is set for the OC-12 POS line card in slot 5: Router(config)# interface pos 5/0/0 Router(config-if)# loopback line Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...

  • Page 57: Chapter 7 Replacing Or Recovering Passwords

    This chapter describes how to recover a lost enable or console login password, and how to replace a lost enable secret password on the Cisco 10000 series router. It is possible to recover the enable or console login password. The enable secret password is encrypted, Note however, and must be replaced with a new enable secret password.

  • Page 58: Password Recovery Procedure

    EXEC password is near the end). The passwords displayed appear similar to the following: enable secret 5 $1$ORPP$s9syZt4uKn3SnpuLDrhuei enable password 23skiddoo line con 0 password onramp Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...
  • Page 59 Enter the copy running-config startup-config command to save the new configuration to NVRAM. Step 18 Enter the reload command to reboot the router. Step 19 Log in to the router using the new or recovered passwords. Step 20 Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...
  • Page 60 Chapter 7 Replacing or Recovering Passwords Password Recovery Procedure Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...

  • Page 61: Appendix

    • • frame-relay map protocol protocol-address dlci [broadcast] [ietf | cisco] [payload-compress {packet-by-packet | frf9 stac [hardware-options]}] frame-relay map ip ip-address dlci [broadcast] [cisco | ietf] [nocompress] tcp • header-compression {active | passive} • frame-relay mincir {in | out} bps frame-relay traffic-rate average [peak] •...
  • Page 62 Appendix A Unsupported Commands Unsupported MLPPP Commands multilink load-threshold • multilink max-fragments • ppp multilink fragment-delay • ppp multilink fragmentation • ppp multilink idle-link • • ppp multilink interleave Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...

  • Page 63: Appendix

    A P P E N D I X Recommended Tools and Test Equipment Table B-1 lists the basic tools and test equipment necessary to perform general maintenance and troubleshooting tasks on the Cisco 10000 series router. Table B-1 Recommended Tools and Test Equipment Equipment...

  • Page 64: Testing With Tdrs And Otdrs

    An OTDR can be used to identify the “signature” of a particular installation, noting attenuation and splice losses. This baseline measurement can then be compared with future signatures if you suspect a problem in the system. Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...

  • Page 65: Testing With Breakout Boxes, Fox Boxes, And Berts/Blerts

    Use network analyzers (also called protocol analyzers) to decode protocol layers in a recorded frame and present the layers as readable abbreviations or summaries, detailing which layer is involved (physical, data link, and so forth) and the function each byte or byte content serves. Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...
  • Page 66 Incorporate an “expert” system in which the analyzer uses a set of rules, combined with information • about the network configuration and operation, to diagnose and solve, or offer potential solutions to, network problems. Cisco 10000 Series Router Troubleshooting Guide OL-0439-02...

  • Page 67: I N D E X

    GBIC 4-6, 4-7 gigabit Ethernet DC PEM faults 2-1 to 2-3 autonegotiation deserializing method GBIC type documentation viii gigabit Ethernet interface converter document conventions viii DS1 Alarms DS3 Alarms Cisco 10000 Series Router Troubleshooting Guide IN-1 OL-0439-02...
  • Page 68 Ethernet show version 1-2 to 1-3 mode conditioning patch cord single mode multimode slot problems startup system startup behavior overheating terminal settings PEM faults troubleshooting basic fiber optic cables power cables flowchart Cisco 10000 Series Router Troubleshooting Guide IN-2 OL-0439-02...
  • Page 69 Index hardware line cards 4-1 to 4-7 serial lines troubleshooting line cards 4-1 to 4-7 Cisco 10000 Series Router Troubleshooting Guide IN-3 OL-0439-02...
  • Page 70 Index Cisco 10000 Series Router Troubleshooting Guide IN-4 OL-0439-02...

Table of Contents