By James Hanback
Like many other people who live in houses, I have a contract with a pest control company. Now and then, someone arrives at my door wearing a crisp uniform and carrying a giant canister full of some kind of magical chemical concoction. This pest controller uses a long wand that is attached to the canister to conjure up a magic invisible shield that prevents six- or eight- or hundred-legged creatures from invading my space. I know the pest control service is working and worth it when I am not confronted by any tiny crawling, flying, or undulating critters. Thus the debugger who visits my house every now and again does so because I don’t want to see any evidence of bugs in my living space.
Debugging in the networking world works a bit differently. If you enable debugging in Cisco IOS, it’s because you are specifically looking for the cause of a problem. In other words, you want to see the bugs. Fortunately for you, there’s the handy Cisco IOS debug command, which exists primarily to overwhelm you with more information than you can ever possibly parse or understand about nearly every aspect of an IOS configuration.
I’m kidding, of course. But have you ever attempted to follow the trail of context-sensitive help for the debug command? It’ll take about 10 screens just to list all the secondary keywords that are available immediately following the command, let alone the tertiary and higher levels.
Nevertheless, the debug command is invaluable when you’re experiencing issues that are not necessarily easy to diagnose by examining a running configuration or by examining the output of the show commands that are related to the technology you’re attempting to troubleshoot. This is because the debug command allows you to examine output in real-time, as the device is attempting to send or receive traffic, rather than relying on the somewhat static output that is presented in show commands.
Of course, there’s a caveat. It is recommended that you use debug commands on a production network only if you have been instructed to do so by someone at the Cisco Technical Assistance Center (TAC). This is because debugging can be a significant resource drain on your hardware, your scrolling finger, and your eyesight. Like the chemical the household pest controller sprays, you should typically use the debug command sparingly and only when appropriate.
Imagine that you handle Enhanced Interior Gateway Routing Protocol (EIGRP) administration on Router1 while another administrator manages Router2 at a remote location. Now imagine that the Router2 admin adds a new network to the Router2 EIGRP configuration and decides that the new network should not be advertised to Router1. Foolishly, the Router2 admin issues the EIGRP passive-interface command for the interface that connects Router2 to Router1. Because EIGRP passive interfaces block both inbound and outbound advertisements, the immediate result is a broken neighbor relationship between the two EIGRP routers.
If you happened to be monitoring the console of Router1 at the time, your first indication that something is wrong might be a Syslog message similar to the following:
%DUAL-5-NBRCHANGE: EIGRP-IPv4 10: Neighbor 192.168.51.50 (FastEthernet0/0) is down: Interface PEER-TERMINATION received
At that point you might just call up the Router2 admin and ask a pertinent question or two. But what if the other administrator was working at an off time? What if you didn’t even notice that there was a problem until the following morning, when users arrived at work and started sending support tickets your way? (Of course, we’re assuming that you did not have the foresight to implement any kind of system monitoring that would alert you as soon as a problem is detected.)
In that case, you might have entirely missed the console message. And because the passive-interface command was issued on the remote router, not the router you administer, the show ip protocols command will turn up zilch in the way of hints. Nor will the show ip eigrp topology, the show ip eigrp neighbors, and the show ip eigrp interfaces commands be of any help (except to inform you that no neighbor relationship exists between Router1 and Router2, which is important information but does not indicate where the cause of the failure lies). Pretty much all you can do with show commands at this point is verify that the configuration on Router1 has not changed since you last examined it and ensure that the interface connected to Router2 is in the up state.
You can also reliably rule out some common adjacency problems simply by noting the absence of certain console messages. For example, you can be fairly certain that K value mismatches are not causing the adjacency failure in this case because there are no repeating K value mismatch errors on the console (or in the error logs, if you use a Syslog server). If K values were the problem, you might see repeating messages on Router1 similar to the following:
%DUAL-5-NBRCHANGE: EIGRP-IPv4 10: Neighbor 192.168.51.50 (FastEthernet0/0) is down: K-value mismatch
So you know the interface and line protocol are up. You know that the EIGRP configuration on Router1 has not changed. And you’re fairly certain that there’s no K-value mismatch between the routers. That narrows down the possible causes of the adjacency failure quite a bit. The next thing you might do is verify that the neighbor router is actually still alive. After all, an interface in the up state does not necessarily mean that the device on the other end is responsive. So you might issue a show cdp neighbors or at least a ping command to verify that Router2 is still there. In this case, let’s assume that you have determined that it is.
Now that you’ve exhausted all your ideas, it’s time to wave the magic wand and issue the debug eigrp packets hello command because you—you smart cookie, you—also know that in order for two routers to establish an EIGRP adjacency, they must be able to exchange Hello packets.
The debug eigrp packets hello command allows you to watch the sending and receiving of EIGRP Hello packets in real-time. Therefore, you should be able to see whether Router1 is actually receiving EIGRP Hello packets from Router2. An EIGRP router that has successfully formed a relationship with its neighbor will display debug eigrp packets hello command output similar to the following:
*Sep 22 10:47:28.415: EIGRP: Sending HELLO on Fa0/0 - paklen 30
*Sep 22 10:47:28.415: AS 10, Flags 0x0:(NULL), Seq 0/0 interfaceQ 0/0 iidbQ un/rely 0/0
*Sep 22 10:47:28.519: EIGRP: Received HELLO on Fa0/0 - paklen 30 nbr 192.168.51.50
*Sep 22 10:47:28.519: AS 10, Flags 0x0:(NULL), Seq 0/0 interfaceQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/0
Notice that the first line of the output indicates that Router1 is sending an EIGRP Hello packet through the FastEthernet 0/0 interface. The second line indicates that the packet was sent in Autonomous System (AS) 10. The third line indicates that the router has received a Hello packet on the FastEthernet 0/0 interface from a neighbor that has a router ID of 192.168.51.50. The last line indicates that the packet was received in AS 10.
By contrast, an EIGRP router that is connected to a neighbor that has been incorrectly configured with the passive-interface command would produce the following output from the debug eigrp packets hello command:
*Sep 22 10:50:14.983: EIGRP: Sending HELLO on Fa0/0 - paklen 20
*Sep 22 10:50:14.987: AS 10, Flags 0x0:(NULL), Seq 0/0 interfaceQ 0/0 iidbQ un/rely 0/0
Router1#
*Sep 22 10:50:19.891: EIGRP: Sending HELLO on Fa0/0 - paklen 20
*Sep 22 10:50:19.895: AS 10, Flags 0x0:(NULL), Seq 0/0 interfaceQ 0/0 iidbQ un/rely 0/0
See the difference? Instead of a series of alternating sent and received Hellos over the FastEthernet 0/0 interface, you see only that Router1 is attempting to send Hello packets out its FastEthernet 0/0 interface. When the passive-interface command is configured for an EIGRP process, both inbound and outbound Hello packets are blocked on the passive interface. Router1 is unable to establish an adjacency with Router2 because Router2 is not receiving Hello packets that are being sent from Router1 and Router2 is not sending Hello packets to Router1.
It’s important to point out that the sample output above assumes that EIGRP is configured on a single router interface. In reality, you might see the successful receipt of Hello packets on other interfaces among the debug output. Therefore, it’s important to pay attention to all the information in a line of output, not just whether it’s a send or receive.
Also, this particular example does not precisely reveal that the cause of the problem is the passive-interface command. You would see the same output if Router2’s administrator had foolishly modified the AS number for the EIGRP process on the Router2 side or if the network command on Router2 had been modified so that the EIGRP process was no longer operating in the same subnet as Router1. Of course, there are other ways of detecting those issues as well, such as the EIGRP log message IP-EIGRP: Neighbor ip address not on common subnet for interface. Let’s assume you looked for and didn’t see that particular message in this scenario. So now you can rest assured that the problem is most likely on the remote side of the connection. Thus you can call up the Router2 administrator and commence with a chewing out in earnest.
Naturally, the level of success you have with the Cisco IOS debug command depends on the value of the information available in the output and your understanding of the technology you are troubleshooting. For example, you could technically detect or eliminate the possibility of an AS mismatch by examining the output of the debug ip packet command. However, the informing output from that command can be difficult to isolate and decipher. An actual AS mismatch might be easier to detect by using a third-party network troubleshooting tool such as Wireshark. In this case, the debug eigrp packets hello command gives you just enough information to determine that you need to place a call to the remote administrator. Not every scenario will reveal that much information. Others will reveal much, much more.
At this point, I should double up on the reminder that, when wielding the mighty power of the debug command, you might put a significant drain on the device’s CPU, which in turn could negatively affect the performance of your network. As soon as you have what you need from debugging, you should disable it. For example, in this scenario you would want to issue the no debug eigrp packets hello command or the no debug all command (which turns off all debugging processes at once) as soon as you think you’ve gathered the information you need from the output. Although you can disable debugging in pieces, it’s far more efficient and more effective to issue the no debug all command as soon as your output requirements are met. Even after you disable debugging, you might continue to see the device belch output at you until the debugging buffer is clear and there’s nothing more to produce. By issuing the no debug all command, you can feel relatively safe that any additional debugging output you see is buffer-related and not because you forgot to disable a specific debug command. Also remember that if the show commands had revealed a problem on Router1 at the outset, such as a shutdown interface, you most likely wouldn’t need to issue the debug eigrp packets hello command at all. Therefore, you should not rely on debug commands as your first level of troubleshooting.
The power of the debug command is mighty, indeed. Wielded properly, you’ll be able to shine a bright light on any creepy crawlers that might be hiding in your network configuration. Unlike in your living space, you want to be able to see the bugs in your network configuration. If you can’t see them, you can’t squish them.
