The Managed IP Phones feature is one of the single greatest additions in recent years to CIC for the system administrator. It allows touchless deployment and automated user provisioning of phones across an organization and avoids the complexities of managing phone firmware and configuration files.
That said, troubleshooting issues with phone provisioning and operation can still seem a bit intimidating. This article should help to make this process a bit more understandable and includes screenshots and information about common troubleshooting steps and required configuration. It will focus on Polycom phones (though the principles can be applied to all SIP audio endpoints) and also refers to examples in this ZIP file.
Provisioning and Registration
For managed IP phones to successfully boot, provision, and register with CIC, a number of components must be in place. I highly recommend that all CIC system administrators take the time to read through the IC Managed IP Phones Administrator’s Guide for more detailed information and other scenarios that may differ from the examples used below.
DHCP – When a Polycom phone boots, it must first contact DHCP to obtain its IP, provisioning server, etc. This may seem a bit obvious, but do this, it must be able to reach the DHCP server. If on a different VLAN, a helper-address will be needed to route the DHCP request to the correct server. If unable to do so, the phone will display an error stating as much. Keep an eye out for rouge DHCP servers when troubleshooting DHCP-related issues.
ip address 192.168.10.1 255.255.255.0
ip helper-address 192.168.5.5
The DHCP server must provide the phone with a number of options (see screenshot below):
- 002 Time Offset – the offset from GMT for the time zone the phone is in. For example, Eastern time is GMT -5. Cisco provides a handy reference of the Hexadecimal values for each GMT offset value
- 003 Router – the default gateway for the subnet
- 004/042 Time Server – IP address of the server/device providing time to the phone
- 006 DNS Servers
- 015 DNS Domain Name – the name of the lookup zone to be used by the phone (used by option 160)
- 160 (custom) Polycom Provisioning – string containing fully qualified address for the provisioning server (should resolve to 2 different IP addresses matching the IPs of both CIC servers in a switchover pair)
DNS – Once it has obtained a reservation from DHCP, the phone must contact the provisioning server. To do this, it must first resolve the FQDN specified in DHCP option 160. This should resolve to the IP addresses of both CIC servers. The DNS lookup zone should include the following entries (others may be needed including media servers, etc.)
- Host records for each of the CIC servers
- 2 Host records for provisioning
- DNS SRV records for registration matching the protocol used: UDP/TCP/TLS (UDP is the default in current versions of CIC)
DNS configuration can be validated by using nslookup to resolve all hostnames and the DNS SRV records.
Troubleshooting – Phone provisioning and registration may fail for many reasons. Here are a few common ones to consider and rule out:
- The phone was unable to reach a DHCP server (check for helper-address and correct VLAN configuration, verify that the DHCP server is operational).
- DHCP was missing a required option or incorrectly configured (verify spelling of the provisioning string in option 160).
- DNS was unreachable or resolution of the FQDN for the provisioning servers failed (verify DNS functionality).
- Provisioning servers were unreachable (verify network path between phone and CIC servers including routes and firewalls, verify that the Provision Server subsystem is running, restart the subsystem if a problem with it is suspected).
- Phone failed to register – URL Call Disabled error on phone after boot (verify that the correct DNS domain and protocol are configured in the Registration Group, verify that the DNS SRV records can be resolved and are correctly configured, verify that a SIP Register message is sent from the phone and that CIC receives it) .
If an initial sweep doesn’t turn up anything obvious, a bit more invasive troubleshooting may be needed. Here are a few possibilities:
- Restart the Provision Server subsystem – Occasionally this subsystem can get into a bad state and needs to be refreshed. To do this, open IC System Manager, select the Provision Server subsystem, right-click, and select Restart Subsystem.
- Phone syslog – To configure the phone to send syslog data, first determine where to send it to. Interactive Intelligence provides a nifty utility called ACSyslog that I generally use for one-off troubleshooting. However, many organizations already have dedicated syslog servers.
If the phone is provisioning, but not registering, this can be configured in Interaction Administrator. Open the phone’s Configuration dialog, select the Options tab, click Advanced Options, and enter the destination IP address in the Syslog Server field. Also set the Syslog Render Level to 0. Reboot the phone once for it to obtain this configuration, then reboot it again to see the full boot process from the phone’s perspective.
If the boot process is not getting this far, enter the phone’s Setup menu as it begins to boot. The default password is typically 456. Select the Syslog Menu, enter the Server Address, set the Render Level to 0, save, and reboot.
This is where the first example comes in (click here to download the examples). The file syslog.txt includes a sample of a normal boot sequence for a Polycom IP 320 phone. It is normal to see partial downloads of firmware files (the phone will cancel the downloads when it determines they are not needed. But keep an eye out for any unusual errors. This is of the most useful troubleshooting tools for phone boot, provisioning, and registration issues as it shows what is happening from the perspective of the phone.
- Provision Server Log – The trace logs on the active CIC server for this subsystem are another useful place to look for errors in the provisioning process (assuming the phone is having trouble obtaining its configuration from CIC). See provisionserver.ininlog in the example ZIP file for a normal boot of the phone at default tracing levels. Again, some errors related to firmware are normal (as indicated in the log text), but keep an eye out for anything unexpected.
Once a phone has provisioned and registered, there may occasionally be problems encountered during its use. These can generally be divided into 3 categories:
1. Hardware – For example, nothing happens when one of the buttons on the keypad is pressed. First, try resetting the phone to factory defaults. Enter the phone’s menu, then select the Admin menu, Reset to Defaults, then Format File System and/or Reset to Factory (these options may vary slightly depending on the model of Polycom phone). If this does not resolve the issue, there is likely something truly wrong with the phone. If under warranty, request an RMA.
2. Call Control – Polycom phones use SIP to communicate with CIC and RTP to transport the audio for a call. SIP is the protocol used to control the setup, flow, and teardown of the call. If a SIP session cannot be successfully established, the call will fail.
One of the best places to troubleshoot SIP issues is in the SIP Engine trace log on the active CIC server. With tracing set to the “Notes” level, this log can be filtered to see SIP messages as they flow to and from CIC. With the log open, Press Ctrl + Alt + F to open the Quick String Filter dialog and enter SIP/2.0 in the Text to Find field.
Once a phone completes provisioning, the first thing it must do is register. To do this, it will send a SIP Register message (making use of the DNS SRV records mentioned earlier) to the active CIC server. This request will initially be rejected and then resent by the phone with digest authentication credentials. See normal_registration.pdf in the example ZIP file.
When working correctly, a phone call placed from the Interaction Client/Desktop will cause CIC to “Invite” the phone to participate in the call and the phone will respond. See normal_call.pdf in the example ZIP file. Once the phone responds, CIC will then place the call via a SIP gateway and connect the call to the phone. If, for some reason, the phone is not reachable, the user will receive an error. This could be because the phone is not registered (no contact address error) or is unreachable (station unreachable error). If the phone has registered, but cannot be reached, this will be visible in the SIP Engine log as a series of Invites to the phone with no response. See phone_unreachable.pdf in the example ZIP file.
There are many other possible scenarios (a TCP session timeout, encryption failure, etc.), but the point is that the SIP Engine log is a really great tool to see why a call failed.
As with provisioning, a syslog from the phone can also be very useful to help rule out what is happening from the phone’s perspective vs. the CIC server’s perspective.
3. Audio – Troubleshooting call audio issues can be the most difficult. But there are a few questions and considerations that can help to quickly narrow down the issue:
- What happened vs. what should have happened? A clear description of the problem is critical.
- When did the issue begin – when was it last working correctly?
- What changed? There is nearly always something even though nearly everyone will answer “nothing”.
- How widespread is the issue – does it only affect a single user, a specific group of users, or everyone? Determining the scope of the issue can help to quickly identify the cause. Perhaps a switch issue, a broadcast storm on a VLAN, or a saturated link.
- Is it reproducible? Problems that can be reproduced are infinitely easier to troubleshoot. If the issue cannot be reproduced and doesn’t reoccur, finding the cause can be impossible.
- Who reports hearing the issue? Determine whether one party, the other, or both experience the issue. For example, an agent may hear an inbound customer call just fine, but the caller may not hear the agent if the agent’s headset is unplugged. For conference calls, always find out who DOES NOT hear the issue. They are the most likely cause.
Once the issue has been clearly identified (assuming it hasn’t already been resolved), there are a couple of tools that can be used to analyze the call audio to determine the cause of the issue:
- Call Recording – If a recording of the call in question exists, listen to it. Whether or not the reported issue is present in the call recording will help you narrow down the cause. For example, an agent might report hearing a buzzing noise that is not present in the recording. This would point to the agent’s phone or headset
- Packet Capture – Always be prepared to perform a packet capture for any call audio issue. Wireshark is the tool of choice for performing network packet captures. For certain call quality issues, the CIC media servers also have the ability to store diagnostic captures for audio that passes through them.
Analyzing packet captures is beyond the scope of this article, but here are a few tips:
- Capture beginning where the issue is not present and working back towards where it is. For example, one party in a call reports not hearing the other. Begin at the phone that was “not heard” and work backwards to the phone where the issue was reported. The point at which the audio disappears will be the culprit.
- Work “outside in”. For reports of audio issues with external calls, it is often useful to begin capturing at the point the audio enters your network and then work backwards into the environment. This is a quick way to rule out whether your environment is involved in the issue at all.
- Verify, verify, verify. The packet capture is the ultimate tool to verify what is and is not happening for just about any call issue (whether audio-related or not). Use captures to confirm whether communication is taking place between endpoints and whether messages are actually sent and do indeed reach their intended destination. Just because an issue is reported a certain way doesn’t mean that it actually happened that way.
If you have any questions about troubleshooting your Managed IP Phones, please reach out to us. We’d be happy to help!