A Necessary Guide to the Avaya traceSBC Utility

A few months ago, I wrote what has become the most popular guide for the Avaya Session Manager SIP tracing tool, traceSM. Of course, it doesn’t hurt that I may have written the only guide, but I will take recognition in any way that I can.

Today, I hope to do the same for the packet trace tool that Avaya wrote for their session border controller – traceSBC.

Before I do that, though, I would like to acknowledge the hard work of my friend and coworker, John Waber. Not only did John suggest that I write this article, but he put together a number of PowerPoint slides about traceSBC that I freely stole from. Thank you, John.

---

My article on traceSM can be found at A Necessary Guide to the Avaya traceSM Utility.

---

Like traceSM, traceSBC is a Perl script that is run from a PuTTY or TuTTY session connected to the SBC’s Management (M1) IP address. However, where traceSM ran on SSH port 22, traceSBC must be run on port 222.

Note, if you have two SBCs setup in a high availability configuration, you need to connect to the management IP of the active server.

You must have root privileges to run traceSBC. If necessary, you can add them from the command line with:

# sudo su

When you are ready to begin tracing, enter “traceSBC” from a command prompt. Like traceSM, capitalization is important.  TraceSBC and tracesbc are not the same as traceSBC.

traceSBC Fundamentals

There are a few important things you must understand about traceSBC in order to get the most out of it:

• traceSBC can be used even when the packets have been encrypted.
• traceSBC only shows packets that have not been rejected. This means that if a message violates a firewall rule or does not match a defined SBC role, the packet is not displayed.
• Some non-SIP messages can be displayed with traceSBC. Specifically, Avaya SIP telephones and soft clients will interact with the Personal Profile Manager via https. traceSBC can display these messages, too.
• traceSBC can only display a maximum of 10,000 messages. Upon reaching this limit, the warning MAX NUM PACKETS 10000 EXCEEDED is shown. The clear command must be applied to display new messages.
• traceSBC can run in Automated Mode. This allows you to define triggers that stop displaying packets when one of the following conditions is met:
  • Quantity of packets captured
  • Quantity of seconds expired
  • A message is captured that contains text matching a pre-defined regular expression

While running, traceSBC supports the following commands:

• c clear the capture screen
• s stop displaying new messages
• w write the displayed information to a file
• q quit/exit traceSBC
• i toggle between displaying the IP address or server name
• r turn on RTP simulation
• u use the full screen for message detail
• f apply a filter
• d switch between classic or call summary views (diagrams)

traceSBC can be launched with the following options:

• -m allow the execution of multiple copies of traceSM (from separate TuTTY/PuTTY connections)
• -w write to a file
• –h show help
• -a automatic mode

traceSBC Filters

An active SBC will typically process far more SIP messages than you care to look at. Thankfully, filters can be applied to narrow down the number of packets displayed.

The following filter options are available:

-u URI|Number	Filter calls that contain URI|Number is the From or To headers
-i IP Address	Filter messages from/to IP Address
-c Call-ID	Filter messages bases on the SIP Call-ID header
-r REGEXP	Filter messages based on the regular expression
-g HEA=VALUE	Filter SIP header filed HEA for the value VALUE
-or	Use a logical OR operator on two filter options
-nr	Do not display REGISTER messages
-ns	Do not display SUBSCRIBE/NOTIFY/PUBLISH messages
-no	Do not display OPTIONS messages
-na	Do not display SBC call processing messages
-np	Do not display Personal Profile Manager messages
-uni	Use Unicode/UTF-8 characters
-m	Allows multiple simultaneous instances of traceSBC
-w file	Set filename for saving filtered messages
-a TYPE	Starts capture in non-interactive mode TYPE. TYPE can be sip, ppm, or callp
-st SEC	Stops capture after SEC seconds
-sp PACKET	Stops capture after a given number of packets have been received
-sr REGEXP	Stops capture if regular expression found a match
-srt SEC	Run trace SEC more seconds after REGEXP match
-srp PACKET	Collect PACKET more messages after REGEXP match

Several filters may be applied at the same time. For example:

-u “4563516” –no –ns –nr

A filter with no options clears any previous settings.

SBC Log Files

As it runs, the Avaya SBC stores captured packet data in log files. This applies to both SIP and PPM.   These files are found in the following directories:

SIP Messages: /archive/log/tracesbc/tracesbc_sip/

PPM Messages: /archive/log/tracesbc/tracesbc_ppm/

A new file is created when the relevant processes restart or the active file reaches 10 megabytes. At this point, the active file is saved and a new one is created.

Active files are stored in the following format:

tracesbc_sip_1408631651

Inactive or closed files are stored in the following format:

tracesbc_sip_1408631651_1408631234_1

traceSBC uses the active file to format and display packets in a user friendly format. If that file reaches the 10 megabyte limit, traceSBC continues processing using the newly created file.

traceSBC Modes

traceSBC can be run in three different manners:

1. Real-Time Mode

In real-time mode, traceSBC processes the active log file. As stated above, the active log file will be regenerated when the 10 megabyte limit is reached.

Real-time mode is invoked as follows:

# traceSBC

2. Non Real-Time Mode

Non real-time mode allows traceSBC to process a non-active file. In this mode, traceSBC can process one or more archived files. Enabling live capture is not possible in non real-time mode.

Non real-time mode is invoked similarly to the following:

# traceSBC tracesbc_sip_1408

3. Automatic Mode

Automatic mode is invoked by starting traceSBC with the –a and –w parameters at a minimum.

For example,

# traceSBC –a “sip|ppm” –w /tmp/trace.log

You can use the mode to stop a capture after a certain condition has been met. The filtered messages are then automatically stored in the file specified on the command line. Multiple stop triggers may be used. Also, cntrl-c will cause traceSBC to stop and save the filtered messages.

User Interface

As with traceSM, navigation is very basic. Up and down arrows move through the filtered messages and enter expands a particular message. Don’t try to mouse around the window. This is a basic TTY interface and your mouse is useless here.

A typical call trace will look like this:

Select a message, hit enter, and the message will expand:

While traceSBC is gathering and filtering packets, you can use any of the runtime commands such as c, s, or f.

When the w command is invoked, traceSBC will store the filtered SIP and/or PPM messages. SIP messages are stored in pcapng format and PPM messages are stored in a separate file with the extension .ppm. These files can then be exported to another machine for further examination. This is necessary if you want to use Wireshark for packet parsing.

For fun, using the s (start) commend, I used traceSBC to gather only Personal Profile Manager (PPM) messages and then logged in my Avaya IOS SIP client.

Selecting the top message and pressing enter gives me this:

Notice that this is an https POST message. Not only does the Avaya SBC process SIP messages, it also supports the https messages required by PPM.

Your Turn

The best way to learn any debug utility is to use it and I suggest this is a good time to do just that. I hope that this explanation has empowered you with enough confidence to give traceSBC a good shakedown. Debugging a complicated SIP call flow isn’t all that bad if you have the right tools. Although it’s far from perfect, traceSBC can often be the only thing you need when it comes to finding and fixing SIP problems.

P.S.

Due to the highly technical nature of this article, I wanted to make sure that I got it right.  To ensure that I did, I sent the article to a number of people prior to clicking “Publish.”  One of my reviewers, Nick Kwiatkowski, offered up the following additions.  Thank you, Nick!

Additional Logging Capabilities

In addition to the logging capabilities shown above, it’s possible to have a rolling log of historical and routing information. The busyness of your system will determine how long the rolling log is, but even on very busy systems, it can extend to well over a day.

To start the log, connect to the EMS (Enterprise Management System) Web Page, log in, and go to Device Specific Settings -> Troubleshooting -> Debugging. Under devices, click on your SBC instance and select all the processes with SSYNDI as the title. Info and Warning should be enough information for 99% of the logs. The system will write these logs to the /usr/local/ipcs/log/ss/logfiles/elog/SSYNDI directory.

Leaving the log running may degrade performance, so make sure you watch the performance of the SBC. However, even with busy SBCs, I haven’t seen noticeable degradation and often keep these logs up and running.

To view the logs in traceSBC, log in to the SBC as above, upgrade your privileges to root, and change to the /usr/local/ipcs/log/ss/logfiles/elog/SSYNDI directory. Execute an ls command to find the closest log with the items you wish to explore. Enter the command traceSBC <logfile>.

After traceSBC opens the log, you can apply filtering and other commands on the file. An added bonus is access to policy and routing decisions. These are incredibly useful when trying to determine why packets match certain policies, user agent settings, etc.

For even more detail, note the time stamp of the entry you are looking at before exiting traceSBC. You can then open the log file and do a search for the time-stamp to get a very verbose look at everything that is going on inside the brains of the SBC for that packet.

Advertisement

Tags: Avaya, SBC, Session Border Controller