A few months ago, I wrote what has become the most popular guide for the Avaya Session Manager SIP tracing tool, traceSM. Of course, it doesn’t hurt that I may have written the only guide, but I will take recognition in any way that I can.
Today, I hope to do the same for the packet trace tool that Avaya wrote for their session border controller – traceSBC.
Before I do that, though, I would like to acknowledge the hard work of my friend and coworker, John Waber. Not only did John suggest that I write this article, but he put together a number of PowerPoint slides about traceSBC that I freely stole from. Thank you, John.
My article on traceSM can be found at A Necessary Guide to the Avaya traceSM Utility.
Like traceSM, traceSBC is a Perl script that is run from a PuTTY or TuTTY session connected to the SBC’s Management (M1) IP address. However, where traceSM ran on SSH port 22, traceSBC must be run on port 222.
Note, if you have two SBCs setup in a high availability configuration, you need to connect to the management IP of the active server.
You must have root privileges to run traceSBC. If necessary, you can add them from the command line with:
# sudo su
When you are ready to begin tracing, enter “traceSBC” from a command prompt. Like traceSM, capitalization is important. TraceSBC and tracesbc are not the same as traceSBC.
There are a few important things you must understand about traceSBC in order to get the most out of it:
- traceSBC can be used even when the packets have been encrypted.
- traceSBC only shows packets that have not been rejected. This means that if a message violates a firewall rule or does not match a defined SBC role, the packet is not displayed.
- Some non-SIP messages can be displayed with traceSBC. Specifically, Avaya SIP telephones and soft clients will interact with the Personal Profile Manager via https. traceSBC can display these messages, too.
- traceSBC can only display a maximum of 10,000 messages. Upon reaching this limit, the warning MAX NUM PACKETS 10000 EXCEEDED is shown. The clear command must be applied to display new messages.
- traceSBC can run in Automated Mode. This allows you to define triggers that stop displaying packets when one of the following conditions is met:
- Quantity of packets captured
- Quantity of seconds expired
- A message is captured that contains text matching a pre-defined regular expression
While running, traceSBC supports the following commands:
- c clear the capture screen
- s stop displaying new messages
- w write the displayed information to a file
- q quit/exit traceSBC
- i toggle between displaying the IP address or server name
- r turn on RTP simulation
- u use the full screen for message detail
- f apply a filter
- d switch between classic or call summary views (diagrams)
traceSBC can be launched with the following options:
- -m allow the execution of multiple copies of traceSM (from separate TuTTY/PuTTY connections)
- -w write to a file
- –h show help
- -a automatic mode
An active SBC will typically process far more SIP messages than you care to look at. Thankfully, filters can be applied to narrow down the number of packets displayed.
The following filter options are available:
|-u URI|Number||Filter calls that contain URI|Number is the From or To headers|
|-i IP Address||Filter messages from/to IP Address|
|-c Call-ID||Filter messages bases on the SIP Call-ID header|
|-r REGEXP||Filter messages based on the regular expression|
|-g HEA=VALUE||Filter SIP header filed HEA for the value VALUE|
|-or||Use a logical OR operator on two filter options|
|-nr||Do not display REGISTER messages|
|-ns||Do not display SUBSCRIBE/NOTIFY/PUBLISH messages|
|-no||Do not display OPTIONS messages|
|-na||Do not display SBC call processing messages|
|-np||Do not display Personal Profile Manager messages|
|-uni||Use Unicode/UTF-8 characters|
|-m||Allows multiple simultaneous instances of traceSBC|
|-w file||Set filename for saving filtered messages|
|-a TYPE||Starts capture in non-interactive mode TYPE. TYPE can be sip, ppm, or callp|
|-st SEC||Stops capture after SEC seconds|
|-sp PACKET||Stops capture after a given number of packets have been received|
|-sr REGEXP||Stops capture if regular expression found a match|
|-srt SEC||Run trace SEC more seconds after REGEXP match|
|-srp PACKET||Collect PACKET more messages after REGEXP match|
Several filters may be applied at the same time. For example:
-u “4563516” –no –ns –nr
A filter with no options clears any previous settings.
SBC Log Files
As it runs, the Avaya SBC stores captured packet data in log files. This applies to both SIP and PPM. These files are found in the following directories:
SIP Messages: /archive/log/tracesbc/tracesbc_sip/
PPM Messages: /archive/log/tracesbc/tracesbc_ppm/
A new file is created when the relevant processes restart or the active file reaches 10 megabytes. At this point, the active file is saved and a new one is created.
Active files are stored in the following format:
Inactive or closed files are stored in the following format:
traceSBC uses the active file to format and display packets in a user friendly format. If that file reaches the 10 megabyte limit, traceSBC continues processing using the newly created file.
traceSBC can be run in three different manners:
- Real-Time Mode
In real-time mode, traceSBC processes the active log file. As stated above, the active log file will be regenerated when the 10 megabyte limit is reached.
Real-time mode is invoked as follows:
- Non Real-Time Mode
Non real-time mode allows traceSBC to process a non-active file. In this mode, traceSBC can process one or more archived files. Enabling live capture is not possible in non real-time mode.
Non real-time mode is invoked similarly to the following:
# traceSBC tracesbc_sip_1408
- Automatic Mode
Automatic mode is invoked by starting traceSBC with the –a and –w parameters at a minimum.
# traceSBC –a “sip|ppm” –w /tmp/trace.log
You can use the mode to stop a capture after a certain condition has been met. The filtered messages are then automatically stored in the file specified on the command line. Multiple stop triggers may be used. Also, cntrl-c will cause traceSBC to stop and save the filtered messages.
As with traceSM, navigation is very basic. Up and down arrows move through the filtered messages and enter expands a particular message. Don’t try to mouse around the window. This is a basic TTY interface and your mouse is useless here.
A typical call trace will look like this:
Select a message, hit enter, and the message will expand:
While traceSBC is gathering and filtering packets, you can use any of the runtime commands such as c, s, or f.
When the w command is invoked, traceSBC will store the filtered SIP and/or PPM messages. SIP messages are stored in pcapng format and PPM messages are stored in a separate file with the extension .ppm. These files can then be exported to another machine for further examination. This is necessary if you want to use Wireshark for packet parsing.
For fun, using the s (start) commend, I used traceSBC to gather only Personal Profile Manager (PPM) messages and then logged in my Avaya IOS SIP client.
Selecting the top message and pressing enter gives me this:
Notice that this is an https POST message. Not only does the Avaya SBC process SIP messages, it also supports the https messages required by PPM.
The best way to learn any debug utility is to use it and I suggest this is a good time to do just that. I hope that this explanation has empowered you with enough confidence to give traceSBC a good shakedown. Debugging a complicated SIP call flow isn’t all that bad if you have the right tools. Although it’s far from perfect, traceSBC can often be the only thing you need when it comes to finding and fixing SIP problems.
Due to the highly technical nature of this article, I wanted to make sure that I got it right. To ensure that I did, I sent the article to a number of people prior to clicking “Publish.” One of my reviewers, Nick Kwiatkowski, offered up the following additions. Thank you, Nick!
Additional Logging Capabilities
In addition to the logging capabilities shown above, it’s possible to have a rolling log of historical and routing information. The busyness of your system will determine how long the rolling log is, but even on very busy systems, it can extend to well over a day.
To start the log, connect to the EMS (Enterprise Management System) Web Page, log in, and go to Device Specific Settings -> Troubleshooting -> Debugging. Under devices, click on your SBC instance and select all the processes with SSYNDI as the title. Info and Warning should be enough information for 99% of the logs. The system will write these logs to the /usr/local/ipcs/log/ss/logfiles/elog/SSYNDI directory.
Leaving the log running may degrade performance, so make sure you watch the performance of the SBC. However, even with busy SBCs, I haven’t seen noticeable degradation and often keep these logs up and running.
To view the logs in traceSBC, log in to the SBC as above, upgrade your privileges to root, and change to the /usr/local/ipcs/log/ss/logfiles/elog/SSYNDI directory. Execute an ls command to find the closest log with the items you wish to explore. Enter the command traceSBC <logfile>.
After traceSBC opens the log, you can apply filtering and other commands on the file. An added bonus is access to policy and routing decisions. These are incredibly useful when trying to determine why packets match certain policies, user agent settings, etc.
For even more detail, note the time stamp of the entry you are looking at before exiting traceSBC. You can then open the log file and do a search for the time-stamp to get a very verbose look at everything that is going on inside the brains of the SBC for that packet.
This is a great post!!! Do you know if there is a way to specify a unique user agent? We are running IPO and we would like to set a unique user agent per device for an extra layer of security instead of the default Avaya one-X Deskphone.
Thank you. Are you asking if there is a way to block all registrations that aren’t of a specific user agent type? I am a bit confused about your question.
Thank you for replying! I am trying to set a unique ID for mobile devices that are using the Avaya OneX Mobile app. For example, when setting the user agent, in SBC, you can use a regular expression Avaya.* or Avaya one-X Deskphone. We would like to set or own User Agent ID. For example “CompanyName-iPhone5478”. Is this possible??
This is great.. Do you know if its possible to write the traceSBC file to an external source.
I have a scenario whereby i need to capture a trace to catch a weird scenario on the SBC over a day or so.
Use the Write (“w”) command.
Andrew, this is a very well written and useful article. I can across it today as I was looking for a way to kill a running instance of the traceSBC process after my laptop crashed while the session was still active. While I don’t see the option in your article, I thought I would provide that information as I came across a similar command that applied to the traceSM process for Session Manager.
“traceSBC -k” killed the running instance and started a new instance for me without issue. Credit goes to RogerthePhoneGuy for the suggestion of the “traceSM -k” command, which should have the same effect on Session Manager.
I hope someone finds this information useful.
Very good post, thank you for sharing Andrew.
A few things I have noticed (SBCE 7.2.2) and may be useful to mention for others
1. Once on the command line logged in with ‘ipcs’ there is no need to su root if you type ‘tracesbc’ (all lowercase) it will run the tool just as well. not sure if this applies to all SW versions but worth a try if you’re stuck for the root PW. If you need to write ‘w’ the capture to a file you’ll still need root access to drag the file off from the home/ipcs folder
2. After runnning ‘tracesbc’, hitting the ‘f’ key for the filter function will only apply the ‘DISPLAY FILETR’, packets are still filling up the buffer in the backgroup and will max out at 10000. To overcome this and give yoursellf more time and file space (especially on very heavily loaded SBC’s). run the filter at the command line.
i.e. tracesbc -u 123456789
This will apply the ‘CAPTURE FILTER’ and keep the file size much lower