A Necessary Guide to the Avaya traceSM Utility

Necessity is the mother of invention.

Case in point is the Avaya utility, traceSM. As hard as I’ve tried, I have yet to find a halfway decent manual that explains how to use what is perhaps the most important Session Manager debugging tool. I would love to be proved inept in my efforts, but I’ve searched through every Session Manager document I can find. I have also asked people who work with traceSM on a regular basis if they’ve found anything useful and I always come up empty.

Therefore, I am documenting what I know in the hope that the next person who endeavors a search will find this blog article and have something to work with.

---

These two articles may help you better understand Avaya SIP and subsequently, traceSM.

The Steps Involved in Booting an Avaya SIP Telephone

Understanding Avaya’s Personal Profile Manager (PPM)

---

traceSM is an interactive perl script that allows an administrator to capture, view, and save call processing activity on a Session Manager. While not as powerful or versatile as wireshark, traceSM is absolutely essential when it comes to working with Avaya SIP. First off, it allows you to view SIP messages even if they have been encrypted with TLS. You cannot do this with wireshark. Second, traceSM allows you to see Avaya specific data such as Personal Profile Manager (PPM) messages and Session Manager call flows.

To run traceSM you need to Telnet into an active Session Manager. Personally, I use the TuTTy program, but you can accomplish the same thing with vanilla PuTTY.

By default, only one traceSM can be running on a Session Manager at any given time.  To invoke multiple instances, it must be run with the -m option.  However, Avaya advises against running multiple instance of traceSM in a production environment as it may cause performance problems.

traceSM -m

traceSM should be accessible through the default path, but in case you ever need to find it, look under /opt/Avaya/contrib/bin.  Remember, Linux is case sensitive.  It’s traceSM and never tracesm or Tracesm.

Since you are in a Telnet session, forget about using your mouse. It’s keystrokes only, Baby.

<UP>,<DOWN>	Select a SIP/SM packet. Or scroll a large SIP packet when displaying the details
<HOME>	Go to the first packet
<END>	Go to the last packet. If the cursor is in the last packet while capturing packets, the screen will update with new arriving packets
<PGUP>,<PGDN>	Page Up and Page Down
<LEFT>,<RIGHT>	Move between different columns (IPs) when they don’t fit in the screen
<ENTER>	Display the SIP/SM details. The SIP URI is highlighted in red, the SIP fields in blue and the content (e.g: SDP, xml) in green.
q	Quit
f	Display the Filter window to view/change filters
w	Write the displayed (filtered) packets to a new file
s	Start or Stop the capture
c	Clear the screen
a	Switch between SM and SM-100 perspective
i	Switch between displaying Names or IPs in the column headers
r	Switch between displaying RTP simulation or not
d	Switch between SIP calls and display mode

It’s important to know that traceSM is a real-time capture tool. You cannot start traceSM and expect to see SIP packets that were sent prior to launching the tool. The capture begins when the application begins.

Also, traceSM’s capture buffer is limited in size to 10,000 packets and once that limit is reached, it stops collecting and displaying packets.  You will need to clear the buffer to get it moving again.

The following is an example of a typical traceSM screen. Note the handy list of commands at the bottom.

Also, take note of the words SIP, PPM, and CallP. SIP and PPM are in green and CallP is in red. This indicates that traceSM is currently only capturing SIP and PPM messages. You can change which packet and message types are captured with the “s” command.

In the following example, only SIP packets will be captured.  To capture PPM or Call Processing messages,”arrow” to the appropriate box and press Enter.  OK will restart the capture with the new parameters.

To view the expanded contents of a packet, up or down arrow to the packet you wish to view and press Enter.

Some SIP messages are so short that they can be completely viewed on the screen while others require you to arrow down to see everything.  The three dots at the bottom center of a message window will tell you if there is more to be seen.

In this example, I’ve selected to view the contents of an INVITE message.  Notice how traceSM displays different aspects of the SIP message in different colors — the Request Line is red, headers are blue, and the SDP is green.

Filters

A busy system will cause traceSM to capture and display packets faster than you can comprehend what is going on.  This is where the filter command comes in handy.  To create a filter, press “f” and select your desired options.

You can filter on everything from IP address to particular header values to specific SIP URIs or numbers.  For example, to only see the packets to or from extension 1902, you would set the following filter:

-u 1902

It’s often useful to create filters that eliminate “unnecessary” information.  For example, you can filter out OPTION messages (-no), registrations (-nr), subscriptions (-ns), and Session Manager’s call routing logic (-na).

By default, filters are logically added as an AND.  Specify -or to create either/or filters.

Specific to Avaya is the AV-Global-Session-ID header.  Filtering on this header allows you to do cradle-to-grave call tracing since the SIP Call-ID will change as a session moves through B2BUA entities, but AV-Global-Session-ID will not.

Filters are cleared by entering “f” without any options.

The following screen shot shows you the many different filters that can be created.

By default, traceSM will display ingress and egress points by name.  Use the “i” command to switch to display by IP address.

Use the “d” command to switch between displaying everything in the capture buffer to just the SIP calls.  Use “d” to then toggle back to a full display of all captured packets.

In this example you will see three SIP calls.  Two were successfully completed and one was rejected.

Earlier versions of traceSM did not allow you to capture and display PPM messages, but thankfully that has been resolved.  You need to enable their capture through the “s” command and traceSM will include them in the display.  As with SIP messages, pressing Enter  on a specific entry shows you the entire PPM message.

You can also trace the actions of Session Manager itself by capturing the Call-P messages.  Seeing Call-P messages can be valuable in debugging routing issues.

Saving Your Work

You can save the contents of the display buffer with the “w” command and clear the display buffer with the “c” command.  Saved traces can be later accessed with traceSM or exported to view with wireshark.

Files are written to the “home directory” of the logged in user.  For example, if you logged as “cust,” the file will be found in the /home/cust directory.

It is important to know that only the information currently shown on the traceSM screen is saved in the file.  This makes it easy to identify a problem with a filter, save the messages to a file, and then email the file to a support resource for further analysis.

Help!

If you find yourself in front of a Session Manager console screen and cannot recall all that I just tried to tell you, all you need to do is ask for help.

traceSM -h gives you the following information:

Mischief Managed

---

For some real-life examples of traceSM in action, please see these articles:

A traceSM View of Avaya Aura IMS Processing

Avaya Aura IMS Processing Once Again

A Look at how Avaya Aura Supports SIP-Based Voice Mail

A Look at SIP-Based Voice Mail Part Two

---

The best way to really learn a tool like traceSM is to use it and hopefully this brief guide will get you started.  Again, if anyone knows of a real document, please let me know.  Pretty much everything I’ve written here is what I’ve figured out on my own and the chances are high that I missed something.

Tags: Avaya, Avaya Aura Session Manager, traceSM